An image registration method and system based on multi-modal feature fusion

Abstract

This invention provides an image registration method and system based on multimodal feature fusion, belonging to the field of multimodal image registration technology. The invention constructs a multi-scale Gaussian image pyramid, performs layer-by-layer cross-modal histogram specification preprocessing, and uses a shared feature network with domain-discriminative constraints to map bimodal deep and shallow features to a modality-invariant space to generate a fused feature map. A topological graph structure is constructed based on superpixels, and multi-scale dense matching relationships are generated through graph convolution matching. Finally, the initial deformation field is solved recursively from coarse to fine along the pyramid, and the registered image is output after layer-by-layer iterative optimization using registration similarity as the criterion. This invention achieves high-precision and highly robust multimodal image registration through cross-modal feature fusion within a multi-scale Gaussian pyramid framework, superpixel topological graph matching, and layer-by-layer iterative optimization of the deformation solution.

Description

Technical Field

[0001] This invention relates to the field of multimodal image registration technology, specifically to an image registration method and system based on multimodal feature fusion. Background Technology

[0002] Image registration is a core foundational technology in computer vision and image processing, used to achieve precise spatial alignment of multiple images acquired from different viewpoints, at different times, or using different imaging devices. Multimodal image registration, in particular, performs spatial matching between images from different imaging mechanisms and is a prerequisite for multimodal information fusion and collaborative analysis of multi-source data. It has significant application value in fields such as clinical diagnosis, remote sensing mapping, intelligent monitoring, and industrial inspection. Existing multimodal image registration techniques are mainly divided into two categories: traditional methods and deep learning methods. Traditional registration methods mainly fall into two categories: one category uses gray-level statistics such as mutual information and normalized mutual information as similarity measures, and achieves alignment by iteratively optimizing the deformation field. However, this type of method relies on gray-level statistical correlation, and is prone to failure when there are large nonlinear differences in gray-level mapping between modes. In addition, it has low iteration efficiency, is prone to getting trapped in local optima in large deformation scenes, and lacks stability. The other category is based on matching handmade feature points and descriptors to solve the transformation model. However, handmade features have limited robustness to cross-modal gray-level differences. It is difficult to extract stable feature points in areas with missing textures and complex structures. Furthermore, sparse matching cannot guarantee pixel-level dense alignment accuracy, and cannot meet the requirements of high-precision registration.

[0003] Existing image registration schemes suffer from multiple technical drawbacks: traditional gray-level statistical matching methods rely on image gray-level correlation, which is prone to failure in scenes with similar gray-level features and weak contrast; features extracted by conventional convolutional networks lack modal alignment constraints, resulting in unstable feature mapping relationships and making it difficult to construct reliable matching benchmarks. Furthermore, traditional feature point matching methods have sparse feature distributions, failing to meet the pixel-level dense registration accuracy requirements; and most matching algorithms utilize only isolated single-point features without integrating pixel neighborhood structures and geometric constraints, easily leading to spatial geometric matching errors and causing numerous incorrect matches, making it difficult to guarantee overall registration accuracy and robustness.

[0004] Therefore, it is necessary to provide an image registration method and system based on multimodal feature fusion to solve the aforementioned problem.

[0005] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide an image registration method and system based on multimodal feature fusion to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: An image registration method based on multimodal feature fusion includes the following steps: S1. Obtain the fixed and floating images to be registered, construct multi-scale Gaussian image pyramids respectively, perform cross-modal histogram specification preprocessing on the images at each level of the pyramid, and generate a standardized multi-scale preprocessed image set; S2. Input the preprocessed images of each level into a shared feature extraction network with domain discriminative constraints. Unify the dual-image feature extraction rules through the shared network parameters. Rely on the domain discriminative constraints to suppress modal feature shift. Fuse the shallow texture features and deep semantic structure features of the image to form heterogeneous features. Map the heterogeneous features to the modality-invariant feature space and output the cross-modal fusion feature maps of each level. S3. Perform superpixel segmentation on the cross-modal fusion feature maps at each level to generate structurally continuous superpixel units. Using superpixels as nodes and adjacency relationships as topological edges, construct the topological graph structures of the two images respectively. Calculate the superpixel feature similarity by combining the topological information through a graph convolutional matching network to generate a dense matching point set and establish multi-scale dense matching relationships. S4. Based on the multi-scale dense matching relationship, the deformation is solved step by step from coarse to fine along the Gaussian pyramid. The initial deformation is obtained by point set interpolation. The coarse-scale optimized deformation field is upsampled as the initial value for the fine-scale solution. The full-resolution initial deformation field is obtained by step-by-step recursion. S5. Using registration similarity as the convergence criterion, the initial deformation field is iteratively optimized in layers, and the deformation deviation is continuously corrected until the full-resolution registration accuracy meets the preset accuracy threshold. The optimal deformation field is obtained, and the original floating image is mapped at the pixel level through a spatial transformation network to output the final registered image.

[0008] Furthermore, multi-scale Gaussian image pyramids were constructed, and cross-modal histogram specification preprocessing was performed on the images at each level of the pyramid to generate a standardized multi-scale preprocessed image set. The method used was as follows: Gaussian kernel convolution blurring is applied to both fixed and floating images, followed by a 2x downsampling operation. The blurring and downsampling process is repeated iteratively to construct a 3-5 level multi-scale Gaussian image pyramid, where the resolution of each level of the pyramid is 1 / 2 of the previous level. For each level of the pyramid image, the grayscale histogram of the corresponding fixed image at the same level is used as a reference template to perform histogram specification mapping on the floating image at the same level, calibrate the grayscale distribution of the floating image, make the grayscale distribution of the two types of images tend to be consistent, eliminate the global grayscale difference of cross-modal images, and obtain the preprocessed image of each level. The preprocessed images from all levels are integrated to form a multi-scale preprocessed image set.

[0009] Further, the specific logic of step S2 is as follows: Construct a parameter-sharing dual-branch shared feature extraction network. Preprocessed images at each level of both the fixed and floating images are synchronously input into the shared feature extraction network. The two branches completely reuse the same set of network parameters to ensure that the operation rules and weight parameters for feature extraction of the two types of images are completely consistent. The network has a built-in domain discrimination constraint module, which identifies and suppresses feature distribution shifts caused by different imaging modalities through domain adversarial training, and removes modality-specific interference features. Simultaneously extract shallow edge texture features and deep semantic structure features of the image, fuse and encode multi-dimensional features to form heterogeneous features, and uniformly map the heterogeneous features to the modality-invariant common feature space to generate cross-modality fusion feature maps that are aligned at each level and are modality-independent.

[0010] Furthermore, superpixel segmentation is performed on the cross-modal fusion feature maps at each level to generate structurally continuous superpixel units. The method used is as follows: Pixel clustering segmentation is performed on the cross-modal fusion feature maps at each level. With texture similarity threshold and spatial distance threshold as constraints, the pixels of the fusion feature image are clustered and aggregated. Pixels with similarity and spatial distance both less than the corresponding threshold are aggregated into the same superpixel unit to complete global clustering segmentation and finally obtain a set of superpixels with continuous structure and uniform distribution. Using superpixels as nodes and adjacency relationships as topological edges, the topological graph structures of the two images are constructed respectively. The method used is as follows: All the independent superpixel units obtained from the segmentation are defined as feature nodes of the topological graph. The spatial positional relationship of all superpixel units is traversed to determine whether there are adjacent contact regions between different superpixel units. A connection line is established between two feature nodes that have a spatial adjacency relationship, and this connection line is the topological edge. The adjacency traversal and topological edge connection of all feature nodes are completed in sequence to form undirected topological graph structures that are adapted to fixed images and floating images respectively.

[0011] Furthermore, by combining graph convolutional matching networks with topological information to calculate superpixel feature similarity, a dense set of matching points is generated, and multi-scale dense matching relationships are established. The method used is as follows: The topological graph structures of fixed and floating images at the same level are synchronously input into a graph attention convolutional network. The neighborhood topological features of each superpixel node are aggregated using the graph attention mechanism, and the feature vectors of each node are iteratively updated to obtain an enhanced node feature set that incorporates global structural information. For the enhanced node feature sets of the two images, cosine similarity is calculated for each pair of superpixel nodes to construct a cross-graph node similarity matrix. An initial superpixel matching pair is then selected using a bidirectional nearest neighbor matching strategy. Based on the spatial coordinate information of the initial superpixel matching pairs, a random sampling consensus algorithm is used to fit a global geometric transformation model. The geometric deviation of each matching pair relative to the model is statistically analyzed to obtain the superpixel matching correspondence. The superpixel-level matching relationship is mapped to the pixel dimension, allowing all pixels within the same superpixel region to inherit the matching attribute of that superpixel. Bilinear interpolation is then applied to the pixel-level matching results to smooth them, generating a uniform and dense set of pixel correspondences at that level. This process is repeated to construct dense matching relationships at all scales.

[0012] Furthermore, the deformation is solved step-by-step along the Gaussian pyramid from coarse to fine, and the initial deformation is obtained through point set interpolation. The method used is as follows: The deformation solution is set to follow the progressive order of the Gaussian pyramid from the coarsest scale to the finest scale. For each pyramid level, the point set corresponding to the dense pixels of that level is used as the deformation fitting control point. The thin plate spline interpolation algorithm is used to fit the nonlinear spatial transformation relationship between the fixed image and the floating image, solve the spatial displacement vector of each pixel, and generate the initial deformation field of the current level. The coarse-scale optimized deformation field upsampling is used as the initial value for the fine-scale solution, and the full-resolution initial deformation field is obtained by iteratively deriving step by step. The method used is as follows: For each fine-scale level except the coarsest one, the deformation field optimized by the previous coarsest scale is upsampled to the current scale resolution via bilinear interpolation and used as the initial reference for the deformation solution at this level. The residual deformation field is solved by combining the dense corresponding point set at this level, and the residual deformation field is superimposed with the initial reference deformation field pixel by pixel. The optimized deformation field of the current level is obtained by iterative optimization. After completing the iterative solution of deformation at all levels of the pyramid, the optimized deformation field corresponding to the finest scale is output as the final full-resolution initial deformation field.

[0013] Furthermore, using registration similarity as the convergence criterion, the initial deformation field is iteratively optimized in layers to continuously correct deformation deviations until the full-resolution registration accuracy meets the preset threshold. The method used is as follows: The registration similarity is used to measure the gray-level correlation and alignment between the fixed and floating images across modalities. The iterative optimization follows a hierarchical advancement strategy of Gaussian pyramid from coarse scale to full-resolution fine scale. The initial deformation field at full resolution is downsampled and matched to the resolution of each pyramid level as the initial deformation benchmark for iterative optimization at each scale. For the current optimization scale, the preprocessed floating image at this level is resampled based on the deformation field at the current scale, and the normalized mutual information value between the resampled floating image and the fixed image at the same level is calculated. With maximizing the normalized mutual information value as the optimization objective, the deformation field parameters at the current scale are continuously iteratively corrected using the gradient descent algorithm until the change in mutual information in a single iteration is less than the preset single-scale convergence threshold. The current scale optimization is then determined to be complete, and the optimized deformation field at this level is output. The deformation field optimized at the current scale is upsampled to the next fine scale through bilinear interpolation as the initial value for the next scale iteration. The intra-scale optimization and cross-scale deformation transfer are completed iteratively. When the iterative optimization advances to the full-resolution level and the normalized mutual information value meets the preset accuracy threshold, the global iteration process is terminated, and the optimal deformation field is output.

[0014] Furthermore, a spatial transformation network is used to perform pixel-level deformation mapping on the original floating image to output the final registered image. The method used is as follows: A spatial transformation network adapted to pixel-level spatial mapping is constructed, and the optimal deformation field obtained by iterative optimization is used as the network transformation parameter input. Based on the pixel displacement information of the optimal deformation field, spatial offset mapping is performed on all pixel coordinates of the original floating image. The bilinear interpolation algorithm is used to perform grayscale resampling and completion on the mapped pixels to complete the global deformation alignment of the original floating image and output the final registered image.

[0015] The present invention also provides an image registration system based on multimodal feature fusion, the system being used to execute the image registration method based on multimodal feature fusion described in any one of the above claims, comprising: The hierarchical processing module is used to acquire the fixed and floating images to be registered, construct multi-scale Gaussian image pyramids respectively, perform cross-modal histogram specification preprocessing on the images at each level of the pyramid, and generate a standardized multi-scale preprocessed image set. The cross-modal fusion module is used to input preprocessed images from each level into a shared feature extraction network with domain discriminative constraints. By unifying the dual-image feature extraction rules through shared network parameters, it suppresses modal feature shift by relying on domain discriminative constraints, fuses shallow texture features and deep semantic structure features of the image to form heterogeneous features, maps the heterogeneous features to the modality-invariant feature space, and outputs cross-modal fusion feature maps of each level. The graph structure matching module is used to perform superpixel segmentation on cross-modal fusion feature maps at various levels, generate structurally continuous superpixel units, construct the topological graph structure of two images with superpixels as nodes and adjacency relationships as topological edges, calculate the superpixel feature similarity by combining topological information through a graph convolutional matching network, generate a dense matching point set, and establish multi-scale dense matching relationships. The deformation field solution module is used to solve the deformation stepwise from coarse to fine along the Gaussian pyramid based on multi-scale dense matching relationship. It obtains the initial deformation through point set interpolation, uses the coarse-scale optimized deformation field upsampled as the initial value for fine-scale solution, and recursively obtains the full-resolution initial deformation field stepwise. The optimized output module is used to iteratively optimize the initial deformation field in layers with registration similarity as the convergence criterion, continuously correct deformation deviation, until the full-resolution registration accuracy meets the preset accuracy threshold, and obtain the optimal deformation field. The original floating image is then subjected to pixel-level deformation mapping through a spatial transformation network to output the final registered image.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention employs a preprocessing mechanism combining multi-scale Gaussian pyramids with cross-modal histogram specification to calibrate the grayscale distribution of bimodal images scale by scale, initially eliminating global modal differences at the grayscale level and providing a unified input basis for multi-scale processing throughout the entire process. Furthermore, relying on a shared feature extraction network with domain-discriminative constraints, it integrates shallow texture details and deep semantic structure features, uniformly mapping heterogeneous features to a modality-invariant feature space. This resolves the feature distribution heterogeneity problem caused by different imaging mechanisms at the feature level, effectively addressing the technical pain points of inconsistent multimodal registration benchmarks and insufficient robustness. Furthermore, this invention employs a matching strategy combining superpixel graph structure and graph convolution matching. It compresses matching computational overhead at the superpixel region level, enhances feature discriminative power with neighborhood topological information, and filters out false matches using geometric constraints. This improves matching efficiency while ensuring the accuracy of dense correspondences, overcoming the technical shortcomings of traditional schemes that struggle to balance matching efficiency and dense alignment accuracy, resulting in high false matching rates. Building upon this, the invention recursively solves the initial deformation field along a Gaussian pyramid from coarse to fine. Coarse-scale deformation locks in the global transformation trend, while fine-scale residuals complete local displacement details, avoiding the pitfalls of single-scale solutions easily getting trapped in local optima in large deformation scenarios and ensuring the global continuity of the deformation field. Finally, it iteratively optimizes the deformation field using registration similarity as the convergence criterion, continuously improving alignment accuracy through global similarity back-correction. The spatial transformation network outputs high-quality registration results, achieving multi-scale end-link synergistic gains and systematically improving the accuracy, robustness, and computational efficiency of multimodal image registration. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall method flow of the present invention; Figure 2 This is a schematic diagram of the system module flow of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0019] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0020] Example: Please see Figure 1 An image registration method based on multimodal feature fusion, the specific steps of which include: S1. Obtain the fixed and floating images to be registered, construct multi-scale Gaussian image pyramids respectively, perform cross-modal histogram specification preprocessing on the images at each level of the pyramid, and generate a standardized multi-scale preprocessed image set.

[0021] Furthermore, a multi-scale Gaussian pyramid construction operation is performed on both the input fixed and floating images. Specifically, the original resolution image to be registered is used as the 0th level (finest scale) of the pyramid; a 5×5 scale is used with a standard deviation of [missing information]. The Gaussian convolution kernel performs convolutional blurring on the current level image to suppress high-frequency noise and avoid spectral aliasing caused by downsampling. After blurring, the image is downsampled by a factor of 2, that is, every other pixel along the row and column directions is taken to generate the next level image with a resolution of half that of the previous level. The above iterative operation of "Gaussian blur-2x downsampling" is repeated to construct a Gaussian image pyramid with four scales, and the resolution of each level is half that of the previous level. For example, when the original input image resolution is 512×512, the resolutions of the pyramid levels 0 to 3 are 512×512, 256×256, 128×128, and 64×64, respectively. The larger the level number, the coarser the image scale and the lower the resolution.

[0022] Furthermore, for each level of the Gaussian image pyramid, histogram specification preprocessing is performed on the floating images of the same level, using the gray-level distribution of the fixed image at that level as a reference. The specific steps are as follows: First, the gray-level histogram of the fixed image at the current level is statistically analyzed, and its cumulative gray-level distribution function is calculated. Second, the gray-level histogram of the floating images at the same level is statistically analyzed, and its corresponding cumulative gray-level distribution function is calculated. Third, a gray-level value mapping relationship is established, and the gray-level value of each pixel in the floating image is converted into a target gray-level value that matches the gray-level distribution of the fixed image through the inverse mapping of the cumulative distribution function of the floating image and the forward mapping of the cumulative distribution function of the fixed image. Finally, the gray-level mapping of the entire floating image is completed, resulting in a preprocessed floating image whose gray-level distribution at that level is consistent with that of the fixed image.

[0023] Furthermore, the fixed and floating images after histogram specification processing at all levels of the pyramid are organized according to scale level to form a standardized multi-scale preprocessed image set, which serves as the unified input data source for subsequent steps S2 to S5.

[0024] In this embodiment, step S1 achieves two core technical effects: First, by constructing a 3-5 level multi-scale Gaussian image pyramid through iterative operations of Gaussian blur and double downsampling, a multi-level resolution processing framework from coarse to fine is formed, providing a unified hierarchical processing carrier for the entire process of subsequent feature extraction, graph structure matching, deformation solving, and iterative optimization. Relying on the progressive logic of prioritizing coarse scales, the global deformation trend can be locked first, effectively avoiding the risk of local optima in large deformation scenarios. At the same time, the hierarchical calculation mode significantly reduces the processing complexity of high-resolution images, taking into account both registration robustness and computational efficiency. Second, at each level of the pyramid, the gray-level histogram of the corresponding level of the fixed image is used as a reference to perform a hierarchical histogram specification mapping on the floating image. In the preprocessing stage, the global gray-level distribution difference between the two types of modal images is initially eliminated, the gray-level mapping range is unified, and the gray-level nonlinear shift caused by different imaging mechanisms is weakened. This provides a more consistent input basis for subsequent feature extraction and similarity calculation, reduces the processing pressure of subsequent modal feature alignment, and effectively improves the stability of subsequent matching and deformation optimization.

[0025] S2. Input the preprocessed images of each level into a shared feature extraction network with domain discriminative constraints. Unify the dual-image feature extraction rules through shared network parameters. Rely on domain discriminative constraints to suppress modal feature shift. Fuse the shallow texture features and deep semantic structure features of the image to form heterogeneous features. Map the heterogeneous features to the modality-invariant feature space and output the cross-modal fusion feature maps of each level.

[0026] Furthermore, a parameter-shared dual-branch convolutional feature encoder is constructed. Specifically, a parameter-shared dual-branch convolutional feature encoder is constructed. The encoder includes four feature extraction convolutional blocks connected in series. Each convolutional block consists of two 3×3 convolutional layers, a batch normalization layer, a ReLU activation function, and a 2x downsampling layer. The first two convolutional blocks constitute a shallow texture feature extraction stage, used to capture detailed features such as edges and textures of the image. The latter two convolutional blocks constitute a deep semantic structure feature extraction stage, used to extract high-level features such as regional semantics and global structure of the image. The preprocessed images at the same level of the fixed image and the floating image are simultaneously input into the convolutional feature encoder as two independent branches. The two branches completely reuse the same set of network weight parameters to ensure that the feature extraction operation rules and feature dimension mapping methods of the two types of images are completely consistent. The encoder output fuses shallow texture features and deep semantic structure features through channel splicing operation to form heterogeneous features, generating an initial depth feature map with completely aligned spatial resolution and channel dimensions, so that the features of the two types of heterogeneous images are initially mapped to the same feature space.

[0027] Furthermore, a gradient inversion layer and a binary classification domain discriminator are connected in series at the output of the convolutional feature encoder to construct a domain adversarial constraint module. The domain discriminator is built using a 3-layer fully connected network and finally outputs a classification probability vector with a dimension of 2, which is used to predict the imaging modality category to which the input feature belongs. During the training phase, an adversarial optimization logic based on minimax game is employed: the initial depth feature maps of two images at the same level are passed through a gradient inversion layer and then input into the domain discriminator. The domain discriminator updates its weights with the goal of minimizing the modality classification cross-entropy loss, thereby improving the modality category recognition accuracy. The convolutional feature encoder propagates the negative gradient of the discriminant loss in reverse through the gradient inversion layer, updating its weights with the goal of maximizing the modality classification error. This forces the encoder to weaken modality-specific features such as grayscale and texture style, and strengthen the structural semantic features shared by the two modalities. After multiple rounds of iterative adversarial training, the encoder can extract modality-invariant features with consistent cross-modality distribution, thus mitigating the feature distribution shift caused by different imaging mechanisms at the feature level. During the inference phase, only the convolutional feature encoder is enabled. The domain discriminator and gradient inversion layer no longer participate in the computation. The deep features are directly mapped from the encoder output to the modality-invariant feature space.

[0028] Furthermore, for each level of the Gaussian image pyramid, the aforementioned weight-sharing encoder and domain adversarial constraint mechanism are used to perform feature extraction, generating cross-modal fusion feature maps of fixed and floating images at the corresponding scale. The cross-modal fusion feature maps of all levels are organized according to scale level to form a multi-scale cross-modal fusion feature map set, which serves as the unified input for the subsequent S3 graph convolution matching step.

[0029] In this embodiment, the technical effect of step S2 is as follows: by using a weight-sharing feature extraction network to simultaneously extract shallow texture and deep semantic features of bimodal images, and outputting feature maps of the same dimension with unified weights, the feature dimension difference barrier is eliminated, multi-level feature information is fused, and the adversarial training mechanism based on domain discrimination constraints forces the feature encoder to weaken modality-specific attributes and strengthen cross-modal shared structural features, and uniformly map bimodal features to modality-invariant feature space, thereby eliminating modality distribution differences at the feature level and providing a stable and consistent feature benchmark for subsequent matching.

[0030] S3. Perform superpixel segmentation on the cross-modal fusion feature maps at each level to generate structurally continuous superpixel units. Using superpixels as nodes and adjacency relationships as topological edges, construct the topological graph structures of the two images respectively. Calculate the superpixel feature similarity by combining the topological information through a graph convolutional matching network to generate a dense matching point set and establish multi-scale dense matching relationships.

[0031] Furthermore, for the fixed image cross-modal fusion feature map and the floating image cross-modal fusion feature map at each level of the Gaussian image pyramid, an improved simple linear iterative clustering algorithm is used to perform superpixel segmentation. Specifically, let the total number of pixels in the current level's fusion feature map be... The preset number of superpixels is The initial seed point grid spacing is Arrange the grid on the feature map in a uniform grid pattern. Initial cluster centers; cluster distance metric is constructed using feature similarity and spatial Euclidean distance for any pixel. With cluster center Its joint distance The calculation formula is: In the formula: For pixels With cluster center The cosine distance between feature vectors is used to measure the semantic similarity of features; For pixels With cluster center The Euclidean distance between the spatial coordinates is used to constrain the spatial compactness of superpixels; , These are the feature distance weight and the spatial distance weight, respectively. In this embodiment, we take... , ; It should be noted that the reason for using a positively correlated linear weighting method to construct the joint distance calculation formula is that superpixel segmentation needs to simultaneously satisfy two core objectives: semantic homogeneity and spatial compactness and continuity. Feature distance measures the semantic similarity between pixels, while spatial distance measures the spatial proximity of pixels; their physical meanings and dimensions differ significantly. Using a linear weighted combination allows for flexible allocation of the contribution proportions of the two constraints through weight coefficients, accurately balancing the two segmentation objectives and avoiding segmentation bias caused by a single metric.

[0032] This scheme employs an iterative optimization logic based on simple linear iterative clustering. The linearly weighted joint distance maintains the monotonicity and convexity of the metric. During the iteration process, the cluster center update direction is stable and the convergence speed is fast, avoiding the problems of local oscillations and unstable convergence results caused by nonlinear combinations. Especially at each scale of the multi-scale pyramid, the reproducibility and consistency of the segmentation results can be guaranteed. The weights are set as follows: This is because we prioritize matching accuracy, strengthen semantic constraints, ensure semantic homogeneity within superpixels and boundary fit with structural contours, improve node feature discriminability, adapt to cross-modal characteristics, focus on clustering of common semantic features, weaken interference from single-modal texture differences, and improve cross-modal consistency of segmentation results. At each seed point Within the neighborhood, pixels are searched, and clusters are assigned to them based on the principle of minimum joint distance. The cluster centers are updated to the mean of the feature vectors and the mean of the spatial coordinates of all pixels in that cluster. This iteration is repeated until the offset of the cluster centers is less than a preset threshold (e.g., 1 pixel), completing superpixel segmentation and obtaining a set of superpixels with boundaries that fit the image structure contours and are evenly distributed. It should be noted that the preset threshold represents the maximum offset of the spatial coordinates of the cluster centers before and after the iteration, in pixels. The value is set to balance the requirements of segmentation accuracy and computational efficiency, with a typical range of 0.5-2 pixels. The higher the accuracy requirement, the smaller the value; the higher the computational efficiency requirement, the larger the value. In this embodiment, 1 pixel is used as the preset threshold, that is, when the spatial offset of all cluster centers is less than 1 pixel, the clustering is considered to have converged stably, and the iteration is terminated.

[0033] Furthermore, based on the superpixel set obtained from segmentation, undirected topological graph structures are constructed for both the fixed and floating images. ,in For a set of nodes, The rules for constructing an edge set are as follows: Each individual superpixel region is mapped to a graph node. Perform global average pooling on the feature vectors of all pixels within the superpixel region, and use the pooling result as the initial node feature for that node. The calculation formula is: in, For the first The total number of pixels included within a superpixel. Indicates the first The entire pixel area covered by a superpixel. For the first superpixel in this superpixel region Feature vector of 1 pixel This is a pixel index, representing a single pixel within a superpixel region. For the superpixel index, and ; Traverse the spatial locations of all superpixels and use the eight-neighbor adjacency rule. If two superpixel regions share a pixel boundary, they are determined to have a spatial adjacency relationship. Establish an undirected edge between the corresponding nodes and finally generate undirected topological graph structures corresponding to the fixed image and the floating image at the same level.

[0034] Furthermore, the topological graph structures of two images at the same level are synchronously input into a graph attention convolutional network. The attention mechanism aggregates neighborhood topological information and updates node features. The specific method is as follows: The initial features of all nodes are linearly transformed using a shared weight matrix to obtain the transformed feature vector. For a node and its neighboring nodes, the attention coefficients are calculated to measure the importance of the neighboring nodes to the central node. The attention coefficients of all neighboring nodes are normalized to obtain the final attention weights. The transformed features of all neighboring nodes are aggregated according to the attention weights and then enhanced node features are obtained after nonlinear activation.

[0035] Furthermore, based on the enhanced node feature set, reliable superpixel correspondences are obtained through a dual screening process of feature similarity matching and geometric consistency verification. The specific method is as follows: For the enhanced node feature sets of both fixed and floating images, cosine similarity is calculated by traversing all superpixel node pairs to construct a cross-image node similarity matrix. The similarity calculation formula is as follows: In the formula: The cosine similarity value represents the similarity between two feature vectors. The enhanced feature vector corresponding to a specific superpixel node in a fixed image. The enhanced feature vector corresponding to a superpixel node in the floating image. , respectively, feature vectors , The L2 norm; A bidirectional nearest neighbor matching strategy is adopted: only when a node It is a node The nearest neighbor, and node It is also a node When the nearest neighbor is, Once identified as an initial superpixel matching pair, the initial matching pair set is obtained through filtering. Based on the spatial coordinates of the initial matching pairs, a random sampling consensus algorithm is used to fit the global affine transformation model. The affine transformation matrix is ​​solved by randomly selecting the minimum number of matching pairs, and the reprojection error of all matching pairs after transformation is calculated. Matching pairs with errors less than a preset threshold (e.g., 3 pixels) are identified as inliers. This process is repeated iteratively until the optimal transformation model with the most inliers is obtained. Mismatched pairs with deviations exceeding the threshold are removed, resulting in the optimized superpixel matching correspondence. It should be noted that this threshold is the boundary for determining whether a matching pair conforms to the global geometric transformation model, measured in pixels. It is used to distinguish between valid inliers and mismatched outliers. The threshold is set comprehensively based on image resolution and registration accuracy requirements: a threshold that is too small will result in insufficient valid inliers, making it impossible to fit a stable transformation model; a threshold that is too large will retain a large number of mismatches, weakening the geometric verification effect. The typical value range is 1-5 pixels, corresponding to the allowable range of spatial deviation. In this embodiment, the threshold is set to 3 pixels, meaning that when the Euclidean distance between the reprojected coordinates of a matching pair after affine transformation and its corresponding coordinates is less than 3 pixels, it is identified as a valid inlier; otherwise, it is identified as a mismatched pair and removed.

[0036] Furthermore, the superpixel-level matching relationships are mapped to the pixel dimension to generate a dense set of corresponding pixel points. Specifically, for all pixels within the same superpixel region, the initial displacement vector inherits the spatial displacement amount corresponding to the superpixel matching pair. Bilinear interpolation smoothing is performed on the displacement vector field of the global pixels to eliminate displacement abrupt changes and block effects at the superpixel boundaries, resulting in a spatially continuous and uniformly distributed set of densely corresponding pixel points. The above segmentation, mapping, matching, and mapping process is repeated for all levels of the Gaussian image pyramid to construct dense matching relationships at each scale level, forming a multi-scale dense matching relationship set as the input basis for subsequent deformation solving.

[0037] In this embodiment, the technical effect of step S3 is as follows: superpixel segmentation is performed based on cross-modal fusion features and a topological graph structure is constructed, reducing pixel-level computation to region-level computation and significantly compressing the matching computation. At the same time, topological edges are constructed based on superpixel adjacency relationships, preserving spatial topological information between regions and providing a structured carrier for subsequent feature aggregation. The neighborhood topological information is aggregated through a graph convolutional matching network to enhance the node feature distinguishability. The false matching rate is reduced by combining similarity matching and geometric consistency verification. Finally, the superpixel-level matching is mapped to pixel-level dense correspondence, which improves matching efficiency while ensuring the accuracy of dense alignment.

[0038] S4. Based on the multi-scale dense matching relationship, the deformation is solved step by step from coarse to fine along the Gaussian pyramid. The initial deformation is obtained by point set interpolation. The coarse-scale optimized deformation field is upsampled as the initial value for the fine-scale solution. The full-resolution initial deformation field is obtained by step-by-step recursion.

[0039] Furthermore, the solution is obtained by progressively deforming the Gaussian pyramid from coarse to fine, using the following method: First, the hierarchical order and processing logic of deformation solving are clarified: Taking a Gaussian image pyramid with four scales as an example, level 0 corresponds to the finest scale of the full resolution, and level 3 corresponds to the coarsest scale of the lowest resolution; deformation solving is strictly performed in order from coarse to fine from level 3 to level 0; the deformation solving of each level takes the set of dense pixels corresponding to that level as the core input, firstly obtains the initial deformation components of the current scale through interpolation algorithm, and then completes the hierarchical optimization by combining the deformation benchmark passed from the coarse scale, ensuring that the global deformation trend is locked by the coarse scale and the local details are filled by the fine scale, thus avoiding the local optimum problem in large deformation scenarios from the framework.

[0040] Furthermore, for each pyramid level, using the set of dense pixels corresponding to that level as control points, a thin-plate spline interpolation algorithm is used to fit the nonlinear spatial transformation to generate the initial deformation field for the current level. The specific implementation is as follows: Suppose that at the current level, the dense corresponding point set includes multiple sets of one-to-one matching pixel pairs; in each pair, the pixel on the fixed image side has corresponding planar coordinates, and the matching pixel on the floating image side has corresponding planar coordinates; the difference between the two sets of coordinates is the spatial displacement vector corresponding to the matching point pair, which includes the horizontal displacement and the vertical displacement, and all matching point pairs together serve as control points for deformation fitting. Thin-plate spline interpolation fits a smooth nonlinear spatial transformation by minimizing bending energy. The deformation function in the two-dimensional image consists of two parts: a global affine transformation term and a local radial basis function term. Taking the horizontal displacement field as an example, the global affine transformation term includes three types of parameters: constant coefficients, horizontal coordinate linear coefficients, and vertical coordinate linear coefficients, which are used to characterize the overall linear deformation trend of the image, such as translation, scaling, and shearing. The local radial basis function term is obtained by weighted summation of the radial basis functions corresponding to all control points. Each control point corresponds to an independent weight coefficient, which is used to characterize the local nonlinear deformation around the control point. The input of the radial basis function is the Euclidean distance between the pixel to be calculated and the corresponding control point, and the output value is the product of the square of the distance and the natural logarithm of the distance. The vertical displacement field adopts a function structure that is completely consistent with that of the horizontal direction, and the parameters are solved and the displacement is calculated independently.

[0041] All parameters of the aforementioned deformation function are solved by minimizing the energy function, which consists of two parts: a corresponding point fitting error term and a bending regularization term. The corresponding point fitting error term is used to calculate the sum of squares of the differences between the displacement calculated by the deformation function and the actual displacement of the control points at all control points, measuring the degree of fit between the deformation field and the matching point set. The bending regularization term is used to characterize the overall bending degree of the deformation field, constrain the spatial smoothness of the deformation field, and suppress the occurrence of severe local deformation distortion.

[0042] The two items are weighted and balanced using a regularization coefficient. In this embodiment, the regularization coefficient is set to 0.001 to ensure the smoothness and reasonableness of the deformation field while guaranteeing the fitting accuracy. After minimizing this energy function using a numerical optimization method, the weight coefficients of all affine transformation coefficients and the radial basis functions of each control point can be obtained.

[0043] After solving all the parameters, the coordinates of each pixel in the current layer image are substituted into the deformation functions in the horizontal and vertical directions respectively to calculate the spatial displacement vector corresponding to each pixel. The displacement vectors of all pixels together constitute the initial deformation field of the current layer, which includes horizontal and vertical displacement components.

[0044] Furthermore, a cascaded recursive model of "coarse-scale reference + fine-scale residual" is finally adopted to achieve layer-by-layer refinement of the deformation field from low resolution to high resolution. The specific process is as follows: For the coarsest scale of the pyramid, without input from the upper coarse scale, the optimized deformation field of this level is obtained directly through the thin plate spline interpolation algorithm mentioned above, and serves as the initial reference for cross-scale transfer. For all fine-scale levels except the coarsest, the processing is performed in three steps: First, baseline deformation upsampling: The optimized deformation field of the previous adjacent coarse-scale level is upsampled to the spatial resolution of the current level using bilinear interpolation to obtain the baseline deformation field of the current level. This baseline deformation field carries the global coarse-grained deformation trend. Second, residual corresponding point calculation: For the dense corresponding point set of the current level, each fixed image-side control point is substituted into the baseline deformation field to obtain the predicted coordinates under the baseline displacement. The residual displacement of the control points is obtained by subtracting the predicted coordinates from the coordinates of the corresponding floating image-side matching points, forming the residual corresponding point set. Third, residual fitting and superposition: Using the residual corresponding point set as control points, the residual deformation field of the current level is fitted using thin-plate spline interpolation. The residual deformation field is then superimposed pixel-by-pixel with the baseline deformation field to obtain the optimized deformation field of the current level. The above benchmark transfer, residual fitting and superposition operations are repeated along the pyramid from coarse to fine. After the deformation solution of all levels is completed, the optimized deformation field corresponding to the finest scale is output as the initial deformation field of full resolution.

[0045] In this embodiment, the technical effect of step S4 is as follows: Deformation solution is performed step by step from coarse to fine along the Gaussian pyramid. The dense matching point set is used as the control point to fit the nonlinear spatial transformation through interpolation. The coarse scale prioritizes locking the global deformation trend, and the fine scale uses the deformation field sampled from the coarse scale as the initial reference. The hierarchical optimization is completed through residual deformation superposition, which effectively avoids the local optimum problem in large deformation scenarios. It simultaneously improves the global consistency and local detail accuracy of deformation fitting. Furthermore, relying on the multi-scale step-by-step recursive solution mechanism, it ensures the spatial continuity and smooth transition of the deformation field across scales. The output full-resolution initial deformation field has both a stable global transformation basis and rich local deformation details.

[0046] S5. Using registration similarity as the convergence criterion, the initial deformation field is iteratively optimized in layers, and the deformation deviation is continuously corrected until the full-resolution registration accuracy meets the preset accuracy threshold. The optimal deformation field is obtained, and the original floating image is mapped at the pixel level through a spatial transformation network to output the final registered image.

[0047] Furthermore, this step uses normalized mutual information as the multimodal registration similarity metric, and performs hierarchical iterative optimization from coarse to fine along the Gaussian pyramid. Unsupervised reverse correction is used to eliminate residual errors from previous steps. The specific implementation process is as follows: Normalized mutual information value is selected as the cross-modal registration similarity index. This index calculates the correlation based on the gray-level statistical distribution of two images, does not rely on the linear correspondence of gray-level values, and has strong adaptability to the nonlinear differences in gray levels across different imaging modalities. Let the fixed image at the current scale be... The floating image after deformation field resampling is The formula for calculating normalized mutual information is: in, , These are the grayscale edge entropy values ​​for the fixed image and the deformed floating image, respectively. The joint gray-level entropy of the two images. For fixed image With the floating image after deformation The normalized mutual information value between the two images; the larger the normalized mutual information value, the stronger the gray-level statistical correlation and the higher the spatial alignment between the two images. The optimization sequence is explicitly mapped one-to-one with the Gaussian pyramid levels: progressing layer by layer from the coarsest scale to the finest scale at full resolution. First, the initial full-resolution deformation field output in step S4 is bilinearly downsampled and mapped to the resolution of each corresponding level of the pyramid, obtaining the initial deformation field for each scale, which serves as the initial baseline for iterative optimization at that level. This initialization method fully transmits the deformation trend from the previous solution to the optimization stage, significantly accelerating the optimization convergence speed and avoiding optimization oscillations caused by random initialization.

[0048] Furthermore, for each optimization scale, in-scale closed-loop iterative optimization is performed, the specific process of which is as follows: Based on the deformation field at the current scale, pixel coordinate mapping and grayscale resampling are performed on the preprocessed floating image at this level to obtain the deformed floating image. The normalized mutual information value between the deformed floating image and the fixed image at the same level is calculated, and an optimization loss function is constructed with the goal of minimizing the loss function (i.e., maximizing the mutual information value). The gradient descent algorithm is used to calculate the gradient of the loss function with respect to the pixel displacement of the deformation field, and the displacement parameters of the deformation field are iteratively corrected along the negative gradient direction, with the iteration step size set to [value missing]. Record the mutual information value after each iteration. When the change in mutual information between two consecutive iterations is less than the preset single-scale convergence threshold, the result is considered a convergence value. When the current scale optimization is converged, output the deformation field optimized at the current scale. After the current scale optimization converges, the optimized deformation field at that scale is upsampled to the spatial resolution of the next adjacent fine scale using bilinear interpolation, serving as the initial deformation field for the next fine scale iteration. This "intra-scale iterative optimization—cross-scale upsampling transfer" loop is repeated until the iteration progresses to the finest scale at full resolution. When the iterative optimization at the full resolution scale meets a preset accuracy threshold, the global optimization process terminates, and the final optimal deformation field is output. It should be noted that this accuracy threshold is set comprehensively based on the modal differences of the image to be registered, the registration accuracy requirements of the actual scene, and the image resolution: the greater the modal differences and the lower the accuracy requirement, the smaller the threshold value; conversely, the higher the accuracy requirement, the larger the threshold value. For general multimodal image registration scenarios, the typical range for this threshold is 1.7-1.9; the higher the threshold, the higher the final registration and alignment accuracy, but the time required for iterative optimization will increase accordingly.

[0049] In this embodiment, for the high-precision multimodal registration requirement, the preset accuracy threshold is set to 1.85. When the normalized mutual information value of the fixed image and the deformed floating image reaches or is not lower than 1.85 during the full-resolution scale iteration process, and the single-scale convergence condition is met, the global registration accuracy is determined to be up to standard, the optimization process is terminated and the optimal deformation field is output.

[0050] Furthermore, a parameter-free spatial transformation network is used to perform the final deformation mapping of the original floating image, generating a high-quality registered image. The specific implementation is as follows: A lightweight spatial transformation network is built, which includes two core modules: a coordinate grid generator and a bilinear sampler. The network has no parameters to be trained and only performs differentiable spatial transformation and resampling operations. It can be embedded into the overall registration process to achieve end-to-end computation. Using the full-resolution spatial coordinates of a fixed image as the target grid, for any pixel coordinate in the target grid... , To fix the horizontal and vertical coordinates of standard pixels within the target grid of the image, the displacement vector corresponding to that pixel is queried through the optimal deformation field. , Given the deformation of pixels in the horizontal and vertical directions, calculate their mapped source coordinates on the original floating image. , To obtain the horizontal and vertical coordinates of the source pixel in the original floating image after the optimal deformation field reverse mapping solution, the reverse mapping mechanism is used to traverse all pixels in the target image. This ensures that each pixel in the target image can be matched with a unique source image coordinate, effectively avoiding the pixel holes and pixel overlap defects that are common in forward mapping algorithms.

[0051] Since the source coordinates are usually sub-pixel level non-integer coordinates, a bilinear interpolation algorithm is used to calculate the gray value at that location: take four adjacent integer pixels around the source coordinates, and calculate the gray value of the four pixels by weighting the fractional part of the horizontal and vertical coordinates to obtain the final gray value corresponding to the target coordinates. Bilinear interpolation can ensure that the gray value of the output image is continuous and smooth, avoiding the jaggedness and block effect caused by coordinate mapping. Traverse all pixel coordinates of the target grid, perform coordinate mapping and grayscale resampling in sequence, and finally generate and output the final registered image that is aligned with the fixed image space, has continuous grayscale, and clear edges.

[0052] In this embodiment, the technical effect of step S5 is as follows: a multi-scale hierarchical iterative optimization mechanism is adopted, and the deformation field is corrected step by step from coarse to fine with the registration similarity as the convergence criterion. The alignment effect is back-verified and optimized through global gray-level statistical indicators, which effectively eliminates the residual errors of previous feature matching and deformation solution. At the same time, it avoids the defect of single-scale optimization being prone to getting trapped in local optima, and takes into account both optimization convergence efficiency and final registration accuracy. Finally, pixel-level deformation mapping is performed through spatial transformation network, and gray-level resampling is completed by combining bilinear interpolation to ensure the gray-level continuity and spatial position accuracy of the mapped image. This achieves a complete closed loop from deformation field optimization to final registered image output, and outputs high-quality registration results with clear edges and accurate alignment.

[0053] Please see Figure 2 The present invention also provides an image registration system based on multimodal feature fusion, the system being used to execute the image registration method based on multimodal feature fusion described in any one of the above claims, comprising: The hierarchical processing module is used to acquire the fixed and floating images to be registered, construct multi-scale Gaussian image pyramids respectively, perform cross-modal histogram specification preprocessing on the images at each level of the pyramid, and generate a standardized multi-scale preprocessed image set. The cross-modal fusion module is used to input preprocessed images from each level into a shared feature extraction network with domain discriminative constraints. By unifying the dual-image feature extraction rules through shared network parameters, it suppresses modal feature shift by relying on domain discriminative constraints, fuses shallow texture features and deep semantic structure features of the image to form heterogeneous features, maps the heterogeneous features to the modality-invariant feature space, and outputs cross-modal fusion feature maps of each level. The graph structure matching module is used to perform superpixel segmentation on cross-modal fusion feature maps at various levels, generate structurally continuous superpixel units, construct the topological graph structure of two images with superpixels as nodes and adjacency relationships as topological edges, calculate the superpixel feature similarity by combining topological information through a graph convolutional matching network, generate a dense matching point set, and establish multi-scale dense matching relationships. The deformation field solution module is used to solve the deformation stepwise from coarse to fine along the Gaussian pyramid based on multi-scale dense matching relationship. It obtains the initial deformation through point set interpolation, uses the coarse-scale optimized deformation field upsampled as the initial value for fine-scale solution, and recursively obtains the full-resolution initial deformation field stepwise. The optimized output module is used to iteratively optimize the initial deformation field in layers with registration similarity as the convergence criterion, continuously correct deformation deviation, until the full-resolution registration accuracy meets the preset accuracy threshold, and obtain the optimal deformation field. The original floating image is then subjected to pixel-level deformation mapping through a spatial transformation network to output the final registered image.

[0054] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0055] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0056] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0057] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A multi-modal feature fusion based image registration method, characterized in that, The specific steps include: S1. Obtain the fixed and floating images to be registered, construct multi-scale Gaussian image pyramids respectively, perform cross-modal histogram specification preprocessing on the images at each level of the pyramid, and generate a standardized multi-scale preprocessed image set; S2. Input the preprocessed images of each level into a shared feature extraction network with domain discriminative constraints. Unify the dual-image feature extraction rules through the shared network parameters. Rely on the domain discriminative constraints to suppress modal feature shift. Fuse the shallow texture features and deep semantic structure features of the image to form heterogeneous features. Map the heterogeneous features to the modality-invariant feature space and output the cross-modal fusion feature maps of each level. S3. Perform superpixel segmentation on the cross-modal fusion feature maps at each level to generate structurally continuous superpixel units. Using superpixels as nodes and adjacency relationships as topological edges, construct the topological graph structures of the two images respectively. Calculate the superpixel feature similarity by combining the topological information through a graph convolutional matching network to generate a dense matching point set and establish multi-scale dense matching relationships. S4. For the coarsest level of the pyramid, the dense matching point set of this level is used as the deformation fitting control point. The thin plate spline interpolation algorithm is used to fit the nonlinear spatial transformation relationship between the fixed image and the floating image, solve the spatial displacement vector of each pixel, and generate the initial deformation field of the current level. The initial deformation field is used as the optimized deformation field for the coarsest level. For each fine-scale level except the coarsest one, the optimized deformation field of the previous coarsest scale is upsampled to the current scale resolution using bilinear interpolation, serving as the initial reference for the deformation solution at this level. The residual deformation field is solved by combining the dense matching point set at this level, and the residual deformation field is superimposed with the initial reference pixel by pixel, iteratively optimizing to obtain the optimized deformation field of the current level. After completing the iterative solution of deformation at all levels of the pyramid, the optimized deformation field corresponding to the finest scale is output as the final full-resolution initial deformation field. S5. The full-resolution initial deformation field is downsampled and matched to the coarsest level resolution of the pyramid to obtain the initial deformation reference of the coarsest level iterative optimization. The coarsest initial deformation reference is used as the initial deformation field for this level of iteration; Starting from the coarsest level, for the current level, the preprocessed floating image at that level is resampled pixel by pixel based on the initial deformation field of the current level iteration. The normalized mutual information value between the resampled floating image and the fixed image at the same level is calculated. With maximizing the normalized mutual information value as the optimization objective, the deformation field parameters of the current scale are continuously iteratively corrected using the gradient descent algorithm until the change in mutual information in a single iteration is less than the preset single-scale convergence threshold. At this point, the current scale optimization is considered complete, and the optimized deformation field of this level is output. The optimized deformation field of this level is then upsampled to the next fine scale using bilinear interpolation as the initial deformation field for the next scale iteration. This process is repeated to complete the intra-scale optimization and cross-scale deformation transfer. When the iterative optimization progresses to the full-resolution level and the normalized mutual information value meets the preset accuracy threshold, the optimized deformation field corresponding to the full-resolution level is output as the optimal deformation field. Superpixel segmentation is performed on cross-modal fusion feature maps at each level to generate structurally continuous superpixel units. The method used is as follows: Pixel clustering segmentation is performed on the cross-modal fusion feature maps at each level. With texture similarity threshold and spatial distance threshold as constraints, the pixels of the fusion feature map are clustered and aggregated. Pixels with similarity and spatial distance both less than the corresponding threshold are aggregated into the same superpixel unit to complete global clustering segmentation and finally obtain a set of superpixels with continuous structure and uniform distribution. Using superpixels as nodes and adjacency relationships as topological edges, the topological graph structures of the two images are constructed respectively. The method used is as follows: All the independent superpixel units obtained from the segmentation are defined as feature nodes of the topological graph. The spatial positional relationship of all superpixel units is traversed to determine whether there are adjacent contact regions between different superpixel units. A connection line is established between two feature nodes that have a spatial adjacency relationship, and this connection line is the topological edge. The adjacency traversal and topological edge connection of all feature nodes are completed in sequence to form undirected topological graph structures that are adapted to fixed images and floating images respectively.

2. The image registration method based on multi-modal feature fusion according to claim 1, characterized in that, Multi-scale Gaussian image pyramids were constructed, and cross-modal histogram specification preprocessing was performed on the images at each level of the pyramids to generate a standardized multi-scale preprocessed image set. The method used was as follows: Gaussian kernel convolution blurring is applied to both fixed and floating images, followed by a 2x downsampling operation. The blurring and downsampling process is repeated iteratively to construct a pyramid of 3-5 level multi-scale Gaussian images, with the resolution of each level of the pyramid being 1 / 2 of the previous level. For each level of the pyramid image, the grayscale histogram of the corresponding fixed image at the same level is used as a reference template to perform histogram specification mapping on the floating image at the same level, calibrate the grayscale distribution of the floating image, make the grayscale distribution of the two types of images tend to be consistent, eliminate the global grayscale difference of cross-modal images, and obtain the preprocessed image of each level. The preprocessed images from all levels are integrated to form a multi-scale preprocessed image set.

3. The image registration method based on multimodal feature fusion according to claim 2, characterized in that, The specific logic of step S2 is as follows: Construct a parameter-sharing dual-branch shared feature extraction network. Preprocessed images at each level of both the fixed and floating images are synchronously input into the shared feature extraction network. The two branches completely reuse the same set of network parameters to ensure that the operation rules and weight parameters for feature extraction of the two types of images are completely consistent. The network has a built-in domain discrimination constraint module, which identifies and suppresses feature distribution shifts caused by different imaging modalities through domain adversarial training, and removes modality-specific interference features. Simultaneously extract shallow edge texture features and deep semantic structure features of the image, fuse and encode multi-dimensional features to form heterogeneous features, and uniformly map the heterogeneous features to the modality-invariant common feature space to generate cross-modality fusion feature maps that are aligned at each level and are modality-independent.

4. The image registration method based on multimodal feature fusion according to claim 3, characterized in that, The method used is to calculate superpixel feature similarity by combining graph convolutional matching networks with topological information, generate a dense set of matching points, and establish multi-scale dense matching relationships. The topological graph structures of fixed and floating images at the same level are synchronously input into a graph convolutional matching network. The network aggregates the neighborhood topological features of each superpixel node using a graph attention mechanism, iteratively updates the feature vectors of each node, and obtains an enhanced node feature set that incorporates global structural information. For the enhanced node feature sets of the two images, cosine similarity is calculated for each pair of superpixel nodes to construct a cross-graph node similarity matrix. An initial superpixel matching pair is selected using a bidirectional nearest neighbor matching strategy. Based on the spatial coordinate information of the initial superpixel matching pairs, a random sampling consensus algorithm is used to fit a global geometric transformation model. The geometric deviation of each matching pair relative to the model is statistically analyzed to obtain the superpixel matching correspondence. The superpixel-level matching relationship is mapped to the pixel dimension, allowing all pixels within the same superpixel region to inherit the matching attribute of that superpixel. Bilinear interpolation is applied to the pixel-level matching results for smoothing, generating a uniform and dense set of matching points at that level. This process is repeated at all scales to construct dense matching relationships.

5. The image registration method based on multimodal feature fusion according to claim 1, characterized in that, The floating image is subjected to pixel-level deformation mapping using a spatial transformation network to output the final registered image. The method used is as follows: A spatial transformation network adapted to pixel-level spatial mapping is constructed, and the optimal deformation field obtained by iterative optimization is used as the network transformation parameter input. Based on the pixel displacement information of the optimal deformation field, spatial offset mapping is performed on all pixel coordinates of the floating image. The bilinear interpolation algorithm is used to perform grayscale resampling and completion on the mapped pixels to complete the global deformation alignment of the floating image and output the final registered image.

6. An image registration system based on multimodal feature fusion, characterized in that, The system is used to execute an image registration method based on multimodal feature fusion as described in any one of claims 1-5, including: The hierarchical processing module is used to acquire the fixed and floating images to be registered, construct multi-scale Gaussian image pyramids respectively, perform cross-modal histogram specification preprocessing on the images at each level of the pyramid, and generate a standardized multi-scale preprocessed image set. The cross-modal fusion module is used to input preprocessed images from each level into a shared feature extraction network with domain discriminative constraints. By unifying the dual-image feature extraction rules through shared network parameters, it suppresses modal feature shift by relying on domain discriminative constraints, fuses shallow texture features and deep semantic structure features of the image to form heterogeneous features, maps the heterogeneous features to the modality-invariant feature space, and outputs cross-modal fusion feature maps of each level. The graph structure matching module is used to perform superpixel segmentation on cross-modal fusion feature maps at various levels, generate structurally continuous superpixel units, construct the topological graph structure of two images with superpixels as nodes and adjacency relationships as topological edges, calculate the superpixel feature similarity by combining graph convolutional matching network with topological information, generate dense matching point sets, and establish multi-scale dense matching relationships. The deformation field solution module is used for the coarsest level of the pyramid. It uses the dense matching point set of this level as the deformation fitting control point, and adopts the thin plate spline interpolation algorithm to fit the nonlinear spatial transformation relationship between the fixed image and the floating image, solves the spatial displacement vector of each pixel, and generates the initial deformation field of the current level. The initial deformation field is used as the optimized deformation field for the coarsest level. For each fine-scale level except the coarsest one, the optimized deformation field of the previous coarsest scale is upsampled to the current scale resolution using bilinear interpolation, serving as the initial reference for the deformation solution at this level. The residual deformation field is solved by combining the dense matching point set at this level, and the residual deformation field is superimposed with the initial reference pixel by pixel, iteratively optimizing to obtain the optimized deformation field of the current level. After completing the iterative solution of deformation at all levels of the pyramid, the optimized deformation field corresponding to the finest scale is output as the final full-resolution initial deformation field. The optimized output module is used to downsample and match the full-resolution initial deformation field to the coarsest level resolution of the pyramid, so as to obtain the initial deformation reference of the coarsest level iterative optimization. The coarsest initial deformation reference is used as the initial deformation field for this level of iteration; Starting from the coarsest level, for the current level, the preprocessed floating image at that level is resampled pixel by pixel based on the initial deformation field of the current level iteration. The normalized mutual information value between the resampled floating image and the fixed image at the same level is calculated. With maximizing the normalized mutual information value as the optimization objective, the deformation field parameters of the current scale are continuously iteratively corrected using the gradient descent algorithm until the change in mutual information in a single iteration is less than the preset single-scale convergence threshold. At this point, the current scale optimization is considered complete, and the optimized deformation field of this level is output. The optimized deformation field of this level is then upsampled to the next fine scale using bilinear interpolation as the initial deformation field for the next scale iteration. This process is repeated to complete the intra-scale optimization and cross-scale deformation transfer. When the iterative optimization progresses to the full-resolution level and the normalized mutual information value meets the preset accuracy threshold, the optimized deformation field corresponding to the full-resolution level is output as the optimal deformation field. Superpixel segmentation is performed on cross-modal fusion feature maps at each level to generate structurally continuous superpixel units. The method used is as follows: Pixel clustering segmentation is performed on the cross-modal fusion feature maps at each level. With texture similarity threshold and spatial distance threshold as constraints, the pixels of the fusion feature map are clustered and aggregated. Pixels with similarity and spatial distance both less than the corresponding threshold are aggregated into the same superpixel unit to complete global clustering segmentation and finally obtain a set of superpixels with continuous structure and uniform distribution. Using superpixels as nodes and adjacency relationships as topological edges, the topological graph structures of the two images are constructed respectively. The method used is as follows: All the independent superpixel units obtained from the segmentation are defined as feature nodes of the topological graph. The spatial positional relationship of all superpixel units is traversed to determine whether there are adjacent contact regions between different superpixel units. A connection line is established between two feature nodes that have a spatial adjacency relationship, and this connection line is the topological edge. The adjacency traversal and topological edge connection of all feature nodes are completed in sequence to form undirected topological graph structures that are adapted to fixed images and floating images respectively.

Images