Nonlinear image registration method and system based on attention fusion

By combining a bi-branch multi-scale feature extraction network and the Transformer attention mechanism, the problems of poor nonlinear deformation adaptation, insufficient cross-image feature fusion, low computational efficiency, and easy folding of deformation fields in existing image registration techniques are solved, achieving high-precision and high-robust image registration.

CN121458770AActive Publication Date: 2026-02-03SICHUAN SHUJU INTELLIGENT MFG TECH CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202512035244.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-12-31
Publication Date
2026-02-03
Estimated Expiration
2045-12-31

AI Technical Summary

Technical Problem

Existing image registration techniques have shortcomings in nonlinear deformation adaptation, cross-image feature fusion, computational efficiency, and easy folding of deformation fields, making it difficult to meet the requirements of high-precision and high-robust image registration.

Method used

We employ a nonlinear image registration method based on attention fusion. This method uses a dual-branch multi-scale feature extraction network combined with the Transformer attention mechanism to perform feature fusion, generating a dense deformation field. We then use multiple loss functions to optimize the deformation field, avoiding folding phenomena and achieving global-local feature linkage and efficient registration.

Benefits of technology

It improves the adaptability to complex nonlinear deformations, enhances cross-image feature fusion, improves registration accuracy and computational efficiency, avoids topological distortion of the deformation field, and adapts to image registration tasks in different scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121458770A_ABST
    Figure CN121458770A_ABST
Patent Text Reader

Abstract

The invention discloses a nonlinear image registration method and system based on attention fusion, can realize high-precision image registration under complex nonlinear deformation, and belongs to the technical field of computer vision and image processing. The problems of poor non-linear deformation adaptation, insufficient cross-image feature fusion, low calculation efficiency and easy folding of a deformation field in the prior art are solved. The method comprises the following steps: preprocessing an input fixed image and a floating image to be registered; and adopting a double-branch multi-scale feature extraction network to respectively register the pre-processed fixed image and floating image. The method is used for industrial visual detection scenes.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The application relates to a nonlinear image registration method and system based on attention fusion, which is used in an industrial visual detection scene and can realize high-precision image registration under complex nonlinear deformation and belongs to the technical field of computer vision and image processing. BACKGROUND

[0002] Image registration is a key technology for mapping two (or more) images obtained at different times, from different perspectives or by different sensors to the same coordinate space, aligning corresponding target positions, and playing an irreplaceable role in medical diagnosis, remote sensing analysis, computer vision and other fields.

[0003] Existing image registration technologies mainly fall into three categories:

[0004] Traditional feature matching-based methods: such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features) and the like, which extract local feature points from images, match the feature points and calculate a transformation matrix (such as a homography matrix or an affine matrix) to achieve registration. However, such methods have poor robustness for nonlinear deformation (such as tissue deformation in medical images, terrain undulation in remote sensing images and changes in the shooting distance of an industrial line scan camera), and are prone to feature point loss in low-texture areas (such as uniform backgrounds and smooth part surfaces), leading to registration failure.

[0005] Traditional deep learning-based methods: such as registration models based on U-Net, V-Net and other convolutional neural networks (CNNs), which directly predict a dense deformation field or transformation parameters through a CNN. However, the receptive field of a CNN is limited, making it difficult to capture global feature correlations between images, and most models only use convolutional layers for feature fusion, lacking fine-grained cross-image (fixed image and floating image) interaction and being insufficient in adapting to complex deformation.

[0006] Deformation field optimization-based methods: some methods directly predict a dense deformation field of the original image size to achieve registration, but the number of parameters of a dense deformation field is huge (such as about 130,000 x / y offset values for a 256x256 image), leading to a sharp increase in model calculation and difficulty in training convergence. Meanwhile, existing methods generally lack effective deformation field regularization mechanisms, and are prone to “folding phenomena” (the Jacobian determinant of the deformation field is less than or equal to 0, causing the topological structure of the corresponding image region to be distorted), which seriously affects registration accuracy.

[0007] In summary, existing technologies have four major problems: poor adaptation to nonlinear deformation, insufficient cross-image feature fusion, low computational efficiency and easy folding of deformation fields, making it difficult to meet the demand for high-precision and high-robustness image registration (such as deformation alignment in industrial images and dynamic scene stitching in remote sensing images). Therefore, there is an urgent need for an image registration scheme that can realize global-local feature linkage, efficiently generate reasonable deformation fields and balance accuracy and efficiency. Summary of the Invention

[0008] To address the problems mentioned above, the present invention aims to provide a nonlinear image registration method and system based on attention fusion, which solves the problems of poor nonlinear deformation adaptation, insufficient cross-image feature fusion, low computational efficiency, and easy folding of deformation fields in existing technologies.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A nonlinear image registration method based on attention fusion includes the following steps:

[0011] Step 1: Preprocess the input fixed and floating images to be registered;

[0012] Step 2: Use a dual-branch multi-scale feature extraction network to register the preprocessed fixed image and floating image respectively.

[0013] Furthermore, the specific steps of step 1 are as follows:

[0014] Step 1.1: Normalize the input fixed image and floating image, specifically by normalizing the pixel values ​​of the fixed image and floating image to the [0,1] range;

[0015] Step 1.2: If the normalized fixed image and the floating image have different resolutions, scale the normalized floating image to the same width and height as the fixed image.

[0016] Step 1.3: Use the Sobel operator to extract the texture information of the fixed and floating images after they are of uniform size, and obtain the preprocessed fixed and floating images.

[0017] Furthermore, the dual-branch multi-scale feature extraction network employs feature extraction networks with identical structures to extract features from the preprocessed fixed image and floating image respectively, a feature fusion layer that fuses the features output by the two feature extraction networks, a deformation layer that deforms the fused features output by the feature fusion layer to obtain a dense deformation field, and a registration layer that registers the fixed image and floating image based on the output of the deformation layer.

[0018] The feature extraction network includes a first downsampling layer, a second downsampling layer, a third downsampling layer, a fourth downsampling layer, and a fifth downsampling layer, which sequentially downsample the preprocessed fixed or floating image, respectively. The features of the fixed and floating images output by each layer are as follows: , , , ,and ;

[0019] The feature fusion layer includes two deepest layer features output by the feature extraction network The self-attention layer is implemented by a Transformer attention mechanism to realize cross-image feature fusion, and the cross-attention layer is used to cross-attention feature fusion on the output of the self-attention layer.

[0020] Further, the self-attention layer performs cross-attention feature fusion on the deepest layer features of the fixed image and the floating image output by the fifth down-sampling layer of the two feature extraction networks And respectively Operation flattening into a fixed image sequence And a floating image sequence ;

[0021] The correlation weight of the pixels in the fixed image sequence and the floating image sequence is calculated again, and the calculation formula is:

[0022]

[0023] Wherein, represents a matrix with a shape of , represents a matrix with a shape of , represents the transpose matrix of the matrix, with a shape of , represents a matrix with a shape of represents the dimension of , , and represents the sequence length;

[0024] The self-attention enhanced fixed image sequence And the floating image sequence are output based on the correlation weight, wherein, represents the batch size of the fixed image sequence or the floating image sequence, represents the dimension size of the fixed image sequence or the floating image sequence, represents the length of the fixed image sequence or the floating image sequence;

[0025] The cross-attention layer cross-fuses the fixed image sequence and the floating image sequence to obtain a fixed image feature fusion sequence And a floating image feature fusion sequence , and then Operation to obtain the fixed image fusion feature And floating image fusion features , specifically: As , As And The cross attention matrix is calculated by inputting the cross attention attention layer, and then the features in the floating image sequence are weighted and aggregated based on the cross attention matrix to generate a fixed image feature fusion sequence related to the fixed image sequence At the same time, the features in the fixed image sequence are weighted and aggregated to generate a floating image feature fusion sequence related to the floating image sequence Finally, after and are operated, the fixed image fusion features and the floating image fusion features are output. .

[0026] Further, the deformation layer initializes the displacement field with all 0s , is the batch size, 2 is the number of channels, and are the width and height of the fifth layer feature map, and the two channels of the displacement field correspond to direction initial offset and direction initial offset , obtaining the spatial dimension is completely consistent with the fixed image fusion features and the floating image fusion features, wherein all 0 values represent the initial assumption that the fixed image fusion features and the floating image fusion features have no spatial offset.

[0027] Resample the floating image fusion features to align them to the spatial coordinate system of the fixed image fusion features , output , the specific steps are as follows:

[0028] Source position calculation: obtain the pixel position of the fixed image fusion features through the initial displacement field , and calculate the source position in the floating image fusion features , the calculation formula is as follows:

[0029]

[0030] Border processing: if Exceeding floating image fusion feature Border, specifically Or , Or , filled with 0 value, , The initial displacement field The displacement amount at each pixel position in the image;

[0031] Bilinear interpolation: after calculating the source position, the resampled feature value is obtained by weighting the 4-neighborhood pixels :

[0032]

[0033] Where: , , Downward rounding, , covering the 4 integer pixel positions around the target point, interpolation weight , The weight sum is 1, and the output resampled feature value is The dimension of the fixed image fusion feature is exactly the same as that of the fixed image fusion feature;

[0034] Calculate the resampled feature The feature of each position in the image is fused with the fixed image feature The cost between the features in the neighborhood corresponding to the position, and the barycenter position of the cost in the field is the coarse deformation vector of the pixel position in the feature layer, and the collection of deformation vectors of all pixel positions is the coarse deformation field ; Specifically:

[0035] For each pixel position in the resampled feature value , set a 3x3 neighborhood at the corresponding position of the fixed image fusion feature , the neighborhood radius , covering the offset , select the cosine distance of the feature to measure the similarity of the feature , the calculation formula is:

[0036]

[0037] Based on the similarity , the cost in the neighborhood is weighted for each pixel, and the temperature parameter is introduced to calculate the optimal offset by "cost barycenter", that is, to calculate the coarse deformation vector , ​), the formula is:

[0038]

[0039] The set of rough deformation vectors of all pixel points is a rough deformation field ;

[0040] The rough deformation field is convolved, and the deformation field output by the two layers of convolution modules connected in turn and the upsampling layer is used for fine-tuning as the initial deformation field , wherein the first layer of convolution module includes convolution layer, normalization layer and activation layer connected in turn, the second layer of convolution module only contains a convolution layer, and the upsampling layer doubles the size and value of the deformation field output by the second convolution module;

[0041] The initial deformation field is taken as a new initial deformation field, and the optimized deformation field is calculated based on the output of the fourth downsampling layer, to obtain an 8-fold down-sampled sparse deformation field , specifically: using to resample to obtain , the similarity between and is calculated again , and based on , the rough deformation field is calculated, and finally the convolution layer is optimized , to obtain the 8-fold down-sampled sparse deformation field ;

[0042] The free-form deformation algorithm is used to interpolate the sparse deformation field into a dense deformation field consistent with the size of the input floating image , specifically: the free-form deformation algorithm defines a three-dimensional parameterized grid through the grid points of the sparse control points, i.e. the sparse deformation field , and calculates the offset of each element in the dense deformation field through bilinear interpolation of each element in the sparse deformation field ;

[0043] The cubic B-spline basis function is defined as:

[0044]

[0045] Based on the defined cubic B-spline basis function , for the pixel of the input floating image, the offset of the pixel in the direction and the direction is:

[0046]

[0047] wherein, is a floor operation, respectively, and are two channels of dense deformation field , each pixel corresponds to and direction offset, the shape size of dense deformation field is , represents the l-th spline basis function.

[0048] Further, the registration layer samples the floating image through the dense deformation field to obtain the registered image aligned with the fixed image , specifically: using bilinear sampling or cubic interpolation sampling, according to the dense deformation field to remap the pixels of the floating image , for each pixel of the fixed image, its corresponding position in the floating image is , ; if the mapping position exceeds the boundary of the floating image , zero padding or mirror padding processing is adopted, and finally the registered image is output, otherwise, the registered image is directly output.

[0049] Further, the dual-branch multi-scale feature extraction network adopts multiple loss functions, specifically three types of loss functions are jointly optimized, including normalized cross-correlation loss , deformation field difference loss and anti-collapse loss ;

[0050] The normalized cross-correlation loss measures the similarity between the registered image and the fixed image , and the formula is:

[0051]

[0052] wherein, is the covariance, is the standard deviation, is the batch size, the smaller, the more similar the registered image and the fixed image;

[0053] The deformation field difference loss measuring the predicted deformation field i.e. the difference with the ground truth deformation field using loss, formula is:

[0054]

[0055] wherein, respectively represent the input image pair number, the height and width of the input floating image and the fixed image of the double-branch multi-scale feature extraction network;

[0056] anti-fold loss formula is:

[0057]

[0058] wherein, is the Jacobian matrix of when the penalty is triggered, and is the height and width of the input fixed image or floating image; Total loss function: weighted sum of three kinds of loss, formula is:

[0059]

[0060]

[0061] wherein, , , is a hyperparameter.

[0062] A nonlinear image registration system based on attention fusion includes a computer program, a processor for processing the computer program, and the processor executes the computer program to realize the steps of the nonlinear image registration method based on attention fusion.

[0063] Compared with the prior art, the present application has the following advantages:

[0064] The present application overcomes the four core problems of the prior art: poor nonlinear deformation adaptation, insufficient cross-image feature fusion, low computational efficiency, and easy deformation field folding, meets the demand for high-precision and high-robustness image registration, and can realize global-local feature linkage, efficient generation of reasonable deformation field, and image registration that balances precision and efficiency, which is embodied in the following aspects:

[0065] I. Stronger ability to adapt to complex nonlinear deformation: the Transformer self-attention mechanism in the present application can capture the global-local feature correlation of the image, and the cross-attention can realize cross-image fine-grained interaction. ​

[0066] II. Higher registration accuracy: This invention uses a dual-branch multi-scale feature extraction network to obtain multi-scale feature fusion to supplement spatial details, and pyramid level optimization to reduce offset error;

[0067] Third, superior computational efficiency: The number of parameters in the sparse deformation field in this invention is only 1 / 64 of that in the dense deformation field, and the computational cost of FFD interpolation is much smaller than that of direct prediction of the dense field.

[0068] IV. Improved stability of the deformation field: This invention resists folding loss by penalizing the folding region with Jacobian determinant, avoiding topological distortion of the registered image;

[0069] V. Enhanced generalization ability: The adaptive data construction method of this invention does not require manual annotation and can quickly adapt to registration tasks in different scenarios such as remote sensing and industry. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the structure of the dual-branch multi-scale feature extraction network in this invention;

[0071] Figure 2 This is a flowchart illustrating the overall training process of the dual-branch multi-scale feature extraction network in this invention. Detailed Implementation

[0072] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0073] A nonlinear image registration method based on attention fusion includes the following steps:

[0074] Step 1: Preprocess the input fixed and floating images to be registered;

[0075] The specific steps are as follows:

[0076] Step 1.1: Normalize the input fixed image and floating image, specifically by normalizing the pixel values ​​of the fixed image and floating image to the [0,1] range;

[0077] Step 1.2: If the normalized fixed image and the floating image have different resolutions, scale the normalized floating image to the same width and height as the fixed image.

[0078] Step 1.3: Use the Sobel operator to extract the texture information of the fixed and floating images after they are of uniform size, and obtain the preprocessed fixed and floating images.

[0079] Step 2: Use a dual-branch multi-scale feature extraction network to register the preprocessed fixed image and floating image respectively.

[0080] like Figure 1The structure of the dual-branch multi-scale feature extraction network shown is as follows: the dual-branch multi-scale feature extraction network uses feature extraction networks with identical structures to extract features from the preprocessed fixed image and floating image respectively; a feature fusion layer fuses the features output by the two feature extraction networks; a deformation layer deforms the fused features output by the feature fusion layer to obtain a dense deformation field; and a registration layer registers the fixed image and floating image based on the output of the deformation layer.

[0081] The feature extraction network includes a first downsampling layer, a second downsampling layer, a third downsampling layer, a fourth downsampling layer, and a fifth downsampling layer, which sequentially downsample the preprocessed fixed or floating image, respectively. The features of the fixed and floating images output by each layer are as follows: , , , ,and ;

[0082] The feature fusion layer includes the two deepest features output by the feature extraction network. A self-attention layer that achieves cross-image feature fusion through the Transformer attention mechanism, and a cross-attention layer that performs cross-attention feature fusion on the output of the self-attention layer.

[0083] The self-attention layer incorporates the deepest features from the fixed and floating images output by the fifth downsampling layer of the two feature extraction networks. and Perform separately Flattening operation for a fixed image sequence and floating image sequences ;

[0084] Next, calculate the association weights of pixels within the fixed image sequence and the floating image sequence. The calculation formula is as follows:

[0085]

[0086] in, express Matrix, shape is , express Matrix, shape is , express The transpose of a matrix has the following shape: , express Matrix, shape is express The dimension, and satisfying , , and Indicates the sequence length;

[0087] The fixed image sequence after self-attention enhancement is output based on the association weights. and floating image sequences ,in, Indicates the batch size of a fixed or floating image sequence. Indicates the dimension size of a fixed or floating image sequence. Indicates the length of a fixed image sequence or a floating image sequence;

[0088] The cross-attention layer cross-fuses the fixed image sequence and the floating image sequence to obtain a fixed image feature fusion sequence. fusion sequence of floating image features Then proceed The operation yields fixed image fusion features Features fused with floating images Specifically: As , As and The cross-attention matrix is ​​calculated by inputting the cross-attention layer, and then the floating image sequence is processed based on the cross-attention matrix. The features in the image are weighted and aggregated to generate a fixed image sequence. Related fixed image feature fusion sequences Meanwhile, for fixed image sequences The features in the data are weighted and aggregated to generate a sequence of floating images. Related floating image feature fusion sequences Finally, and through The output after the operation is a fixed image fusion feature. Features fused with floating images .

[0089] The displacement field of the deformable layer is initialized to all zeros. , shape , 2 represents the batch size, and 2 represents the number of channels. and The width and height of the fifth layer feature map, and the displacement field. The two channels correspond to Initial offset of direction and Initial offset of direction To obtain spatial dimensions Completely consistent with fixed image fusion features and floating image fusion features, where all zero values ​​indicate that the initial assumption is that the fixed image fusion features and floating image fusion features have no spatial offset;

[0090] Floating image fusion features Resample and align it to a fixed image fusion feature. Spatial coordinate system, output The specific steps are as follows:

[0091] Source location calculation: using the initial displacement field Obtaining fixed image fusion features pixel position And calculate the floating image fusion features. Source location in The calculation formula is as follows:

[0092]

[0093] Boundary handling: If Beyond floating image fusion features Boundary, specifically or , or Fill with 0 values. , For the initial displacement field The displacement at each pixel position;

[0094] Bilinear interpolation: After calculating the source location, the resampled feature value is obtained by weighting the values ​​of the 4 neighboring pixels. :

[0095]

[0096] in: , , To round down, The interpolation weights cover four integer pixel positions surrounding the target point. , The weights sum to 1 output the resampled feature values. Features fused with fixed images The dimensions are completely identical;

[0097] Calculate resampling features Features at each location in the image are fused with fixed image features. The cost between features in the neighborhood of a corresponding position, and the centroid of the cost in the neighborhood, is the coarse deformation vector of that pixel position on that feature layer. The set of deformation vectors for all pixel positions is the coarse deformation field. Specifically:

[0098] Resampled feature values Each pixel position In fixed image fusion features The corresponding position is set to a 3×3 neighborhood, with a neighborhood radius of... Cover offset The cosine distance of selected features is used to measure feature similarity. The calculation formula is:

[0099]

[0100] Based on similarity For each pixel, a temperature parameter is introduced, weighted by the cost within its neighborhood. The optimal offset is calculated using the "cost centroid," which is equivalent to calculating the roughness deformation vector. , The formula is:

[0101]

[0102] The set of rough deformation vectors of all pixels is the rough deformation field. ;

[0103] The rough deformation field is convolved, and the deformation field finely adjusted by two sequentially connected convolutional modules and an upsampling layer is used as the initial deformation field. The first convolutional module includes a convolutional layer, a normalization layer, and an activation layer connected in sequence. The second convolutional module contains only one convolutional layer. The upsampling layer amplifies the size and value of the deformation field output by the second convolutional module by two times.

[0104] The initial deformation field As a new initial deformation field, and based on the output of the fourth downsampling layer, the optimized deformation field is calculated to obtain a sparse deformation field with 8 times downsampling. Specifically: using right After resampling, the result is Calculate again and similarity between Based on Calculate the rough deformation field Finally, it is optimized through convolutional layers. A sparse deformable field with 8x downsampling was obtained. ;

[0105] Free-form deformation algorithm , sparse deformable field The interpolation is a dense deformation field with the same size as the input floating image. Specifically, it refers to: free-form deformation algorithm Through sparse control points, i.e., sparse deformation fields The grid points define a three-dimensional parametric mesh, which incorporates sparse deformation fields. Each element's offset in the dense deformation field is calculated using bilinear interpolation;

[0106] Define cubic B-spline basis functions :

[0107]

[0108] Based on the defined cubic B-spline basis functions For input floating image pixels ,That direction and Offset in direction:

[0109]

[0110] in, For floor operations, respectively and For dense deformation field Two channels, each pixel corresponds to and directional offset, dense deformation field The shape and size are , Let l represent the l-th spline basis function.

[0111] The registration layer will float the image. Through dense deformation field Sampling is performed to obtain a fixed image. Aligned registration images Specifically, this involves using bilinear sampling or cubic interpolation sampling, based on the dense deformation field. For floating images Remapping is performed on the pixels of a fixed image. Its floating image The corresponding position in is , If any mapped position exceeds the floating image The boundaries are processed using zero-fill or mirror fill, and the final output is a registered image. Otherwise, directly output the registered image. (shape: N×3×H×W, 3 is the number of RGB channels, grayscale image is 1 channel), complete the forward inference process.

[0112] The dual-branch multi-scale feature extraction network employs multiple loss functions, specifically three types of loss functions jointly optimized, including normalized cross-correlation loss. Deformation field difference loss and resistance to folding loss ;

[0113] Normalized cross-correlation (NCC) loss Measuring the registered images With fixed image The similarity is expressed by the formula:

[0114]

[0115] in, For covariance, Standard deviation For batch size, The smaller the value, the more similar the registered image is to the fixed image;

[0116] Deformation field difference loss Measuring the predicted deformation field That is, dense deformation field With real deformation field Differences, using The loss (reducing the impact of outliers) is calculated using the following formula:

[0117]

[0118] in, These represent the image logarithm of the input dual-branch multi-scale feature extraction network, and the height and width of the input floating and fixed images, respectively.

[0119] Anti-folding loss The formula is:

[0120]

[0121] in, for Jacobian matrix (2×2 matrix, describing the linear transformation of local deformation). When the penalty is triggered, and Enter the height and width of a fixed or floating image;

[0122] Total loss function: weighted summation of three types of losses, the formula is:

[0123]

[0124] in, , , This is a hyperparameter.

[0125] The training data is constructed using arbitrary images (such as medical image datasets or remote sensing image datasets) as the original floating images. Randomly generate dense deformation fields without folding. (True Deformation Field): Offsets are generated through random perturbations (such as Gaussian noise, elastic deformation) to ensure the Jacobian determinant of the deformation field. (Without folding); will pass Sampling is used to obtain fixed images for training. (i.e., the baseline image for the supervision signal); the training dataset is {( , , The batch input model (two-branch multi-scale feature extraction network) is trained.

[0126] Two-branch multi-scale feature extraction network iterative optimizer: Employs the Adam or AdamW optimizer, with an initial learning rate of 1e-4, adjusted by learning rate decay (e.g., StepLR, decreasing to 0.1 every 10 epochs); Training iteration: Inputs training data into the model in batches and calculates the total loss. The parameters of the dual-branch multi-scale feature extraction network are updated through backpropagation; convergence condition: when the validation set... If there is no decrease after 10 consecutive epochs, or if the training epochs reach the preset maximum value (e.g., 200 epochs), stop training, save the optimal parameters of the bi-branch multi-scale feature extraction network, and refer to the overall training process. Figure 2 .

[0127] This invention acquires multi-scale features of fixed / floating images using the same feature extraction network, and combines self-attention to enhance the correlation of single-image features and cross-attention to achieve fine-grained fusion across images, solving the problems of limited receptive field and insufficient feature interaction in traditional CNNs. Pyramid upsampling optimizes the sparse deformation field step by step, optimizing the sparse deformation field from coarse to fine. The sparse-dense deformation field generation mechanism first predicts the sparse deformation field with 1 / 8 downsampling (only grid point offset), and then interpolates it to the dense deformation field of the original image size using the FFD algorithm, balancing "computational efficiency" and "registration accuracy" and avoiding the surge in computational load caused by directly predicting the dense field. The multi-loss function joint optimization scheme uses NCC loss to ensure registration similarity, L1 loss to constrain deformation field accuracy, and anti-folding loss to prevent deformation field folding. The three are weighted and jointly optimized to ensure that the model takes into account both accuracy and rationality in deformation field. Training data is constructed by randomly generating non-folded dense deformation fields, eliminating the need for manual labeling of registration pairs, reducing data dependence, and improving the model's generalization ability.

[0128] The above are merely representative embodiments among the many specific applications of this invention, and do not constitute any limitation on the scope of protection of this invention. All technical solutions formed by transformation or equivalent substitution fall within the scope of protection of this invention.

Claims

1. A nonlinear image registration method based on attention fusion, characterized in that, Includes the following steps: Step 1: Preprocess the input fixed and floating images to be registered; Step 2: Use a dual-branch multi-scale feature extraction network to register the preprocessed fixed image and floating image respectively.

2. The nonlinear image registration method based on attention fusion according to claim 1, characterized in that, The specific steps of step 1 are as follows: Step 1.1: Normalize the input fixed image and floating image, specifically by normalizing the pixel values ​​of the fixed image and floating image to the [0,1] range; Step 1.2: If the normalized fixed image and the floating image have different resolutions, scale the normalized floating image to the same width and height as the fixed image. Step 1.3: Use the Sobel operator to extract the texture information of the fixed and floating images after they are of uniform size, and obtain the preprocessed fixed and floating images.

3. The nonlinear image registration method based on attention fusion according to claim 2, characterized in that, The dual-branch multi-scale feature extraction network employs feature extraction networks with identical structures to extract features from the preprocessed fixed image and floating image respectively. A feature fusion layer fuses the features output by the two feature extraction networks. A deformation layer deforms the fused features output by the feature fusion layer to obtain a dense deformation field. A registration layer registers the fixed image and the floating image based on the output of the deformation layer. The feature extraction network includes a first downsampling layer, a second downsampling layer, a third downsampling layer, a fourth downsampling layer, and a fifth downsampling layer, which sequentially downsample the preprocessed fixed or floating image, respectively. The features of the fixed and floating images output by each layer are as follows: , , , ,and ; The feature fusion layer includes the two deepest features output by the feature extraction network. A self-attention layer that achieves cross-image feature fusion through the Transformer attention mechanism, and a cross-attention layer that performs cross-attention feature fusion on the output of the self-attention layer.

4. The nonlinear image registration method based on attention fusion according to claim 3, characterized in that, The self-attention layer incorporates the deepest features from the fixed and floating images output by the fifth downsampling layer of the two feature extraction networks. and Perform separately Flattening operation for a fixed image sequence and floating image sequences ; Next, calculate the association weights of pixels within the fixed image sequence and the floating image sequence. The calculation formula is as follows: in, express Matrix, shape is , express Matrix, shape is , express The transpose of a matrix has the following shape: , express Matrix, shape is express The dimension, and satisfying , , and Indicates the sequence length; The fixed image sequence after self-attention enhancement is output based on the association weights. and floating image sequences ,in, Indicates the batch size of a fixed or floating image sequence. Indicates the dimension size of a fixed or floating image sequence. Indicates the length of a fixed image sequence or a floating image sequence; The cross-attention layer cross-fuses the fixed image sequence and the floating image sequence to obtain a fixed image feature fusion sequence. fusion sequence of floating image features Then proceed The operation yields fixed image fusion features Features fused with floating images Specifically: As , As and The cross-attention matrix is ​​calculated by inputting the cross-attention layer, and then the floating image sequence is processed based on the cross-attention matrix. The features in the image are weighted and aggregated to generate a fixed image sequence. Related fixed image feature fusion sequences Meanwhile, for fixed image sequences The features in the data are weighted and aggregated to generate a sequence of floating images. Related floating image feature fusion sequences Finally, and through The output after the operation is a fixed image fusion feature. Features fused with floating images .

5. The nonlinear image registration method based on attention fusion according to claim 4, characterized in that, The displacement field of the deformable layer is initialized to all zeros. , shape , 2 represents the batch size, and 2 represents the number of channels. and The width and height of the fifth layer feature map, and the displacement field. The two channels correspond to Initial offset of direction and Initial offset of direction To obtain spatial dimensions Completely consistent with fixed image fusion features and floating image fusion features, where all zero values ​​indicate that the initial assumption is that the fixed image fusion features and floating image fusion features have no spatial offset; Floating image fusion features Resample and align it to a fixed image fusion feature. Spatial coordinate system, output The specific steps are as follows: Source location calculation: using the initial displacement field Obtaining fixed image fusion features pixel position And calculate the floating image fusion features. Source location in The calculation formula is as follows: Boundary handling: If Beyond floating image fusion features Boundary, specifically or , or Fill with 0 values. , For the initial displacement field The displacement at each pixel position; Bilinear interpolation: After calculating the source location, the resampled feature value is obtained by weighting the values ​​of the 4 neighboring pixels. : in: , , To round down, The interpolation weights cover four integer pixel positions surrounding the target point. , The weights sum to 1 output the resampled feature values. Features fused with fixed images The dimensions are completely identical; Calculate resampling features Features at each location in the image are fused with fixed image features. The cost between features in the neighborhood of a corresponding position, and the centroid of the cost in the neighborhood, is the coarse deformation vector of that pixel position on that feature layer. The set of deformation vectors for all pixel positions is the coarse deformation field. Specifically: Resampled feature values Each pixel position In fixed image fusion features The corresponding position is set to a 3×3 neighborhood, with a neighborhood radius of... Cover offset The cosine distance of selected features is used to measure feature similarity. The calculation formula is: Based on similarity For each pixel, a temperature parameter is introduced, weighted by the cost within its neighborhood. The optimal offset is calculated using the "cost centroid," which is equivalent to calculating the roughness deformation vector. , The formula is: The set of rough deformation vectors of all pixels is the rough deformation field. ; The rough deformation field is convolved, and the deformation field finely adjusted by two sequentially connected convolutional modules and an upsampling layer is used as the initial deformation field. The first convolutional module includes a convolutional layer, a normalization layer, and an activation layer connected in sequence. The second convolutional module contains only one convolutional layer. The upsampling layer amplifies the size and value of the deformation field output by the second convolutional module by two times. The initial deformation field As a new initial deformation field, and based on the output of the fourth downsampling layer, the optimized deformation field is calculated to obtain a sparse deformation field with 8 times downsampling. Specifically: using right After resampling, the result is Calculate again and similarity between Based on Calculate the rough deformation field Finally, it is optimized through convolutional layers. A sparse deformable field with 8x downsampling was obtained. ; Free-form deformation algorithm , sparse deformable field The interpolation is a dense deformation field with the same size as the input floating image. Specifically, it refers to: free-form deformation algorithm Through sparse control points, i.e., sparse deformation fields The grid points define a three-dimensional parametric mesh, which incorporates sparse deformation fields. Each element's offset in the dense deformation field is calculated using bilinear interpolation; Define cubic B-spline basis functions : Based on the defined cubic B-spline basis functions For the pixels of the input floating image ,That direction and Offset in direction: in, For floor operations, respectively and For dense deformation field Two channels, each pixel corresponds to and directional offset, dense deformation field The shape and size are , Let l represent the l-th spline basis function.

6. The nonlinear image registration method based on attention fusion according to claim 4, characterized in that, The registration layer will float the image. Through dense deformation field Sampling is performed to obtain a fixed image. Aligned registration images Specifically, this involves using bilinear sampling or cubic interpolation sampling, based on the dense deformation field. For floating images Remapping is performed on the pixels of a fixed image. Its floating image The corresponding position in is , If any mapped position exceeds the floating image The boundaries are processed using zero-fill or mirror fill, and the final output is a registered image. Otherwise, directly output the registered image. .

7. The nonlinear image registration method based on attention fusion according to claim 5, characterized in that, The dual-branch multi-scale feature extraction network employs multiple loss functions, specifically three types of loss functions jointly optimized, including normalized cross-correlation loss. Deformation field difference loss and resistance to folding loss ; Normalized cross-correlation loss Measuring the registered images With fixed image The similarity is expressed by the formula: in, For covariance, Standard deviation For batch size, The smaller the value, the more similar the registered image is to the fixed image; Deformation field difference loss Measuring the predicted deformation field That is, dense deformation field With real deformation field Differences, using The loss is calculated using the following formula: in, These represent the image logarithm of the input dual-branch multi-scale feature extraction network, and the height and width of the input floating and fixed images, respectively. Anti-folding loss The formula is: in, for Jacobian matrix, When the penalty is triggered, and Enter the height and width of a fixed or floating image; Total loss function: weighted summation of three types of losses, the formula is: in, , , This is a hyperparameter.

8. A nonlinear image registration system based on attention fusion, comprising a computer program and a processor for processing the computer program, characterized in that: The processor executes a computer program to implement the steps of the method described in any one of claims 1-7.

Citation Information

Patent Citations

  • Image registration method based on Swin Transform and CNN double-branch coupling

    CN115082293A

  • CARDIAC MAGNETIC RESONANCE IMAGE REGISTRATION METHOD BASED ON MASK AUTOCODER CNN-TRANForMER

    CN116012344A

  • Image registration method

    CN116228824A

  • Image registration method based on cross neighborhood attention enhancement mechanism

    CN118447061A

  • Large-deformation medical image registration method based on multi-window gating multilayer perceptron and context residual enhancement

    CN118982566A