Nonlinear image registration method and system based on attention fusion

Abstract

This invention discloses a nonlinear image registration method and system based on attention fusion, which can achieve high-precision image registration under complex nonlinear deformations. Belonging to the field of computer vision and image processing technology, it solves the problems of poor adaptation to nonlinear deformations, insufficient cross-image feature fusion, low computational efficiency, and easy folding of deformation fields in existing technologies. This invention preprocesses the input fixed and floating images to be registered; a dual-branch multi-scale feature extraction network is used to register the preprocessed fixed and floating images respectively. This invention is applicable to industrial visual inspection scenarios.

Description

Technical Field

[0001] A nonlinear image registration method and system based on attention fusion is proposed for industrial visual inspection scenarios. It can achieve high-precision image registration under complex nonlinear deformation and belongs to the field of computer vision and image processing technology. Background Technology

[0002] Image registration is a key technology that maps two (or more) images acquired at different times, from different perspectives, or from different sensors to the same coordinate space, aligning the positions of corresponding targets. It plays an irreplaceable role in fields such as medical diagnosis, remote sensing analysis, and computer vision.

[0003] Existing image registration techniques are mainly divided into three categories:

[0004] Traditional feature matching methods, such as SIFT (Scale Invariant Feature Transform) and SURF (Speeded Robust Feature Transform), achieve registration by extracting local feature points from the image, matching these feature points, and then calculating a transformation matrix (such as a homography matrix or affine matrix). However, these methods are not robust to nonlinear deformations (such as tissue deformation in medical images, terrain undulations in remote sensing images, and changes in shooting distance of industrial line scan cameras), and are prone to feature point loss in low-texture areas (such as uniform backgrounds and smooth part surfaces), leading to registration failure.

[0005] Traditional deep learning methods, such as registration models based on convolutional neural networks (CNNs) like U-Net and V-Net, directly predict dense deformation fields or transformation parameters. However, CNNs have limited receptive fields, making it difficult to capture global feature relationships in images; and most models only perform feature fusion through convolutional layers, lacking fine-grained interactions across images (fixed and floating images), and have insufficient adaptability to complex deformations.

[0006] Deformation field optimization-based methods: Some methods achieve registration by directly predicting the dense deformation field of the original image size, but the number of parameters in the dense deformation field is huge (e.g., a 256×256 image requires about 130,000 x / y offsets), which leads to a surge in model computation and difficulty in training convergence. At the same time, existing methods generally lack effective deformation field regularization mechanisms, which are prone to "folding phenomenon" (deformation field Jacobian determinant ≤ 0, corresponding to the distortion of the topological structure of the image region), which seriously affects the registration accuracy.

[0007] In summary, existing technologies suffer from four core problems: poor adaptation to nonlinear deformation, insufficient cross-image feature fusion, low computational efficiency, and easy folding of deformation fields. These issues make it difficult to meet the requirements of high-precision and high-robustness image registration (such as deformation alignment in industrial images and dynamic scene stitching in remote sensing images). Therefore, there is an urgent need for an image registration scheme that can achieve global-local feature linkage, efficiently generate reasonable deformation fields, and balance accuracy and efficiency. Summary of the Invention

[0008] To address the problems mentioned above, the present invention aims to provide a nonlinear image registration method and system based on attention fusion, which solves the problems of poor nonlinear deformation adaptation, insufficient cross-image feature fusion, low computational efficiency, and easy folding of deformation fields in existing technologies.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A nonlinear image registration method based on attention fusion includes the following steps:

[0011] Step 1: Preprocess the input fixed and floating images to be registered;

[0012] Step 2: Use a dual-branch multi-scale feature extraction network to register the preprocessed fixed image and floating image respectively.

[0013] Furthermore, the specific steps of step 1 are as follows:

[0014] Step 1.1: Normalize the input fixed image and floating image, specifically by normalizing the pixel values ​​of the fixed image and floating image to the [0,1] range;

[0015] Step 1.2: If the normalized fixed image and the floating image have different resolutions, scale the normalized floating image to the same width and height as the fixed image.

[0016] Step 1.3: Use the Sobel operator to extract the texture information of the fixed and floating images after they are of uniform size, and obtain the preprocessed fixed and floating images.

[0017] Furthermore, the dual-branch multi-scale feature extraction network employs feature extraction networks with identical structures to extract features from the preprocessed fixed image and floating image respectively, a feature fusion layer that fuses the features output by the two feature extraction networks, a deformation layer that deforms the fused features output by the feature fusion layer to obtain a dense deformation field, and a registration layer that registers the fixed image and floating image based on the output of the deformation layer.

[0018] The feature extraction network includes a first downsampling layer, a second downsampling layer, a third downsampling layer, a fourth downsampling layer, and a fifth downsampling layer, which sequentially downsample the preprocessed fixed or floating image, respectively. The features of the fixed and floating images output by each layer are as follows: , , , ,and ;

[0019] The feature fusion layer includes the two deepest features output by the feature extraction network. A self-attention layer that achieves cross-image feature fusion through the Transformer attention mechanism, and a cross-attention layer that performs cross-attention feature fusion on the output of the self-attention layer.

[0020] Furthermore, the self-attention layer combines the deepest features of the fixed and floating images output by the fifth downsampling layer of the two feature extraction networks. and Perform separately Flattening operation for a fixed image sequence and floating image sequences ;

[0021] Next, calculate the association weights of pixels within the fixed image sequence and the floating image sequence. The calculation formula is as follows:

[0022]

[0023] in, express Matrix, shape is , express Matrix, shape is , express The transpose of a matrix has the following shape: , express Matrix, shape is express The dimension, and satisfying , , and Indicates the sequence length;

[0024] The fixed image sequence after self-attention enhancement is output based on the association weights. and floating image sequences ,in, Indicates the batch size of a fixed or floating image sequence. Indicates the dimension size of a fixed or floating image sequence. Indicates the length of a fixed image sequence or a floating image sequence;

[0025] The cross-attention layer cross-fuses the fixed image sequence and the floating image sequence to obtain a fixed image feature fusion sequence. fusion sequence of floating image features Then proceed The operation yields fixed image fusion features Features fused with floating images Specifically: As , As and The cross-attention matrix is ​​calculated by inputting the cross-attention layer, and then the floating image sequence is processed based on the cross-attention matrix. The features in the image are weighted and aggregated to generate a fixed image sequence. Related fixed image feature fusion sequences Meanwhile, for fixed image sequences The features in the data are weighted and aggregated to generate a sequence of floating images. Related floating image feature fusion sequences Finally, and through The output after the operation is a fixed image fusion feature. Features fused with floating images .

[0026] Furthermore, the deformation layer is initialized with a displacement field of all zeros. , shape , 2 represents the batch size, and 2 represents the number of channels. and The width and height of the fifth layer feature map, and the displacement field. The two channels correspond to Initial offset of direction and Initial offset of direction To obtain spatial dimensions Completely consistent with fixed image fusion features and floating image fusion features, where all zero values ​​indicate that the initial assumption is that the fixed image fusion features and floating image fusion features have no spatial offset;

[0027] Floating image fusion features Resample and align it to a fixed image fusion feature. Spatial coordinate system, output The specific steps are as follows:

[0028] Source location calculation: using the initial displacement field Obtaining fixed image fusion features pixel position And calculate the floating image fusion features. Source location in The calculation formula is as follows:

[0029]

[0030] Boundary handling: If Beyond floating image fusion features Boundary, specifically or , or Fill with 0 values. , For the initial displacement field The displacement at each pixel position;

[0031] Bilinear interpolation: After calculating the source location, the resampled feature value is obtained by weighting the values ​​of the 4 neighboring pixels. :

[0032]

[0033] in: , , To round down, The interpolation weights cover four integer pixel positions surrounding the target point. , The weights sum to 1 output the resampled feature values. Features fused with fixed images The dimensions are completely identical;

[0034] Calculate resampling features Features at each location in the image are fused with fixed image features. The cost between features in the neighborhood of a corresponding position, and the centroid of the cost in the neighborhood, is the coarse deformation vector of that pixel position on that feature layer. The set of deformation vectors for all pixel positions is the coarse deformation field. Specifically:

[0035] Resampled feature values Each pixel position In fixed image fusion features The corresponding position is set to a 3×3 neighborhood, with a neighborhood radius of... Cover offset The cosine distance of selected features is used to measure feature similarity. The calculation formula is:

[0036]

[0037] Based on similarity For each pixel, a temperature parameter is introduced, weighted by the cost within its neighborhood. The optimal offset is calculated using the "cost centroid," which is equivalent to calculating the roughness deformation vector. , The formula is:

[0038]

[0039] The set of rough deformation vectors of all pixels is the rough deformation field. ;

[0040] The rough deformation field is convolved, and the deformation field finely adjusted by two sequentially connected convolutional modules and an upsampling layer is used as the initial deformation field. The first convolutional module includes a convolutional layer, a normalization layer, and an activation layer connected in sequence. The second convolutional module contains only one convolutional layer. The upsampling layer amplifies the size and value of the deformation field output by the second convolutional module by two times.

[0041] The initial deformation field As a new initial deformation field, and based on the output of the fourth downsampling layer, the optimized deformation field is calculated to obtain a sparse deformation field with 8 times downsampling. Specifically: using right After resampling, the result is Calculate again and similarity between Based on Calculate the rough deformation field Finally, it is optimized through convolutional layers. The sparse deformation field with 8x downsampling was obtained. ;

[0042] Free-form deformation algorithm , sparse deformable field The interpolation is a dense deformation field with the same size as the input floating image. Specifically, it refers to: free-form deformation algorithm Through sparse control points, i.e., sparse deformation fields The grid points define a three-dimensional parametric mesh, which incorporates sparse deformation fields. Each element's offset in the dense deformation field is calculated using bilinear interpolation;

[0043] Define cubic B-spline basis functions :

[0044]

[0045] Based on the defined cubic B-spline basis functions For the pixels of the input floating image ,That direction and Offset in direction:

[0046]

[0047] in, For floor operations, respectively and For dense deformation field Two channels, each pixel corresponds to and directional offset, dense deformation field The shape and size are , Let l represent the l-th spline basis function.

[0048] Furthermore, the registration layer will register the floating image. Through dense deformation field Sampling is performed to obtain a fixed image. Aligned registration images Specifically, this involves using bilinear sampling or cubic interpolation sampling, based on the dense deformation field. For floating images Remapping is performed on the pixels of a fixed image. Its floating image The corresponding position in is , If any mapped position exceeds the floating image The boundaries are processed using zero-fill or mirror fill, and the final output is a registered image. Otherwise, directly output the registered image. .

[0049] Furthermore, the dual-branch multi-scale feature extraction network employs multiple loss functions, specifically three types of loss functions jointly optimized, including normalized cross-correlation loss. Deformation field difference loss and resistance to folding loss ;

[0050] Normalized cross-correlation loss Measuring the registered images With fixed image The similarity is expressed by the formula:

[0051]

[0052] in, For covariance, Standard deviation, For batch size, The smaller the value, the more similar the registered image is to the fixed image;

[0053] Deformation field difference loss Measuring the predicted deformation field That is, dense deformation field With real deformation field Differences, using The loss is calculated using the following formula:

[0054]

[0055] in, These represent the image logarithm of the input dual-branch multi-scale feature extraction network, and the height and width of the input floating and fixed images, respectively.

[0056] Anti-folding loss The formula is:

[0057]

[0058] in, for Jacobian matrix, When the penalty is triggered, and Enter the height and width of a fixed or floating image;

[0059] Total loss function: weighted summation of three types of losses, the formula is:

[0060]

[0061] in, , , This is a hyperparameter.

[0062] A nonlinear image registration system based on attention fusion includes a computer program, a processor for processing the computer program, and the processor executing the computer program to implement the steps of the nonlinear image registration method based on attention fusion.

[0063] Compared with the prior art, the beneficial effects of this invention are as follows:

[0064] This invention overcomes four core problems of existing technologies: poor adaptation of nonlinear deformation, insufficient cross-image feature fusion, low computational efficiency, and easy folding of deformation fields. It meets the requirements for high-precision and high-robustness image registration, and can achieve global-local feature linkage, efficient generation of reasonable deformation fields, and image registration that balances accuracy and efficiency. Specifically, it is manifested in the following ways:

[0065] I. Enhanced adaptability to complex nonlinear deformations: The Transformer self-attention mechanism in this invention can capture the correlation between global and local features of an image, and cross-attention enables fine-grained interaction across images;

[0066] II. Higher registration accuracy: This invention uses a dual-branch multi-scale feature extraction network to obtain multi-scale feature fusion to supplement spatial details, and pyramid level optimization to reduce offset error;

[0067] Third, superior computational efficiency: The number of parameters in the sparse deformation field in this invention is only 1 / 64 of that in the dense deformation field, and the computational cost of FFD interpolation is much smaller than that of direct prediction of the dense field.

[0068] IV. Improved stability of the deformation field: This invention resists folding loss by penalizing the folding region with Jacobian determinant, avoiding topological distortion of the registered image;

[0069] V. Enhanced generalization ability: The adaptive data construction method of this invention does not require manual annotation and can quickly adapt to registration tasks in different scenarios such as remote sensing and industry. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the structure of the dual-branch multi-scale feature extraction network in this invention;

[0071] Figure 2 This is a flowchart illustrating the overall training process of the dual-branch multi-scale feature extraction network in this invention. Detailed Implementation

[0072] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0073] A nonlinear image registration method based on attention fusion includes the following steps:

[0074] Step 1: Preprocess the input fixed and floating images to be registered;

[0075] The specific steps are as follows:

[0076] Step 1.1: Normalize the input fixed image and floating image, specifically by normalizing the pixel values ​​of the fixed image and floating image to the [0,1] range;

[0077] Step 1.2: If the normalized fixed image and the floating image have different resolutions, scale the normalized floating image to the same width and height as the fixed image.

[0078] Step 1.3: Use the Sobel operator to extract the texture information of the fixed and floating images after they are of uniform size, and obtain the preprocessed fixed and floating images.

[0079] Step 2: Use a dual-branch multi-scale feature extraction network to register the preprocessed fixed image and floating image respectively.

[0080] like Figure 1The structure of the dual-branch multi-scale feature extraction network shown is as follows: the dual-branch multi-scale feature extraction network uses feature extraction networks with identical structures to extract features from the preprocessed fixed image and floating image respectively; a feature fusion layer fuses the features output by the two feature extraction networks; a deformation layer deforms the fused features output by the feature fusion layer to obtain a dense deformation field; and a registration layer registers the fixed image and floating image based on the output of the deformation layer.

[0081] The feature extraction network includes a first downsampling layer, a second downsampling layer, a third downsampling layer, a fourth downsampling layer, and a fifth downsampling layer, which sequentially downsample the preprocessed fixed or floating image, respectively. The features of the fixed and floating images output by each layer are as follows: , , , ,and ;

[0082] The feature fusion layer includes the two deepest features output by the feature extraction network. A self-attention layer that achieves cross-image feature fusion through the Transformer attention mechanism, and a cross-attention layer that performs cross-attention feature fusion on the output of the self-attention layer.

[0083] The self-attention layer incorporates the deepest features from the fixed and floating images output by the fifth downsampling layer of the two feature extraction networks. and Perform separately Flattening operation for a fixed image sequence and floating image sequences ;

[0084] Next, calculate the association weights of pixels within the fixed image sequence and the floating image sequence. The calculation formula is as follows:

[0085]

[0086] in, express Matrix, shape is , express Matrix, shape is , express The transpose of a matrix has the following shape: , express Matrix, shape is express The dimension, and satisfying , , and Indicates the sequence length;

[0087] The fixed image sequence after self-attention enhancement is output based on the association weights. and floating image sequences ,in, Indicates the batch size of a fixed or floating image sequence. Indicates the dimension size of a fixed or floating image sequence. Indicates the length of a fixed image sequence or a floating image sequence;

[0088] The cross-attention layer cross-fuses the fixed image sequence and the floating image sequence to obtain a fixed image feature fusion sequence. fusion sequence of floating image features Then proceed The operation yields fixed image fusion features Features fused with floating images Specifically: As , As and The cross-attention matrix is ​​calculated by inputting the cross-attention layer, and then the floating image sequence is processed based on the cross-attention matrix. The features in the image are weighted and aggregated to generate a fixed image sequence. Related fixed image feature fusion sequences Meanwhile, for fixed image sequences The features in the data are weighted and aggregated to generate a sequence of floating images. Related floating image feature fusion sequences Finally, and through The output after the operation is a fixed image fusion feature. Features fused with floating images .

[0089] The displacement field of the deformable layer is initialized to all zeros. , shape , 2 represents the batch size, and 2 represents the number of channels. and The width and height of the fifth layer feature map, and the displacement field. The two channels correspond to Initial offset of direction and Initial offset of direction To obtain spatial dimensions Completely consistent with fixed image fusion features and floating image fusion features, where all zero values ​​indicate that the initial assumption is that the fixed image fusion features and floating image fusion features have no spatial offset;

[0090] Floating image fusion features Resample and align it to a fixed image fusion feature. Spatial coordinate system, output The specific steps are as follows:

[0091] Source location calculation: using the initial displacement field Obtaining fixed image fusion features pixel position And calculate the floating image fusion features. Source location in The calculation formula is as follows:

[0092]

[0093] Boundary handling: If Beyond floating image fusion features Boundary, specifically or , or Fill with 0 values. , For the initial displacement field The displacement at each pixel position;

[0094] Bilinear interpolation: After calculating the source location, the resampled feature value is obtained by weighting the values ​​of the 4 neighboring pixels. :

[0095]

[0096] in: , , To round down, The interpolation weights cover four integer pixel positions surrounding the target point. , The weights sum to 1 output the resampled feature values. Features fused with fixed images The dimensions are completely identical;

[0097] Calculate resampling features Features at each location in the image are fused with fixed image features. The cost between features in the neighborhood of a corresponding position, and the centroid of the cost in the neighborhood, is the coarse deformation vector of that pixel position on that feature layer. The set of deformation vectors for all pixel positions is the coarse deformation field. Specifically:

[0098] Resampled feature values Each pixel position In fixed image fusion features The corresponding position is set to a 3×3 neighborhood, with a neighborhood radius of... Cover offset The cosine distance of selected features is used to measure feature similarity. The calculation formula is:

[0099]

[0100] Based on similarity For each pixel, a temperature parameter is introduced, weighted by the cost within its neighborhood. The optimal offset is calculated using the "cost centroid," which is equivalent to calculating the roughness deformation vector. , The formula is:

[0101]

[0102] The set of rough deformation vectors of all pixels is the rough deformation field. ;

[0103] The rough deformation field is convolved, and the deformation field finely adjusted by two sequentially connected convolutional modules and an upsampling layer is used as the initial deformation field. The first convolutional module includes a convolutional layer, a normalization layer, and an activation layer connected in sequence. The second convolutional module contains only one convolutional layer. The upsampling layer amplifies the size and value of the deformation field output by the second convolutional module by two times.

[0104] The initial deformation field As a new initial deformation field, and based on the output of the fourth downsampling layer, the optimized deformation field is calculated to obtain a sparse deformation field with 8 times downsampling. Specifically: using right After resampling, the result is Calculate again and Similarity between Based on Calculate the rough deformation field Finally, it is optimized through convolutional layers. The sparse deformation field with 8x downsampling was obtained. ;

[0105] Free-form deformation algorithm , sparse deformable field The interpolation is a dense deformation field with the same size as the input floating image. Specifically, it refers to: free-form deformation algorithm Through sparse control points, i.e., sparse deformation fields The grid points define a three-dimensional parametric mesh, which incorporates sparse deformation fields. Each element's offset in the dense deformation field is calculated using bilinear interpolation;

[0106] Define cubic B-spline basis functions :

[0107]

[0108] Based on the defined cubic B-spline basis functions For input floating image pixels ,That direction and Offset in direction:

[0109]

[0110] in, For floor operations, respectively and For dense deformation field Two channels, each pixel corresponds to and directional offset, dense deformation field The shape and size are , Let l represent the l-th spline basis function.

[0111] The registration layer will float the image. Through dense deformation field Sampling is performed to obtain a fixed image. Aligned registration images Specifically, this involves using bilinear sampling or cubic interpolation sampling, based on the dense deformation field. For floating images Remapping is performed on the pixels of a fixed image. Its floating image The corresponding position in is , If any mapped position exceeds the floating image The boundaries are processed using zero-fill or mirror fill, and the final output is a registered image. Otherwise, directly output the registered image. (shape: N×3×H×W, 3 is the number of RGB channels, grayscale image is 1 channel), complete the forward inference process.

[0112] The dual-branch multi-scale feature extraction network employs multiple loss functions, specifically three types of loss functions jointly optimized, including normalized cross-correlation loss. Deformation field difference loss and resistance to folding loss ;

[0113] Normalized cross-correlation (NCC) loss Measuring the registered images With fixed image The similarity is expressed by the formula:

[0114]

[0115] in, For covariance, Standard deviation, For batch size, The smaller the value, the more similar the registered image is to the fixed image;

[0116] Deformation field difference loss Measuring the predicted deformation field That is, dense deformation field With real deformation field Differences, using The loss (reducing the impact of outliers) is calculated using the following formula:

[0117]

[0118] in, These represent the image logarithm of the input dual-branch multi-scale feature extraction network, and the height and width of the input floating and fixed images, respectively.

[0119] Anti-folding loss The formula is:

[0120]

[0121] in, for Jacobian matrix (2×2 matrix, describing the linear transformation of local deformation). When the penalty is triggered, and Enter the height and width of a fixed or floating image;

[0122] Total loss function: weighted summation of three types of losses, the formula is:

[0123]

[0124] in, , , This is a hyperparameter.

[0125] The training data is constructed using arbitrary images (such as medical image datasets or remote sensing image datasets) as the original floating images. Randomly generate dense deformation fields without folding. (True Deformation Field): Offsets are generated through random perturbations (such as Gaussian noise, elastic deformation) to ensure the Jacobian determinant of the deformation field. (Without folding); will pass Sampling is used to obtain fixed images for training. (i.e., the baseline image for the supervision signal); the training dataset is {( , , The batch input model (two-branch multi-scale feature extraction network) is trained.

[0126] Two-branch multi-scale feature extraction network iterative optimizer: Employs the Adam or AdamW optimizer, with an initial learning rate of 1e-4, adjusted by learning rate decay (e.g., StepLR, decreasing to 0.1 every 10 epochs); Training iteration: Input training data into the model in batches and calculate the total loss. The parameters of the dual-branch multi-scale feature extraction network are updated through backpropagation; convergence condition: when the validation set... If there is no decrease after 10 consecutive epochs, or if the training epochs reach the preset maximum value (e.g., 200 epochs), stop training, save the optimal parameters of the bi-branch multi-scale feature extraction network, and refer to the overall training process. Figure 2 .

[0127] This invention acquires multi-scale features of fixed / floating images using the same feature extraction network, and combines self-attention to enhance the correlation of single-image features and cross-attention to achieve fine-grained fusion across images, solving the problems of limited receptive field and insufficient feature interaction in traditional CNNs. Pyramid upsampling optimizes the sparse deformation field step by step, optimizing the sparse deformation field from coarse to fine. The sparse-dense deformation field generation mechanism first predicts the sparse deformation field with 1 / 8 downsampling (only grid point offset), and then interpolates it to the dense deformation field of the original image size using the FFD algorithm, balancing "computational efficiency" and "registration accuracy" and avoiding the surge in computational load caused by directly predicting the dense field. The multi-loss function joint optimization scheme uses NCC loss to ensure registration similarity, L1 loss to constrain deformation field accuracy, and anti-folding loss to prevent deformation field folding. The three are weighted and jointly optimized to ensure that the model takes into account both accuracy and rationality in deformation field. Training data is constructed by randomly generating non-folded dense deformation fields, eliminating the need for manual labeling of registration pairs, reducing data dependence, and improving the model's generalization ability.

[0128] The above are merely representative embodiments among the many specific applications of this invention, and do not constitute any limitation on the scope of protection of this invention. All technical solutions formed by transformation or equivalent substitution fall within the scope of protection of this invention.

Claims

1. A nonlinear image registration method based on attention fusion, characterized in that, Includes the following steps: Step 1: Preprocess the input fixed and floating images to be registered; Step 2: A dual-branch multi-scale feature extraction network is used to register the preprocessed fixed image and the floating image respectively. A feature extraction network with the same structure is used to extract features from the preprocessed fixed image and the floating image respectively. Then, the features are fused through a feature fusion layer. The fused features output by the feature fusion layer are deformed to obtain a deformation layer with a dense deformation field. The registration layer registers the fixed image and the floating image based on the output of the deformation layer. The displacement field of the deformable layer is initialized to all zeros. , shape is , 2 represents the batch size, and 2 represents the number of channels. and The width and height of the fifth layer feature map, and the displacement field. The two channels correspond to Initial offset of direction and Initial offset of direction To obtain spatial dimensions Completely consistent with fixed image fusion features and floating image fusion features, where all zero values ​​indicate that the initial assumption is that the fixed image fusion features and floating image fusion features have no spatial offset; Floating image fusion features Resample and align it to a fixed image fusion feature. Spatial coordinate system, output The specific steps are as follows: Source location calculation: using the initial displacement field Obtaining fixed image fusion features pixel position And calculate the floating image fusion features. Source location in The calculation formula is as follows: Boundary handling: If Beyond floating image fusion features Boundary, specifically or , or Fill with 0 values. , For the initial displacement field The displacement at each pixel position; Bilinear interpolation: After calculating the source location, the resampled feature value is obtained by weighting the values ​​of the 4 neighboring pixels. : in: , , To round down, The interpolation weights cover four integer pixel positions surrounding the target point. , The weights sum to 1 output the resampled feature values. Features fused with fixed images The dimensions are completely identical; Calculate resampling features Features at each location in the image are fused with fixed image features. The cost between features in the neighborhood of a corresponding position, the centroid of the cost in the neighborhood is the coarse deformation vector of that pixel position on the feature layer, and the set of deformation vectors of all pixel positions is the coarse deformation field. Specifically: Resampled feature values Each pixel position In fixed image fusion features The corresponding position is set to a 3×3 neighborhood, with a neighborhood radius of... Cover offset The cosine distance of selected features is used to measure feature similarity. The calculation formula is: Based on similarity For each pixel, a temperature parameter is introduced, weighted by the cost within its neighborhood. The optimal offset is calculated using the "cost centroid," which is equivalent to calculating the roughness deformation vector. , The formula is: The set of rough deformation vectors of all pixels is the rough deformation field. ; The rough deformation field is convolved, and the deformation field finely adjusted by two sequentially connected convolutional modules and an upsampling layer is used as the initial deformation field. The first convolutional module includes a convolutional layer, a normalization layer, and an activation layer connected in sequence. The second convolutional module contains only one convolutional layer. The upsampling layer amplifies the size and value of the deformation field output by the second convolutional module by two times. The initial deformation field As a new initial deformation field, and based on the output of the fourth downsampling layer, the optimized deformation field is calculated to obtain a sparse deformation field with 8 times downsampling. Specifically: using right After resampling, the result is Calculate again and similarity between Based on Calculate the rough deformation field Finally, it is optimized through convolutional layers. The sparse deformation field with 8x downsampling was obtained. ; Free-form deformation algorithm , sparse deformable field The interpolation is a dense deformation field with the same size as the input floating image. Specifically, it refers to: free-form deformation algorithm Through sparse control points, i.e., sparse deformation fields The grid points define a three-dimensional parametric mesh, which incorporates sparse deformation fields. Each element's offset in the dense deformation field is calculated using bilinear interpolation; Define cubic B-spline basis functions : Based on the defined cubic B-spline basis functions For the pixels of the input floating image ,That direction and Offset in direction: in, For floor operations, respectively and For dense deformation field Two channels, each pixel corresponds to and directional offset, dense deformation field The shape and size are , Let l represent the l-th spline basis function.

2. The nonlinear image registration method based on attention fusion according to claim 1, characterized in that, The specific steps of step 1 are as follows: Step 1.1: Normalize the input fixed image and floating image, specifically by normalizing the pixel values ​​of the fixed image and floating image to the [0,1] range; Step 1.2: If the normalized fixed image and the floating image have different resolutions, scale the normalized floating image to the same width and height as the fixed image. Step 1.3: Use the Sobel operator to extract the texture information of the fixed and floating images after they are of uniform size, and obtain the preprocessed fixed and floating images.

3. The nonlinear image registration method based on attention fusion according to claim 2, characterized in that, The feature extraction network includes a first downsampling layer, a second downsampling layer, a third downsampling layer, a fourth downsampling layer, and a fifth downsampling layer, which sequentially downsample the preprocessed fixed or floating image, respectively. The features of the fixed and floating images output by each layer are as follows: , , , ,and ; The feature fusion layer includes the two deepest features output by the feature extraction network. A self-attention layer that achieves cross-image feature fusion through the Transformer attention mechanism, and a cross-attention layer that performs cross-attention feature fusion on the output of the self-attention layer.

4. The nonlinear image registration method based on attention fusion according to claim 3, characterized in that, The self-attention layer incorporates the deepest features from the fixed and floating images output by the fifth downsampling layer of the two feature extraction networks. and Perform separately Flattening operation for a fixed image sequence and floating image sequences ; Next, calculate the association weights of pixels within the fixed image sequence and the floating image sequence. The calculation formula is as follows: in, express Matrix, shape is , express Matrix, shape is , express The transpose of a matrix has the following shape: , express Matrix, shape is express The dimension, and satisfying , , and Indicates the sequence length; The fixed image sequence after self-attention enhancement is output based on the association weights. and floating image sequences ,in, Indicates the batch size of a fixed or floating image sequence. Indicates the dimension size of a fixed or floating image sequence. Indicates the length of a fixed image sequence or a floating image sequence; The cross-attention layer cross-fuses the fixed image sequence and the floating image sequence to obtain a fixed image feature fusion sequence. fusion sequence of floating image features Then proceed The operation yields fixed image fusion features Features fused with floating images Specifically: As , As and The cross-attention matrix is ​​calculated by inputting the cross-attention layer, and then the floating image sequence is processed based on the cross-attention matrix. The features in the image are weighted and aggregated to generate a fixed image sequence. Related fixed image feature fusion sequences Meanwhile, for fixed image sequences The features in the data are weighted and aggregated to generate a sequence of floating images. Related floating image feature fusion sequences Finally, and through The output after the operation is a fixed image fusion feature. Features fused with floating images .

5. The nonlinear image registration method based on attention fusion according to claim 4, characterized in that, The registration layer will float the image. Through dense deformation field Sampling is performed to obtain a fixed image. Aligned registration images Specifically, this involves using bilinear sampling or cubic interpolation sampling, based on the dense deformation field. For floating images Remapping is performed on the pixels of a fixed image. Its floating image The corresponding position in is , If any mapped position exceeds the floating image The boundaries are processed using zero-fill or mirror fill, and the final output is a registered image. Otherwise, directly output the registered image. .

6. The nonlinear image registration method based on attention fusion according to claim 5, characterized in that, The dual-branch multi-scale feature extraction network employs multiple loss functions, specifically three types of loss functions jointly optimized, including normalized cross-correlation loss. Deformation field difference loss and resistance to folding loss ; Normalized cross-correlation loss Measuring the registered images With fixed image The similarity is expressed by the formula: in, For covariance, Standard deviation, For batch size, The smaller the value, the more similar the registered image is to the fixed image; Deformation field difference loss Measuring the predicted deformation field That is, dense deformation field With real deformation field Differences, using The loss is calculated using the following formula: in, These represent the image logarithm of the input dual-branch multi-scale feature extraction network, and the height and width of the input floating and fixed images, respectively. Anti-folding loss The formula is: in, for Jacobian matrix, When the penalty is triggered, and Enter the height and width of a fixed or floating image; Total loss function: weighted summation of three types of losses, the formula is: in, , , This is a hyperparameter.

7. A nonlinear image registration system based on attention fusion, comprising a computer program and a processor for processing the computer program, characterized in that: The processor executes a computer program to implement the steps of the method described in any one of claims 1-6.

Images