A medical image registration method combined with semantic segmentation

Abstract

The present application belongs to the field of computer vision of deep learning technology, and provides a medical image registration method combined with semantic segmentation. The present application firstly constructs a shared feature extraction network, and the extracted features are used for registration of deformation field and prediction of segmentation map. In the decoding stage, the bidirectional deformation field is used as a bridge to twist the features of the moving image and the fixed image to each other's segmentation branch, so as to gradually supplement additional context information at multiple levels to realize segmentation. By establishing such an entangled correspondence, the segmentation loss can indirectly regularize the registration, so that the semantic layout of the segmentation can be accurately projected to improve the final registration performance. In addition, a position-related calculation for registration is designed to more easily capture the spatial correlation of the image from the shared features.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and relates to deep learning and computer vision, particularly to a medical image registration method based on joint semantic segmentation. Background Technology

[0002] Deformable medical image registration has numerous applications in clinical practice, including but not limited to clinical case tracking and surgical planning navigation. In recent years, deep learning technology has driven the development of medical image registration, with researchers proposing many deep learning-based registration methods. Since obtaining large amounts of labeled medical data is difficult, these deep learning-based registration methods are often optimized through unsupervised methods. While unsupervised methods can train the network by maximizing the similarity of image pairs, registration performance can plateau due to the lack of guidance from regions of interest (ROIs). Furthermore, medical image annotation typically requires professionally trained personnel, and obtaining large-scale labeled data is time-consuming and labor-intensive.

[0003] Image registration methods for joint semantic segmentation can address the problem of insufficient labeled data by leveraging the complementarity of segmentation and registration. Specifically, registration aligns the segmentation results of a moving image with those of a fixed image, providing additional supervision information to the registration network by maximizing the alignment. Simultaneously, this approach involves unlabeled images in the training of the segmentation network, improving the generalization ability of the segmentation model. Existing models focus on designing joint loss functions to achieve joint learning. In these models, there is no explicit feature interaction between the two tasks. It is well known that reasonable interaction helps the model capture additional key features to improve the network's generalization ability; furthermore, reasonable interaction can guide the network to optimize in the desired direction. Moving and fixed images are highly correlated, and their semantic information can be mutually utilized. The deformation field in registration can transform the contextual information of moving and fixed images, providing additional semantic features for segmentation and thus improving segmentation performance. At the same time, this transformation operation drives the deformation field to project a semantic layout from the moving image to the fixed image, which can improve the final registration performance. Summary of the Invention

[0004] The technical problem to be solved by this invention is to make up for the lack of accuracy in current medical image registration by proposing a medical image registration method based on joint semantic segmentation, which can significantly improve registration performance by using a very small number of image annotations.

[0005] This invention first constructs a shared feature extraction network, where the extracted features are simultaneously used for registration deformation field and segmentation map prediction. During the decoding stage, a bidirectional deformation field is used as a bridge to warp features from both the moving and stationary images onto each other's segmentation branches, thereby progressively supplementing additional contextual information at multiple levels to achieve segmentation. By establishing this entangled correspondence, the segmentation loss can indirectly regularize the registration, ensuring it accurately projects the semantic layout of the segmentation, thus improving the final registration performance. Furthermore, location-related computations for registration are designed to more easily capture the spatial correlation of the images from the shared features.

[0006] The technical solution of this invention:

[0007] A joint semantic segmentation method for medical image registration comprises three parts: a shared feature extraction network, a coupled decoder, and a segmentation decoder. The shared feature extraction network extracts features from both moving and stationary images, which are then input into the coupled decoder and segmentation decoder. The coupled decoder predicts the deformation field increment and coupling features, where the deformation field increments are summed step-by-step to obtain the final deformation field used for registration. The coupling features are then input into the segmentation decoder. The segmentation decoder predicts the segmentation results for both moving and stationary images. The specific steps are as follows:

[0008] Step (1) Construct a shared feature extraction network

[0009] The feature extraction network uses two parameter-shared 3D residual networks as its backbone to extract features from moving and stationary images, respectively. Different layers of the network output features at different scales. The features of the first layer are extracted by ordinary 3D convolution, while the features of the remaining layers are extracted by several residual modules, ultimately yielding four sets of moving image feature maps. i∈[1,4], and feature maps of 4 fixed images i∈[1,4].

[0010] Step (2) Constructing the coupled decoder

[0011] The coupled decoder consists of several coupled decoding modules, each serving two purposes: first, providing coupling features for the segmentation decoder; and second, predicting the deformation field. Each coupled decoding module corresponds one-to-one with the features extracted by the feature extraction network, and each level of the coupled decoding module is sequentially connected. Besides processing the fourth-level features (…),… and Apart from the fourth-level coupled decoding module, the remaining coupled decoding modules have four inputs: and in, and These represent the positive and negative deformation fields, respectively. They are obtained by upsampling the positive and negative (bidirectional) deformation fields output by the previous coupled decoding module through trilinear interpolation. and These represent the features of the i-th level moving image and the i-th level fixed image extracted by the feature extraction network, respectively.

[0012] The features obtained in step (1) and The input is fed into the fourth-level coupled decoding module, and the output is... and Will and Upsampling, and then combining with the features obtained in step (1) The inputs are fed together into the third-level coupled decoding module, which outputs another set of data. and This process continues until the signal is sent to the first-level coupling decoding module. Except for the fourth-level coupling decoding module, each of the remaining coupling decoding modules utilizes the positive and negative deformation fields output by the previous-level coupling decoding module. Dewarp moving image features and fixed image features The warped features are input into the corresponding level of the segmentation decoder as coupled features; specifically, using... Come to Perform a warp operation and then transfer the warped features. As complementary information, it is input into the corresponding level of the segmentation decoder of the fixed image; similarly, using right Perform a warp operation, and then obtain the features. The input is fed into the corresponding level of the segmentation decoder for the moving image.

[0013] Step (3) Feature Correlation Calculation

[0014] To measure the similarity between each voxel and its neighboring voxels, the warped features are... and fixed image features Point-by-point correlation calculations are performed to obtain the positive correlation matrix. Based on the positive correlation matrix, the displacement increment between the moving image and the stationary image at the current level is determined. When calculating the correlation matrix, positional information is embedded into the features. Let E be the positional encoding and r be the displacement radius. The feature correlation calculation formula is as follows:

[0015]

[0016] in E represents the learnable feature map initialized to 0, and x, y, and Z represent the coordinate indices of the feature map in the three directions, respectively. The feature map is moved point by point along the x, y, and z directions with a radius r. Each move generates a one-dimensional correlation map, resulting in a final dimension of (2r+1). 3 The correlation matrix;

[0017] Because a two-way deformation field needs to be generated. and Correlation calculations also need to be performed between them to obtain the inverse correlation matrix; the calculation method is the same as for the positive correlation matrix, only F1 needs to be replaced with F2 replaced with

[0018] Step (4) Deformation field prediction

[0019] After obtaining the positive and inverse correlation matrices, two field estimators with shared parameters are used to estimate the positive and negative deformation field increments of the current stage. Each of these field estimators consists of three residual modules. Specifically, each field estimator extracts features from its corresponding correlation matrix and outputs a three-channel feature map. The values ​​of the three channels represent the displacement increments of each voxel in the x, y, and z directions of the three-dimensional coordinate system, respectively. The two obtained deformation field increments are added to the positive and negative deformation fields output by the previous stage's coupled decoding module to obtain two new deformation fields, which are then input into the next stage's coupled decoding module. This process generates deformation fields from coarse to fine.

[0020] Step (5) Constructing the segmentation decoder

[0021] Each stage of the segmentation decoder contains three inputs: the output features of the previous stage segmentation decoder, the features extracted by the feature extraction network, and the coupled features obtained in step (2). These three parts are concatenated and subjected to two convolution operations. The resulting features are then trilinearly upsampled and input into the next stage segmentation decoder. The weights in the segmentation decoders for fixed and moving images are shared. The number of categories in the segmentation task is set to N, meaning the total number of semantic categories is N. Therefore, the last stage segmentation decoder outputs a feature map with N channels. A softmax operation is performed along the channel direction to obtain the probability value of each category at all locations in the image. The category with the highest probability value is taken as the segmentation result for each point. A joint loss function is constructed using the segmentation results to provide additional supervision information for registration.

[0022] Step (6) Construct the loss function

[0023] The loss function consists of three parts: registration loss, segmentation loss, and joint loss.

[0024] Registration loss L regComposed of two components—similarity and deformation field smoothing—this method aims to align moving and stationary images while ensuring the smoothness of the deformation field. It uses locally normalized cross-correlation as the similarity function and L2 regularization of the deformation field gradient as the smoothing term. The specific calculation formula is as follows:

[0025]

[0026] Where LNCC represents locally normalized cross-correlation, ω represents the window size of locally normalized cross-correlation, and F and These represent the fixed image and the moved image after warping, respectively. λ represents the gradient of the deformation field, and λ1 is the hyperparameter that balances these two terms.

[0027] Segmentation loss function L seg Let L be a mixture function consisting of weighted cross-entropy and Dice, where L is the weighted cross-entropy. wce (S, S) * The formula for calculating ) is:

[0028]

[0029] Where N represents the total number of semantic categories, n represents the channels for the corresponding category, P represents the number of voxels in each channel, and w n S represents the ratio of all voxels to voxels in the nth class, and S is the segmentation result. * Represents the segmentation label. Dice loss function L dice (S, S) * The expression for ) is:

[0030]

[0031] For L wce (S, S) * ) and L dice (S, S) * ), with the same weight, i.e., L seg =L wce (S, S) * )+L dice (S,S * ).

[0032] Joint loss L joint Using Dice loss, the joint loss function takes the segmentation result S of a fixed image as input. F Segmentation result of the moving image after warping M °φ.

[0033] Step (7) Network Training

[0034] Implemented using the PyTorch framework and optimized with the Adam algorithm, the training batch size is 1, the initial learning rate is set to 0.0001, and training is performed iteratively for 100 epochs. The first m epochs are trained using registration and segmentation loss functions, while the remaining epochs are trained using segmentation, registration, and joint loss functions. The dataset used is the publicly available OASIS dataset. x labeled images are randomly selected from the dataset as stationary images, and all other images are used as moving images. The moving images are randomly divided into training, validation, and test sets for training, validation, and testing of the entire network, respectively.

[0035] After training, the moving and stationary images are input into the network, which outputs a deformation field. The warped image is obtained from the deformation field.

[0036] The beneficial effects of this invention are as follows: This invention fully utilizes the complementarity of semantic segmentation and image registration. By progressively coupling the segmentation of moving and fixed images, an entangled relationship between registration and segmentation is established, improving the registration performance of deformable medical images and possessing strong application value. A correlation calculation method for location embedding is designed, which, while measuring feature similarity, provides spatial information for the field estimator, making the estimation of the deformable field more accurate. Attached Figure Description

[0037] Figure 1 This is a diagram of the entire network structure of the present invention. The first half is a shared feature extraction network. In the second half, the upper and lower branches are segmentation decoders for moving images and fixed images, respectively, and the middle branch is a coupling decoder. CD is the coupling decoding module.

[0038] Figure 2 The detailed structure of the coupled decoder is shown below. Corr is used for correlation calculation, Estimator is the forward field estimator, and Inv-Estimator is the inverse field estimator. This indicates that pixels are added one by one.

[0039] Figure 3 Example of input moving image;

[0040] Figure 4 Example of inputting a fixed image;

[0041] Figure 5 This is an example of the registration result.

[0042] Figure 6 This is an example of a deformation field. Detailed Implementation

[0043] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0044] A medical image registration method based on joint semantic segmentation is as follows:

[0045] (1) Constructing the overall network framework

[0046] The network's overall framework consists of three parts: a shared feature extraction network, a coupled decoder, and a segmentation decoder. Among them, in... Figure 1 The first part represents the shared feature extraction network, which is responsible for extracting features from both moving and stationary images. The extracted features are then input into the coupled decoder and the segmentation decoder. Figure 1 The CD module in the diagram represents the coupled decoding module, which is responsible for predicting the deformation field increment and coupling features. The deformation field increment is added step by step to obtain the final deformation field used for registration, and the coupling features are input into the segmentation decoder. Figure 1 The middle and latter half represent the segmentation decoder, which is responsible for predicting the segmentation results for moving and stationary images, respectively.

[0047] (2) Constructing a shared feature extraction network

[0048] The feature extraction network uses two parameter-shared 3D residual networks as its backbone to extract features from both moving and stationary images. These extracted features are then used for both registration and segmentation. Different layers of the network output features representing 1 / 2, 1 / 4, and 1 / 8 of the image size. The first layer's features are extracted by two 3×3×3 convolutions, while the remaining features are extracted by... Figure 1 The resblock1-3 segments are extracted, and their corresponding channel numbers are 8, 16, 32, and 64, respectively.

[0049] (3) Constructing a coupled decoder

[0050] The coupled decoder consists of several coupled decoding modules, each serving two purposes: first, to provide coupling features for the segmentation decoder; and second, to predict the deformation field. For example... Figure 2 As shown, besides being responsible for processing level 4 features ( and Apart from the fourth-level coupled decoding module, the remaining coupled decoding modules have four inputs: and in, and These represent the positive and negative deformation fields, respectively. They are obtained by upsampling the positive and negative (bidirectional) deformation fields output by the previous coupled decoding module through trilinear interpolation. and Let represent the features of the i-th level moving image and the i-th level stationary image extracted by the feature extraction network, respectively. The features obtained in step (1) are... and The input is fed into the first coupled decoding module, and the output is... and Will and Upsampling, and then combining with the features obtained in step (1) The inputs are combined into the third-level coupled decoding module, and the output is... and This process continues until the signal is sent to the first-level coupling decoding module. Except for the fourth-level coupling decoding module, each of the remaining coupling decoding modules utilizes the positive and negative deformation fields output by the previous-level coupling decoding module. Dewarp moving image features and fixed image features The warped features are input into the segmentation decoder as coupling features, thus giving the segmentation network richer semantic information; specifically, using... Come to Perform a warp operation and then transfer the warped features. As complementary information, it is input into the corresponding level of the segmentation decoder of the fixed image; similarly, using right Perform a warp operation, and then obtain the features. The input is fed into the corresponding level of the segmentation decoder for the moving image.

[0051] (4) Feature correlation calculation

[0052] The purpose of calculating feature correlation is to measure the feature similarity between moving and stationary images, thereby predicting the deformation field more accurately at each level. To measure the similarity between each voxel and its neighboring voxels, the warped features are... and fixed image features Perform correlation calculations point by point to obtain the correlation matrix, i.e. Figure 2 The Corr module in the code. Because it requires generating a two-way deformation field, and Correlation calculations also need to be performed between them to obtain the inverse correlation matrix, i.e. Figure 2 Inv-Corr in the context of correlation matrix calculations. Based on the correlation matrix, the displacement increment between the moving and stationary images at the current layer can be determined. Embedding positional information into the features during correlation matrix calculation helps the field estimator understand the spatial relationships of voxels and reduces semantic confusion caused by shared encoders. Let E be the positional code and r be the displacement radius; the feature correlation calculation formula is:

[0053]

[0054] in E represents the learnable feature map initialized to 0, where x, y, and z represent the coordinate indices of the feature map in the three directions. The feature map is moved point-by-point along the x, y, and z directions with a radius r. Each move generates a one-dimensional correlation map, ultimately resulting in a map with a dimension of (2r+1). 3 The correlation matrix. Considering the computational cost, r can be set to 1. The calculation method for the inverse correlation matrix is ​​similar, simply replacing F1 with... F2 replaced with That's all.

[0055] (5) Deformation field prediction

[0056] Figure 2 In this model, the Estimator represents the deformation field prediction model. After obtaining the positive and inverse correlation matrices, two field estimators with shared parameters are used to estimate the positive and negative deformation field increments of the current stage. Each of these field estimators consists of three residual modules. Specifically, each field estimator extracts features from its corresponding correlation matrix and outputs a three-channel feature map. The values ​​of the three channels represent the displacement increments of each voxel in the x, y, and z directions of the three-dimensional coordinate system, respectively. The two obtained deformation field increments are added to the positive and negative deformation fields output by the previous stage coupled decoding module to obtain two new deformation fields. In this way, the deformation fields of the registration branch are predicted from coarse to fine. The low-resolution deformation field is responsible for global registration, while the high-resolution deformation field focuses on fine-grained registration. This combination of deformation fields effectively solves the problem of complex misalignments in real-world scenarios.

[0057] (6) Constructing a segmentation decoder

[0058] Each stage of the segmentation decoder contains three inputs: the output features of the previous stage segmentation decoder, the features extracted by the feature extraction network, and the coupled features obtained in step (2). These three parts are concatenated and subjected to two convolution operations. The resulting features are then trilinearly upsampled and input into the next stage segmentation decoder. The weights in the segmentation decoders for fixed images and medical images are shared. This invention involves segmentation tasks for 36 categories. Therefore, the final stage decoder outputs a 36-channel feature map, which is then subjected to a softmax operation along the channel direction to obtain the probability values ​​of each category at all locations in the image. The category with the highest probability value is taken as the segmentation result for each point. A joint loss function is constructed using the segmentation results to provide additional supervision information for registration.

[0059] (7) Construct the loss function

[0060] The loss function consists of three parts: registration loss, segmentation loss, and joint loss.

[0061] The registration loss consists of two terms: similarity and deformation field smoothing. It aims to align moving and stationary images while ensuring the smoothness of the deformation field. Locally normalized cross-correlation is used as the similarity function, and L2 regularization of the deformation field gradient is used as the smoothing term. The specific calculation formula is as follows:

[0062]

[0063] Where LNCC represents locally normalized cross-correlation, ω represents the window size of locally normalized cross-correlation, and F and These represent the fixed image and the warped medical image, respectively. λ represents the gradient of the deformation field, and λ1 is the hyperparameter that balances these two terms.

[0064] Segmentation loss function L seg Let be a mixture function consisting of weighted cross-entropy and Dice. The formula for calculating weighted cross-entropy is:

[0065]

[0066] Where N represents the total number of semantic categories, n represents the channels for the corresponding category, P represents the number of voxels in each channel, and w n S represents the ratio of voxels in class n to all voxels, and S is the segmentation result. * This represents the segmentation label. The expression for the Dice loss function is:

[0067]

[0068] For these two segmentation loss functions, they have the same weight, namely L. seg =L wce (S,S * )+L dice (S,S * ).

[0069] Joint loss L joint Using Dice loss, the joint loss function takes the segmentation result S of a fixed image as input. F Segmentation results of moving images after warping

[0070] (8) Training methods

[0071] Implemented using the PyTorch framework and optimized with the Adam algorithm, the training batch size was 1, the initial learning rate was set to 0.0001, and training was performed iteratively for 100 epochs. The first 5 epochs used registration and segmentation loss functions for training, and the last 95 epochs used segmentation, registration, and joint loss functions for training. The dataset used was the publicly available OASIS dataset, which contains 414 3D brain MRI medical images. The image size was cropped from 160×192×244 to 160×192×144 to remove unnecessary black backgrounds and save GPU memory. Five labeled images were randomly selected from the dataset as stationary images, and the remaining 409 images were used as moving images. The labels on the moving images were not used in the experiment to simulate a scenario with only a few labels. The moving images were randomly divided into 255, 22, and 132 pairs as training, validation, and test sets, respectively. Therefore, a total of 1275, 110, and 660 image pairs were used for training, validation, and testing. After training, the model can be used to predict the deformation field. Figure 3 and Figure 4 Both moving and stationary images are input into the network, and the output is... Figure 6 The deformation field shown, and then the deformation field acts on Figure 3 The image obtained after warping is as follows: Figure 5 As shown.

Claims

1. A medical image registration method in combination with semantic segmentation, characterized by, The whole network framework is composed of three parts of a shared feature extraction network, a coupling decoder and a segmentation decoder; wherein the shared feature extraction network is responsible for extracting the features of the moving image and the fixed image, and the extracted features are input into the coupling decoder and the segmentation decoder; the coupling decoder is responsible for predicting the deformation field increment and the coupling feature, wherein the deformation field increment is added step by step to obtain the final deformation field for registration, and the coupling feature is input into the segmentation decoder; the segmentation decoder is responsible for predicting the segmentation result map of the moving image and the fixed image; the specific steps are as follows: Step (1) constructing a shared feature extraction network The feature extraction network adopts a 3D residual network with two parameter sharing as a backbone to extract features of the moving image and the fixed image respectively; different layers of the network output different scale features of the image, wherein the features of the first layer are extracted by a common 3D convolution, and the features of the remaining layers are extracted by a plurality of residual modules respectively, and finally four groups of moving image feature maps are obtained and four groups of fixed image feature maps Step (2) constructing a coupling decoder The coupling decoder is composed of several coupling decoding modules. The coupling decoding module has two functions. The first is to provide coupling features for the segmentation decoder, and the second is to be responsible for the prediction of the deformation field. The coupling decoding module is one-to-one corresponding to the features extracted by the feature extraction network, and each level of the coupling decoding module is sequentially connected. Except for the fourth level coupling decoding module responsible for processing the fourth level features and , the remaining coupling decoding modules have four inputs: and wherein, and represent the positive and negative deformation fields, respectively, which are obtained by upsampling the positive and negative bidirectional deformation fields output by the upper level coupling decoding module through trilinear interpolation; and represent the features of the i-th level moving image and the i-th level fixed image extracted by the feature extraction network, respectively. the features obtained in step (1) are input into the fourth stage coupled decoding module, and output and the features obtained in step (1) are input into the fourth stage coupled decoding module, and output and the features obtained in step (1) are input into the fourth stage coupled decoding module, and output and the features obtained in step (1) are input into the fourth stage coupled decoding module, and output and the features obtained in step (1) are input into the fourth stage coupled decoding module, and output and the features obtained in step (1) are input into the fourth stage coupled decoding module, and output and the features obtained in step (1) are input into the fourth stage coupled decoding module, and output and the features obtained in step (1) are input into the fourth stage coupled decoding module, and output and the features obtained in step (1) are input into the fourth stage coupled decoding module, and output and the features obtained in step (1) are input into the fourth stage coupled decoding module, and output Step (3) feature correlation calculation To measure the similarity between each voxel and its neighboring voxels, the features after warping and the fixed image features The correlation is calculated point by point to obtain a positive correlation matrix; according to the positive correlation matrix, the displacement increment between the moving image and the fixed image at the current level is determined; when the correlation matrix is calculated, the position information is embedded into the features, E is the position coding, r is the displacement radius, and the feature correlation calculation formula is: wherein E represents a learnable feature map initialized to 0, x, y, z represent the coordinate indexes of the feature map in three directions respectively; the feature map is moved point by point along the three directions of x, y, z with a radius r, and each movement generates a one-dimensional correlation graph, and finally a correlation matrix with a dimension of (2r+1) 3 is generated. Since the bidirectional deformation field needs to be generated, and The inverse correlation matrix is obtained by calculating the correlation between and Step (4) deformation field prediction After obtaining the positive correlation matrix and the inverse correlation matrix, two parameter-shared field estimators are used to estimate the positive and negative deformation field increments of the current level, and each field estimator is composed of three residual modules; specifically, each field estimator extracts the features in the corresponding correlation matrix and outputs a three-channel feature map, and the values of the three channels represent the displacement increments of each voxel in the x, y and z directions of the three-dimensional coordinate system; the two obtained deformation field increments are added to the positive and negative deformation fields output by the coupling decoding module of the previous level to obtain two new deformation fields input into the coupling decoding module of the next level, so as to generate the deformation field from coarse to fine; Step (5) constructing a segmentation decoder Each level of the segmentation decoder contains three parts of input, which are the output features of the segmentation decoder of the previous level, the features extracted by the feature extraction network and the coupling features obtained in step (2); after splicing the three parts, two convolution operations are performed, and then the obtained features are three-linearly up-sampled and input into the segmentation decoder of the next level; wherein the weights in the segmentation decoders of the fixed image and the moving image are shared; the class of the segmentation task is N, that is, the total number of semantic classes is N, therefore, the segmentation decoder of the last level outputs a feature map with N channel numbers, which is subjected to softmax operation along the channel direction to obtain the probability value of each class at all positions of the image, and each point takes the class with the maximum probability value as the segmentation result of the point; a joint loss function is constructed by using the segmentation result to provide additional supervision information for registration; step (6) constructing a loss function The loss function is composed of registration loss, segmentation loss and joint loss; registration loss L reg consisting of similarity and deformation field smoothing terms, aiming to align the moving image and the fixed image and guarantee the smoothness of the deformation field; the local normalized cross correlation is used as the similarity function and the L2 regularization of the deformation field gradient is used as the smoothing term, and the specific calculation formula is: where LNCC represents the local normalized cross-correlation, ω represents the window size of the local normalized cross-correlation, F and respectively represent the fixed image and the moving image after warping, represents the gradient of the deformation field, λ1 is a hyperparameter balancing the two terms; Loss function L of segmentation seg The loss function L is a hybrid function composed of weighted cross-entropy and Dice, and the weighted cross-entropy L wce The calculation formula of (S, S * ) is: where N represents the total number of semantic categories, n represents the channel corresponding to the category, P represents the number of voxels in each channel, w n represents the proportion of all voxels to voxels in the nthcategory, S is the segmentation result, S * represents the segmentation label; the Dice loss function L dice (S, S * ) is expressed as: For L wce (S,S * ) and L dice (S,S * ), have the same weight, i.e. L seg = L wce (S,S * )+ L dice (S,S * ); Joint loss L joint Using the Dice loss, the input to the joint loss function is the segmentation result S of the fixed image F and the segmentation result of the moving image after warping Step (7) network training Based on the Pytorch framework, the Adam algorithm is used for optimization, the training batch size is 1, the initial learning rate is set to 0.0001, and the iterative training is 100 epochs; the first m epochs are trained using the registration and segmentation loss functions, and the remaining all epochs are trained using the segmentation, registration and joint loss functions; the public OASIS dataset is used, x labeled images are randomly selected from the dataset as fixed images, and the remaining all images are used as moving images; the moving images are randomly divided into training set, validation set and test set, which are used for training, validation and testing of the whole network respectively; After training, the moving and stationary images are input into the network, which outputs a deformation field. The warped image is obtained from the deformation field.

Images