System for deformation registration of MRI brain nerve images based on double-flow cross network

Abstract

The present application relates to a deformation registration system of MRI brain nerve images based on a double-flow cross network. The system comprises an acquisition module, a training module and an output module. The acquisition module is configured to acquire an MRI brain nerve image dataset. The training module is configured to input the dataset into a deformation image registration model based on a double-flow cross network, obtain a forward registration deformation field and a reverse registration deformation field, transform the obtained forward registration deformation field and reverse registration deformation field through a spatial transformation network, and generate a forward registration image and a reverse registration image. The output module is configured to output the forward registration image and the reverse registration image of two MRI brain nerve images to be deformation registered using the deformation image registration model learned in the training module. The system can automatically complete the deformation registration of any two 3D MR brain nerve images. When the images contain labels, brain atlas label migration can also be achieved, so that another image is automatically labeled.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, and particularly relates to a deformation registration system for MRI brain nerve images based on a dual-stream cross network. BACKGROUND

[0002] Magnetic resonance imaging technology (MRI) is a technology developed in the 1980s, and has become the imaging gold standard for observing and evaluating brain structure. In particular, high-field (3T and above) three-dimensional T1-weighted images not only provide clear images of brain gray matter and white matter structures, but also can detect brain nerve anatomical structures as small as 0.67mm. Through quantitative calculation by brain atlas labels or brain nerve structure segmentation results, reliable measurement results can be generated. Through registration of MRI brain nerve images, i.e. through a series of spatial transformations to match the spatial positions of two MRI brain nerve images, the changes in the patient's disease from the latent period to the onset period can be objectively recorded.

[0003] At present, the registration method of MRI brain nerve images mainly includes a traditional registration method and a registration method based on deep learning. The traditional registration method needs to perform iterative optimization for each pair of images, and therefore has problems such as large amount of calculation and time consumption. The registration method based on deep learning refers to training a model through a large amount of data, and then using the trained model to register a new pair of images, so that the registration speed is much faster than the traditional registration method.

[0004] Therefore, in order to shorten the registration time, a deformation registration system for MRI brain nerve images based on deep learning is needed.

[0005] The above statements of background art are only for the convenience of deep understanding of the technical solutions of the present application (the technical means used, the technical problems solved and the technical effects generated, etc.), and should not be regarded as acknowledging or in any form implying that the message constitutes the prior art known to those skilled in the art. SUMMARY

[0006] In view of the defects in the prior art, the present application provides a deformation registration system for MRI brain nerve images based on a dual-stream cross network, which can quickly generate a registration deformation field by calling a trained network model, and significantly shorten the registration time.

[0007] According to one embodiment of the present application, a deformation registration system for MRI brain nerve images based on a dual-flow cross network is provided, characterized by comprising the following modules: an acquisition module configured to acquire an MRI brain nerve image dataset; a training module configured to input the MRI brain nerve image dataset into a deformation image registration model based on a dual-flow cross network, obtain a forward registration deformation field and a reverse registration deformation field, transform the obtained forward registration deformation field and reverse registration deformation field through a spatial transformation network, and generate a forward registration image and a reverse registration image using a nearest neighbor interpolation method; an output module configured to output the forward registration image and the reverse registration image of two MRI brain nerve images to be deformation registered using the deformation image registration model learned in the training module; wherein the deformation image registration model based on the dual-flow cross network comprises two deformation image registration networks with the same structure, the first MRI brain nerve image data sample and the second MRI brain nerve image data sample in the MRI brain nerve image dataset are input into the two deformation image registration networks respectively; each of the two deformation image registration networks comprises an encoder and a decoder; and the decoders of the two deformation image registration networks are cross-connected.

[0008] Preferably, each deformation image registration network takes a 3D U-Net framework with encoding-decoding as a backbone network, and the number of levels of the encoder of each deformation image registration network is equal to the number of levels of the decoder; the encoder and the decoder of each deformation image registration network are connected by a skip connection, and each level of the decoder of one deformation image registration network is cross-connected with the corresponding level of the decoder of the other deformation image registration network; an attention mechanism module is arranged between each level of the decoder of each deformation image registration network; and the weights of the encoders of the two deformation image registration networks are shared.

[0009] Preferably, the deformation image registration model based on the dual-flow cross network is obtained based on minimizing the following loss function to obtain an optimal model:

[0010]

[0011] wherein X and Y represent the first MRI brain nerve image data sample and the second MRI brain nerve image data sample, denotes a deformation sampling operation, denotes a deformation field obtained by registering X to Y, denotes a deformation field obtained by registering Y to X, and the hyperparameter λ represents a parameter used to balance the similarity measure and the smoothing regular term.

[0012] Preferably, the encoder of each morphing image registration network comprises a first-level encoder to an Nth-level encoder, and outputs feature maps with gradually reduced sizes; the decoder of each morphing image registration network comprises a first-level decoder to an Nth-level decoder, and the nth-level decoder is connected with the (N-n+1)th-level encoder by skip connection, the nth-level decoder is configured to receive the feature map from the (N-n+1)th-level encoder and perform feature extraction, generate the feature map of the nth-level decoder and exchange it with the feature map of the nth-level decoder generated by the nth-level decoder of another morphing image registration network, perform feature fusion on the received feature map of the (N-n+1)th-level encoder and the feature map of the nth-level decoder of another morphing image registration network obtained after exchange, and output a new feature map; wherein n≤N, and N is an integer.

[0013] Preferably, the first-level encoder to the Nth-level encoder each comprises a convolutional structure of a 3x3x3 convolutional layer with a step size of 1 and a 3x3x3 convolutional layer with a step size of 2; and the first-level decoder to the Nth-level decoder each comprises a convolutional structure of a 3x3x3 convolutional layer with a step size of 1 and a 2x2x2 transposed convolutional layer with a step size of 1.

[0014] Preferably, the morphing image registration model based on the dual-flow cross network further comprises a multi-scale attention feature fusion module, the multi-scale attention feature fusion module comprises an attention feature fusion module; the attention feature fusion module is arranged between each level of the decoder of each morphing image registration network, and is configured to perform level-by-level or cross-level fusion processing on two feature maps output by each level of the decoder of the morphing image registration network.

[0015] Preferably, the level-by-level or cross-level fusion processing on the two feature map information output by each level of the decoder is performed by using the following feature fusion equation:

[0016]

[0017] wherein X and Y represent any two feature maps output by each level of the decoder, A represents an attention mechanism, represents point-by-point addition, represents point-by-point multiplication.

[0018] Preferably, the multi-scale attention feature fusion module further comprises an up-sampling module, the up-sampling module is configured to make the two feature maps output by each level of the decoder have the same size by up-sampling processing.

[0019] Preferably, the acquisition module is configured to preprocess the collected MRI brain nerve image to obtain the MRI brain nerve image dataset, and the preprocessing includes at least one of format conversion, steering, ACPC alignment, bias field correction, skull removal, resampling, size adjustment, and label generation.

[0020] According to one embodiment of the present application, a computer readable storage medium is provided, and computer instructions are stored on the computer readable storage medium, and the computer instructions are executed by a processor to implement the following steps: acquiring an MRI brain nerve image dataset; inputting the MRI brain nerve image dataset into a deformation image registration model based on a double-flow cross network to obtain a forward registration deformation field and a reverse registration deformation field, transforming the obtained forward registration deformation field and reverse registration deformation field through a spatial transformation network, and generating a forward registration image and a reverse registration image by using a nearest neighbor interpolation method; using a learned deformation image registration model in a training module, outputting the forward registration image and the reverse registration image of two MRI brain nerve images to be deformation registered; wherein the deformation image registration model based on the double-flow cross network includes two deformation image registration networks with the same structure, and the two deformation image registration networks are respectively inputted with a first MRI brain nerve image data sample and a second MRI brain nerve image data sample in the MRI brain nerve image dataset; the two deformation image registration networks respectively include an encoder and a decoder; and the decoders of the two deformation image registration networks are cross-connected.

[0021] The present application adopts the above technical solution, and has the following beneficial effects:

[0022] The deformation registration system for MRI brain nerve images based on the double-flow cross network can automatically complete deformation registration of any two 3D MR brain nerve images, and has strong universality, high intelligence, and does not depend on artificial or expert labels, nor is limited to MRI brain data of a single source or a single scanning protocol.

[0023] In addition, when the source image or the target image contains a label, the system of the present application can also realize brain atlas label transfer, so that the other image is automatically labeled.

[0024] Compared with a traditional numerical iterative optimization algorithm, the system of the present application can obtain a registration deformation field by calling a pre-trained model, and significantly shortens the registration time.

[0025] Compared with the traditional unsupervised learning deformation image registration method, the application does not deliberately distinguish between moving images and fixed images, but simultaneously learns bidirectional deformation registration. In the encoder, the deformation image registration model based on the double-flow cross network uses two deformation image registration networks to extract features from the two images respectively, and then in the decoding stage, the upper and lower deformation image registration networks cross and fuse features at each level to capture large-scale deformation information.

[0026] Finally, the system of the application is provided with a multi-scale feature fusion module based on an attention mechanism, which is applied to the two deformation image registration networks respectively, and calculates the final deformation field by fusing high-level and low-level semantic information. BRIEF DESCRIPTION OF DRAWINGS

[0027] Exemplary embodiments of the application will be described in greater detail below with reference to the accompanying drawings. For the sake of clarity, the same components in different drawings are denoted by the same reference numerals. It should be noted that the drawings only serve to illustrate and are not necessarily drawn to scale. In these drawings:

[0028] Figure 1 is a block diagram of the configuration of a deformation registration system for MRI brain nerve images based on a double-flow cross network according to an embodiment of the application.

[0029] Figure 2 is a pre-processing flowchart of an internal data set according to an embodiment of the application.

[0030] Figure 3 is a schematic diagram of a deformation image registration model based on a double-flow cross network according to an embodiment of the application.

[0031] Figure 4 is a schematic diagram of the AFF module fusing two feature maps according to an embodiment of the application.

[0032] Figure 5 is a working principle diagram of the MAFF module fusing feature maps in a step-by-step fusion manner according to an embodiment of the application.

[0033] Figure 6 is a working principle diagram of the MAFF module fusing feature maps in a cross-level fusion manner according to an embodiment of the application.

[0034] Figure 7 is a working principle diagram of the MAFF module fusing feature maps in another cross-level fusion manner according to an embodiment of the application.

[0035] Figure 8 is a schematic diagram of a deformation image registration model based on a double-flow cross network according to another embodiment of the application.

[0036] Figure 9A is an example of an individual gray scale image; Figure 9B is an example of an individual label image; Figure 9C is an example of a brain template gray scale image; Figure 9D is an example of a brain template label image.

[0037] Figure 10 is a box plot showing the comparison results of different comparison algorithms on 15 anatomical structures.

[0038] Figure 11 is a schematic diagram showing the visualization results of individual and brain template registration by different comparison algorithms.

[0039] Figure 12 is a schematic diagram showing the visualization results of individual and inter-individual bidirectional registration based on the embodiments of the present application. DETAILED DESCRIPTION

[0040] The embodiments of the present application are described in detail below, which are implemented on the premise of the technical solutions of the present application, and detailed implementation manners and specific operation processes are given, but the protection scope of the present application is not limited to the following embodiments.

[0041] Figure 1 is a block diagram of the configuration of a deformation registration system of MRI brain nerve images based on a dual-flow cross network according to an embodiment of the present application. The deformation registration system of MRI brain nerve images based on a dual-flow cross network according to the present application comprises an acquisition module, a training module and an output module. The acquisition module is configured to acquire a MRI brain nerve image dataset. The training module is configured to input the MRI brain nerve image dataset into a deformation image registration model based on a dual-flow cross network, obtain a forward registration deformation field and a reverse registration deformation field, transform the obtained forward registration deformation field and reverse registration deformation field through a spatial transformation network, generate a forward registration image and a reverse registration image by using a nearest neighbor interpolation method. The output module is configured to output the forward registration image and the reverse registration image of two MRI brain nerve images to be deformed and registered by using the deformation image registration model learned in the training module.

[0042] In the following, the processing of each module of the deformation registration system of MRI brain nerve images based on a dual-flow cross network of the present application is described in detail.

[0043] Based on the application applied to three-dimensional brain MRI image, the acquisition module is configured to acquire MRI brain nerve image dataset. In an embodiment, the MRI brain nerve image dataset used is respectively 131 cases of internal 3D high-definition human brain MRI dataset (i.e. internal dataset) and 40 cases of public neuroimaging laboratory probability brain atlas (The LONI Probabilistic Brain Atlas 40, LPBA40) dataset. The acquisition parameters of the internal dataset are shown in Table 1 below, and the acquisition parameters of the LPBA40 dataset are shown in Table 2 below.

[0044] Table 1

[0045] Apparatus Philips Medical Systems Archieva Sequence WIP T1W_3D_TFE_ref CLEAR Field strength 3T TR 11.2 ms TE 5.1 ms Flip angle 8° Slice thickness 0.7 mm Inter-slice gap 0.7 mm Voxel size 0.67 x 0.67 x 0.7 mm 3 <!-- 4 -->]]> Matrix 335 x 335 x 256 voxels

[0046] Table 2

[0047]

[0048] After the 131 cases of internal dataset and the 40 cases of LPBA40 dataset are collected via other external devices (as shown in Table 1 and Table 2), the acquisition module can be configured to pre-process the collected data for network training, and the pre-processing can include at least one, part or all of the operations of format conversion, turning, ACPC alignment, bias field correction, skull removal, resampling, size adjustment and label generation.

[0049] In an embodiment of the application, the LPBA40 dataset has a total of 40 brain images, which have been pre-processed, and the image size is 160x192x160 voxels, and the resolution is isotropic 1mmx1mmx1mm. Each brain image in the LPBA40 dataset contains segmentation labels of 56 brain anatomical structures manually outlined by artificial experts.

[0050] Figure 2 is a pre-processing flowchart of the internal dataset according to an embodiment of the application. As shown in Figure 2 For the internally collected dataset, the original DICOM data is sequentially subjected to format conversion from DICOM format to NIFTI format, turning, ACPC alignment, bias field correction, skull removal. After skull removal, the image resolution can be resampled to isotropic 1mmx1mmx1mm, the image size is adjusted to 160x192x160 voxels, and then 26 anatomical structure segmentation labels for each image are automatically generated by Freesurfer (which is a public software).

[0051] It is worth noting that the purpose of resampling is to make the parameters of the internal dataset and the LPBA40 dataset consistent for comparison, but there is no limit to the resolution of the dataset as long as the GPU memory is sufficient. For sub-millimeter isotropic resolution datasets, the resampling step can be omitted to maintain high-resolution image matching effects. Therefore, in other embodiments of the application, the resampling and resizing steps can be omitted.

[0052] According to the needs of subsequent training, in one exemplary embodiment, the 131 internal datasets and the 40 LPBA40 datasets can be divided into 78, 27, and 26 images and 25, 5, and 10 images for training sets, validation sets, and test sets, respectively. Accordingly, the acquisition module acquires 78 images of the internal dataset and 25 images of the LPBA40 dataset as the MRI brain nerve image dataset, which is only an example, and the application is not limited to the above division number.

[0053] According to the embodiments of the application, the training module is configured to input the MRI brain nerve image dataset into the deformation image registration model based on the dual-flow cross network, obtain the forward registration deformation field and the reverse registration deformation field, transform the obtained forward registration deformation field and reverse registration deformation field through the spatial transformation network, and generate the forward registration image and the reverse registration image using the nearest neighbor interpolation method. Wherein, the deformation image registration model based on the dual-flow cross network includes two deformation image registration networks with the same structure, the first MRI brain nerve image data sample and the second MRI brain nerve image data sample in the MRI brain nerve image dataset are input into the two deformation image registration networks respectively; the two deformation image registration networks each include an encoder and a decoder; the decoders of the two deformation image registration networks are cross-connected.

[0054] Thus, the application performs deformation registration of MRI brain nerve images based on the principle of deep learning. Compared with traditional numerical iterative optimization algorithms, the system of the application can obtain registration deformation field by calling a pre-trained model, significantly shortening the registration time. The deformation registration method based on deep learning can further include a supervised deep learning registration method and an unsupervised deep learning registration method, and the application belongs to a deformation registration based on unsupervised deep learning.

[0055] In most deformation image registration methods based on unsupervised learning, a pair of three-dimensional MR images is often taken as a fixed image and a moving image respectively, and then merged together as an input on the channel, only the single mapping of the moving image to the fixed image is inferred, and the reverse mapping and the differential homeomorphism property are ignored. In the embodiments of the present application, there is no distinction between the moving image and the fixed image, but any two MRI brain nerve image data samples in the MRI brain nerve image dataset are respectively input, so that the two data samples simultaneously learn the bidirectional deformation registration. In the encoder, the deformation image registration model based on the dual-flow cross network of the present application is designed to extract features from the images of the two data samples respectively by using two branches (i.e., deformation image registration networks), and then in the decoding stage, the upper and lower branches cross and fuse the features at each level to capture large-scale deformation information. In the following, the deformation image registration model based on the dual-flow cross network of the present application is described in detail with reference to Figure 3 The deformation image registration model based on the dual-flow cross network according to an embodiment of the present application is illustrated in FIG. 1. As shown in FIG. 1, each deformation image registration network takes a 3D U-Net framework of encoding-decoding as the backbone network. The number of levels of the encoder of each deformation image registration network is equal to the number of levels of the decoder. The encoder and the decoder of each deformation image registration network are connected by a skip connection, and each level of the decoder of one deformation image registration network is cross-connected with the corresponding level of the decoder of the other deformation image registration network. An attention mechanism module is arranged between each level of the decoder of each deformation image registration network. The weights of the encoders of the two deformation image registration networks are shared.

[0056] Figure 3 The deformation image registration model based on the dual-flow cross network according to an embodiment of the present application is illustrated in FIG. 1. As shown in FIG. 1, each deformation image registration network takes a 3D U-Net framework of encoding-decoding as the backbone network. The number of levels of the encoder of each deformation image registration network is equal to the number of levels of the decoder. The encoder and the decoder of each deformation image registration network are connected by a skip connection, and each level of the decoder of one deformation image registration network is cross-connected with the corresponding level of the decoder of the other deformation image registration network. An attention mechanism module is arranged between each level of the decoder of each deformation image registration network. The weights of the encoders of the two deformation image registration networks are shared. Figure 3 According to an embodiment of the present application, the encoder of each deformation image registration network includes a first-level encoder to an Nth-level encoder, and outputs feature maps with decreasing sizes level by level. Specifically, for each level of the encoder, a convolution structure is adopted to output a feature map with a size smaller than that of the previous level, so as to extract high-level features of the input image. Preferably, the first-level encoder to the Nth-level encoder respectively include a convolution structure of a 3×3×3 convolution layer with a step of 1 and a 3×3×3 convolution layer with a step of 2. Through such a convolution structure, the size of the feature map is reduced by half at each layer.

[0057]

[0058] ​The decoder of each morphological image registration network comprises a first-level decoder to an Nth-level decoder, and the Nth-level decoder is connected to the (N-n+1)th-level encoder by a skip connection, the Nth-level decoder is configured to receive the feature map from the (N-n+1)th-level encoder and perform feature extraction, generate the Nth-level decoder feature map of the Nth-level decoder and exchange it with the Nth-level decoder feature map generated by the Nth-level decoder of another morphological image registration network, perform feature fusion on the received (N-n+1)th-level encoder feature map and the exchanged Nth-level decoder feature map of another morphological image registration network, and output a new feature map.

[0059] In the foregoing, N is an integer, and n≤N, n takes values 1, 2, …, N in turn. Preferably, N takes a value of 3 or 4. Figure 3 The schematic diagram of the morphological image registration model based on the dual-stream cross network is shown in the case of N=4. As shown in the figure, the structure of the morphological image registration model based on the dual-stream cross network is modified on the basis of the 3D U-Net, which is composed of a 4-level hierarchical encoder-decoder and a skip connection, uses a dual-stream network that crosses each other, and separates the two processes of feature extraction and feature interaction into the encoder and the decoder, respectively. Figure 3 The structure of the morphological image registration model based on the dual-stream cross network is shown in the figure, which is modified on the basis of the 3D U-Net, which is composed of a 4-level hierarchical encoder-decoder and a skip connection, uses a dual-stream network that crosses each other, and separates the two processes of feature extraction and feature interaction into the encoder and the decoder, respectively.

[0060] Each encoder is designed to be 4 levels, and the weights of the encoders of the upper and lower branches are shared. For each level, two consecutive convolution layers are used to extract high-level features of the input image, including a 3×3×3 convolution layer with a step size of 1 and a 3×3×3 convolution layer with a step size of 2. In this way, the size of the feature map is reduced by half at each layer.

[0061] In the decoder, the feature maps of different layers of the decoders of the two branches are cross-fused respectively, and the feature maps from the encoder and the feature maps of the same scale from the decoder of the opposite branch are merged by a skip connection to form a new feature map. In this way, the feature information of all layers of the two branch decoders is fused by this cross method.

[0062] For example, as shown in the figure, Figure 3As shown, the first-level decoder is connected to the fourth-level encoder in a skip connection, receives the feature map from the fourth-level encoder and performs feature extraction to generate a first-level decoder feature map, which is exchanged with the first-level decoder feature map generated by the first-level decoder of the other branch. The first-level decoder performs feature fusion on the received fourth-level encoder feature map and the exchanged first-level decoder feature map of the other branch and outputs a new feature map. The second-level decoder is connected to the third-level encoder in a skip connection, receives the feature map from the third-level encoder and performs feature extraction to generate a second-level decoder feature map, which is exchanged with the second-level decoder feature map generated by the second-level decoder of the other branch. The second-level decoder performs feature fusion on the received third-level encoder feature map and the exchanged second-level decoder feature map of the other branch and outputs a new feature map, and so on.

[0063] In addition, a Squeeze-and Excitation-Attention (SEA) channel attention mechanism is used between each level of decoder, which can calculate different weights on channels and adaptively recalibrate the feature responses of the channel direction by explicitly modeling the interdependence between channels, so that the network designed by the present application learns to use global information, selectively emphasizes informative features, and captures specific mapping relationships between image pairs. Figure 3

[0064] After the feature map passes through the SEA, a 3x3x3 convolution with a step size of 1 and a 2x2x2 transpose convolution layer are used to upsample the features to twice the original size to expand the size of the feature maps of different sizes and perform feature fusion in the next step. In an exemplary embodiment, a rectified linear unit (ReLU) is provided after each convolution layer except the final output convolution layer.

[0065] Through the compact network layout described above, the morphing image registration model based on the dual-flow cross network according to the present application can capture more long-distance corresponding relationships between images, while calculating bidirectional morphing fields and inferring more reliable and reversible spatial transformation relationships at each voxel.

[0066] ​Thus, each level of the decoder outputs a new feature map, in which case, the feature maps need to be fused, however, the common attention mechanism only allows one input feature map, therefore, the present application provides an attention feature fusion (AFF) module which can fuse two input features. In an embodiment of the present application, the AFF module can fuse any two feature map information output by each level of the decoder.

[0067] Figure 4 is the principle diagram of the AFF module according to an embodiment of the present application for fusing two feature maps. As shown in Figure 4 , the input into the attention mechanism, the output of the attention mechanism Thus, for a given two features Figure X , , which is the abbreviation of the image array, the feature fusion of the AFF module can be expressed by the following equation:

[0068]

[0069] wherein X, Y represent two feature maps, A represents the attention mechanism, represents point-by-point addition, represents point-by-point multiplication. The features Figure X , Y are added pixel by pixel as the input of the attention mechanism, and finally a weight matrix with the same size as the features Figure X , Y is output, and different weight values are given to the features Figure X , Y after addition, so as to achieve the purpose of feature fusion.

[0070] In an embodiment of the present application, Shuffle Attention is used as the attention mechanism of the feature fusion module, which is a lightweight and efficient comprehensive module that simultaneously fuses spatial and channel attention mechanisms. In other embodiments of the present application, according to the different networks in which the AFF module is inserted, the attention mechanism can be flexibly replaced and adjusted, such as CBAM, SEA, etc.

[0071] It is worth noting that the two input feature maps of the AFF module provided by the present application need to have the same size, as long as the given input has the same size, the feature fusion can be performed. Since Figure 3 the feature map sizes output by each level of the decoder shown in the figure are different, it is necessary to process them by upsampling so that the two feature maps output by each level of the decoder have the same size.

[0072] Therefore, in order to fuse the feature information of various scales, according to the AFF module and the up-sampling module, the application sets a Multilevel Attention Feature Fusion (MAFF) module in the deformation image registration model based on the dual-flow cross network, which is respectively applied to two deformation image registration networks, and the final deformation field is calculated by fusing the semantic information of high and low levels.

[0073] The MAFF module includes the AFF module and the up-sampling module. The AFF module is arranged between each level of the decoder of the deformation image registration network, and is configured to perform step-by-step or cross-level fusion processing on two feature maps output by each level of the decoder of the deformation image registration network. The up-sampling module is configured to make the two feature maps output by each level of the decoder have the same size through up-sampling processing.

[0074] Figures 5 to 7 is a working principle diagram of the MAFF module according to the embodiment of the application for fusing feature maps. In Figures 5 to 7 , “up” represents up-sampling, and “A” represents the attention feature fusion module. In the following, the working principle of the MAFF module for fusing feature maps by using different fusion modes is described. Figures 5 to 7

[0075] Figure 5 is a working principle diagram of the MAFF module according to the embodiment of the application for fusing feature maps by using step-by-step fusion. Figure 5 In the embodiment of the application, four feature maps D1, D2, D3 and D4 of different scales (corresponding to the feature maps output by each level of the decoder shown in Figure 3 ) are given, and the corresponding resolution sizes are multiplied in turn. First, in order to fuse the feature maps D1 and D2, D1 is up-sampled to obtain D1*, so that D1* and D2 have the same size, and then D1* and D2 are input into the AFF module to obtain a new fused feature map. The output is up-sampled to obtain D2*, at which time D2* and D3 have the same size, and then input into the AFF module. Through this iterative fusion mode, feature information of different scales can be combined to enhance the learning ability of feature response and specific information of interest.

[0076] It is worth noting that the above-mentioned Multilevel Attention Feature Fusion adopts a step-by-step fusion mode, and the application can also adopt different Multilevel Attention Feature Fusion modes, such as cross-level feature fusion.

[0077] Figure 6 is a working principle diagram of the MAFF module according to the embodiment of the application for fusing feature maps by using cross-level fusion. As Figure 6 ​As shown, D1 is first upsampled to obtain D1* with the same size as D3. Then, D1* and D3 are input into the AFF, and the output is upsampled to obtain D3* with the same size as D4. Similarly, D2 is upsampled to obtain D2* with the same size as D4. Then, D2* and D4 are input into the AFF and fused to obtain D4*. Finally, D3* and D4* are input into the AFF and fused to obtain the final feature map.

[0078] Similarly, Figure 7 This is a schematic diagram illustrating the working principle of the MAFF module according to an embodiment of the present invention, which uses a different cross-level fusion method to fuse feature maps. Figure 7 As shown, D1 and D4 can be fused to obtain D4*, D2 and D3 can be fused to obtain D3*, and then D3* and D4* can be fused to obtain the final feature map.

[0079] Therefore, the embodiments of the present invention provide an MRI brain neural image registration network based on cross-dual-stream multi-scale attention feature fusion (hereinafter referred to as "MAFF-Net").

[0080] by Figure 3 For example, after passing through the MAFF module, both deformed image registration networks output the final feature maps. These two final feature maps are then input into the scaling and squaring layer, as shown in the diagram. Figure 3 As indicated by “ss”, preferably, the number of iterations T = 7 for the shrinking exponential layer is used to obtain the deformation field of the input data sample and its corresponding output final feature map using the differential homeomorphic registration method.

[0081] Differential homeomorphism refers to the property that for two given smooth manifolds M and N, if f: M→N is a bijection (i.e., a one-to-one mapping between them), and f and f -1 If all deformations are smooth mappings, then f is called a differential homeomorphism. Only deformations possessing differential homeomorphism can guarantee the preservation of topological structure, preventing spatial folding and anomalous distortion. The deformation field of a differential homeomorphism can be defined as:

[0082]

[0083] in, It is an identity transformation, where t corresponds to time. By integrating the velocity field v over time t = [0, 1], the final deformation field can be obtained.

[0084] Therefore, as Figure 3As shown, the MRI brain nerve image data set is input into the deformation image registration model based on the double-flow cross network to obtain the bidirectional registered deformation field, wherein one deformation image registration network obtains the forward registered deformation field, and the other deformation image registration network obtains the reverse registered deformation field. wherein θ represents a learning parameter in the network, and respectively represent the differential homeomorphic deformation field of deforming X to Y and deforming Y to X.

[0085] It should be noted that the purpose of the present application is to perform bidirectional registration at the same time, and the bidirectional deformation field of forward and reverse registration of two branches needs to be calculated. Therefore, the designed loss function simultaneously estimates the similarity measure L similarity and smoothness L smooth The expression of the loss function of the final network is as follows:

[0086]

[0087] wherein X, Y represent the first MRI brain nerve image data sample and the second MRI brain nerve image data sample, ° represents a deformation sampling operation, represents the deformation field obtained by registering X to Y, represents the deformation field obtained by registering Y to X. L sim represents a similarity measure, L reg represents a regularization smooth term, and the hyperparameter λ is a parameter used to balance the similarity measure and the smooth regularization term, which can be set as λ=2. The final network training is obtained by minimizing the loss function to obtain the optimal model.

[0088] As described above, in Figure 3 the deformation registration system of the MRI brain nerve image based on the double-flow cross network shown, the two branches of the MAFF-Net each generate a velocity field, and the deformation field corresponding to the input velocity field is obtained by using the differential homeomorphic registration method, so as to obtain the final deformation field of each branch, which are the forward registered deformation field and the reverse registered deformation field respectively.

[0089] In addition, according to another embodiment of the present application, the MAFF-Net can be used as the basis, and the half field of the forward and reverse velocity field is taken respectively by using the velocity field generated by the two branches of the MAFF-Net, so as to obtain the final complete deformation field of the forward and the reverse. Figure 8 is a schematic diagram of a deformation image registration model based on a double-flow cross network according to another embodiment of the present application.

[0090] As Figure 8As shown, X and Y are two MRI brain nerve image data samples obtained by the acquisition module, which are input into the MAFF-Net and output forward registration velocity field v xy and reverse registration velocity field v yx . By taking the inverse of v xy and v yx , the inverse field of the forward registration velocity field-v xy and the inverse field of the reverse registration velocity field-v yx can be obtained. The four velocity fields are input into the shrink-squaring layer, and the shrink-squaring layer is as shown in “ss” in Figure 8 . Preferably, the iteration step number T of the shrink-squaring layer is 7, and the differential homeomorphism registration method is used to obtain the input velocity field and the corresponding half-way deformation field.

[0091] Further, the and are respectively input into the spatial transformation network (not shown in the figure) to obtain the half-way deformation image registered from X to Y and the half-way deformation image registered from Y to X. The forward registration half-way deformation field and the reverse registration inverse half-way deformation field are spliced together to obtain the complete deformation image Figure 8 registered from X to Y. The splicing operation is as shown in “c” in . Similarly, the forward registration inverse half-way deformation field and the reverse registration half-way deformation field are spliced together to obtain the complete deformation image registered from Y to X.

[0092] In order to ensure the accuracy of the forward and reverse registration, the loss function is further used to constrain the image similarity and the smoothness of the deformation field. The complete loss function is as follows:

[0093]

[0094] wherein X and Y represent the first MRI brain nerve image data sample and the second MRI brain nerve image data sample, ° represents the deformation sampling operation, represents the half-way deformation field registered from X to Y, represents the half-way deformation field registered from Y to X. represents the complete deformation field registered from X to Y, represents the complete deformation field registered from Y to X. L sim represents the similarity measure, and L regdenotes the regularization smooth term, and the hyper-parameter is a parameter used to balance the similarity measure and the smooth regularization term, and can be set as 2.

[0095] The present application utilizes the principle and method of deformation image registration to register a source image (a moving image) onto a target image (a fixed image or a reference image). First, deformation fields are obtained by maximizing the similarity between the source image and the target image, so that the deformation fields contain the mapping relationship between each voxel between the two images. Then, the generated deformation fields are passed through a spatial transformation network (STN) and finally the registration between the source image and the target image is realized by using the nearest neighbor interpolation method.

[0096] Further, according to the embodiments of the present application, the moving image and the fixed image are not intentionally distinguished, but bidirectional deformation registration is simultaneously learned to obtain a deformation field as a forward registration deformation field and a reverse registration deformation field, and finally the registration is realized as a forward registration image and a reverse registration image.

[0097] In the present application, unsupervised learning of deformation image registration is used, which is a nonlinear registration process that attempts to establish a dense one-to-one mapping between a pair of images. A typical unsupervised deformable image registration can be defined as:

[0098]

[0099] wherein, denotes the optimal deformation field parameter, F denotes the fixed image, and M denotes the moving image, denotes the deformation field parameter, L sim denotes the similarity measure, L reg denotes the regularization smooth term. In the design of the present application, NCC is used as the similarity measure, and L2 regularization is used to ensure the smoothness of the deformation field. The definition of L2 regularization is as follows:

[0100]

[0101] wherein, L reg denotes the regularization smooth term, v is the velocity field, p denotes a voxel at a certain position, and represents the entire real number domain.

[0102] After the deformation image registration model based on the dual-flow cross network and the spatial transformation network are trained by the above method, the output module is configured to output forward registration images and reverse registration images of two MRI brain nerve images to be deformed and registered by using the learned deformation image registration model in the training module. The two MRI brain nerve images to be deformed and registered can be of different sources, for example, can be individual and individual, or can be individual and a brain template. In this paper, the brain template can represent the common characteristics of a group of individual data, for example, a brain template constructed from 100 individuals.

[0103] Therefore, the present application realizes the deformation registration of 3D T1WI brain images of different sources based on the above unsupervised deep learning principle and method. Figure 9A is an example of an individual grayscale image, Figure 9B is an example of an individual label image, Figure 9C is an example of a brain template grayscale image, Figure 9D is an example of a brain template label image.

[0104] When the source image and the target image are respectively any two examples of 3D T1WI brain images in the data set, the brain parenchyma deformation registration between individual ( Figure 9A ) and individual ( Figure 9A ) can be realized. The method is extended to pair-wise matching in the data set, and the automatic construction of the brain template image can be realized by using the group-wise matching method. When the source image is any one example of 3D T1WI brain image in the data set ( Figure 9A ), and the target image is a pre-constructed MR brain template image in the data set ( Figure 9C ), the registration between the individual image and the template image can be realized. In addition, when the source image or the target image contains a label ( Figure 9B and Figure 9D obtained by manual or automatic segmentation), the present application can also realize the brain atlas label transfer, so that the other image is automatically labeled.

[0105] According to the embodiment of the present application, the deformation registration system of MRI brain nerve images based on the dual-flow cross network realizes the differential homeomorphic image registration network for three-dimensional high-field nuclear magnetic resonance T1 brain nerve images, which is used for brain atlas label transfer and brain template construction.

[0106] According to the embodiment of the present application, the deformation registration system of MRI brain nerve images based on the dual-flow cross network can automatically complete the deformation registration of any two examples of 3D MR brain nerve images. The system has strong universality and high intelligence, and does not depend on artificial or expert labels, nor is it limited to single-source or single-scan-protocol MRI brain data.

[0107] To verify the reliability of the present application, the present application selects a plurality of currently recognized advanced unsupervised deformation image registration methods for comparison, including the traditional algorithm SyN and the deep learning-based algorithms Voxelmorph (VM), Voxelmorph-diffeomorphic (VM-Dif) and SYM-Net. In order to compare fairly, NCC is used as the similarity measurement loss function, and the comparison algorithms are trained using the publicly available codes and parameters on the Internet.

[0108] Regarding the evaluation index, the Dice similarity coefficient (DSC) is used to evaluate the method designed by the present application, which can measure the consistency between the brain template anatomical structure and the anatomical structure segmentation based on the template registration. In addition, the percentage of voxels with negative Jacobian determinant is used to measure the differential homeomorphism of the deformation field. In addition, Recall, average surface distance (ASD) are also used for quantitative evaluation. For negative Jacobian determinant, ASD, lower values indicate better performance. For DSC and Recall, higher values indicate better registration accuracy.

[0109] Regarding the quantitative result display, Table 3 below shows the registration results of different comparison algorithms on the internal dataset and the LPBA40 dataset. It can be seen from the table that the MAFF-Net of the present application achieves the best results in the DSC, negative Jacobian determinant and ASD indicators on the two datasets, and is slightly better than the optimal model in the Recall indicator, proving the reliability and robustness of the design of the present application.

[0110] Table 3

[0111]

[0112] Table 4 below shows the registration results of the algorithm using MAFF-Net+bidirectional splicing of the present application as shown in Figure 8

[0113] Table 4

[0114]

[0115] Regarding the qualitative result display, Figure 10 ​​is a box plot showing the DSC index of different comparison algorithms (SyN, VM, VM-Dif, SYM-Net and the morphological registration system of MRI brain nerve image based on double-stream cross network of the application (labeled as MAFF-Net)) on 15 anatomical structures in the internal dataset, and the compared structures include brain stem (BS), thalamus (Th), cerebellar cortex (Cblmc), lateral ventricle (LV), cerebellar white matter (WM), dura (Pu), caudate nucleus (Ca), globus pallidus (Pa), hippocampus (Hi), third ventricle (3V), fourth ventricle (4V), amygdala (Am), cerebrospinal fluid (CSF), cerebral cortex (CeblC) and cerebral white matter (CeblWM). As can be seen from the results, the MAFF-Net of the application has the highest result in 13 of the 15 structures.

[0116] Figure 11 is a diagram showing the segmentation results of different comparison algorithms (SyN, VM, VM-Dif, SYM-Net and the MAFF-Net of the application) on the caudate nucleus (red), dura (green) and hippocampus (blue) respectively. The image shown is a randomly selected individual image as the moving image, and the brain template as the reference image. As can be seen from Figure 10 , the MAFF-Net of the application has good ability to calculate large deformation registration, and has better matching on the structure edge.

[0117] Figure 12 is a diagram showing the morphological image and its corresponding anatomical structure label of the bidirectional registration based on the MAFF-Net of the application. The top from left to right is respectively a randomly selected individual image X, Y, a morphological image of X registered to Y, a difference image of X minus Y, and a difference image of the morphological image minus Y. According to the difference image of X minus Y, the dark blue part between the brain tissues is the difference between the individual image X and the individual image Y. By comparing the difference image of the morphological image minus Y with the difference image of X minus Y, it can be obviously observed that the dark blue color between the brain tissues decreases and the grayish white color increases, indicating that the matched morphological image and the image Y are better.

[0118] The bottom from left to right is respectively the label image of the individual image X, the label image of the individual image Y, the label migration image of the morphological image, the difference image of the X label image minus the Y label image, and the difference image of the label migration image minus the Y label image. Similarly, by comparing the difference image of the X label image minus the Y label image with the difference image of the label migration image minus the Y label image, the subtraction effect before and after the matching of the image X label and the image Y label can be obviously observed.

[0119] Therefore, the MAFF-Net of the application can achieve good registration accuracy between any individuals during registration, and at the same time complete label migration.

[0120] The various embodiments of the application are not an exhaustive list of all possible combinations, but are intended to describe representative aspects of the application, and what is described in the various embodiments can be applied independently or in combination of two or more.

[0121] The above description presented in the exemplary embodiments is only to illustrate the technical solutions of the application, and is not intended to be exhaustive, nor intended to limit the application to the exact form described. Obviously, many changes and variations are possible for those of ordinary skill in the art based on the above teachings. The exemplary embodiments are selected and described in order to explain the specific principles of the application and its practical application, so that other skilled persons in the art can easily understand, implement and utilize various exemplary embodiments of the application and various selected forms and modified forms thereof. The scope of protection of the application is intended to be defined by the appended claims and their equivalents.

Claims

1. A system for morphing registration of MRI brain images based on a dual-flow cross network, characterized by The method comprises the following modules: an acquisition module configured to acquire an MRI brain nerve image dataset; a training module configured to input the MRI brain nerve image dataset into a deformation image registration model based on a dual-flow cross network, obtain a forward registration deformation field and a reverse registration deformation field, transform the obtained forward registration deformation field and reverse registration deformation field through a spatial transformation network, and generate a forward registration image and a reverse registration image by using a nearest neighbor interpolation method; an output module configured to output the forward registration image and the reverse registration image of two MRI brain nerve images to be subjected to deformation registration by using the deformation image registration model learned in the training module; wherein the deformation image registration model based on the dual-flow cross network comprises two deformation image registration networks with the same structure, and the two deformation image registration networks are respectively input with a first MRI brain nerve image data sample and a second MRI brain nerve image data sample in the MRI brain nerve image dataset; each of the two deformation image registration networks comprises an encoder and a decoder; the decoders of the two deformation image registration networks are cross-connected; each of the deformation image registration networks takes a 3D U-Net framework with encoding-decoding as a backbone network, and the number of levels of the encoder of each of the deformation image registration networks is equal to the number of levels of the decoder; the encoders and the decoders of each of the deformation image registration networks are connected by jumping, and each level of the decoder of one of the deformation image registration networks is cross-connected with a corresponding level of the decoder of the other of the deformation image registration networks; each level of the decoder of each of the deformation image registration networks is provided with an attention mechanism module; the encoders of the two deformation image registration networks share weights.

2. The system for morphing registration of MRI brain nerve images based on dual-flow cross network according to claim 1, wherein, The training of the deformation image registration model based on the dual-flow cross network obtains an optimal model based on minimizing the following loss function: wherein X, Y represent the first and second MRI brain nerve image data samples, denotes a deformation sampling operation, denotes a deformation field obtained by registering X to Y, denotes a deformation field obtained by registering Y to X, and the hyper-parameter λ represents a parameter used to balance the similarity measure and the smoothing regularizer.

3. The deformation registration system for MRI brain nerve images based on a dual-flow cross network according to claim 1, wherein the encoder of each of the deformation image registration networks comprises a first-level encoder to an Nth-level encoder, and each of the encoders outputs a feature map with a size gradually decreasing; the decoder of each of the deformation image registration networks comprises a first-level decoder to an Nth-level decoder, and the Nth-level decoder of each of the decoders is connected with the (N-n+1)th-level encoder by jumping, the Nth-level decoder is configured to receive a feature map from the (N-n+1)th-level encoder and perform feature extraction, generate a feature map of the Nth-level decoder, and exchange the feature map of the Nth-level decoder with a feature map of the Nth-level decoder generated by the other of the deformation image registration networks, and perform feature fusion on the received feature map of the (N-n+1)th-level encoder and the feature map of the Nth-level decoder of the other of the deformation image registration networks after the exchange, and output a new feature map; wherein n≤N, and N is an integer.

4. The deformation registration system for MRI brain nerve images based on a dual-flow cross network according to claim 3, wherein the first-level encoder to the Nth-level encoder each comprise a convolution structure of a 3×3×3 convolution layer with a step of 1 and a 3×3×3 convolution layer with a step of 2. The first-stage decoder to the Nth-stage decoder respectively include a convolution structure of a 3*3*3 convolution layer with a step of 1 and a 2*2*2 transposed convolution layer with a step of 1.

5. The dual-flow cross-network-based MRI brain nerve image morphing registration system according to claim 1, wherein, The dual-flow cross-network-based morphing image registration model further comprises a multi-scale attention feature fusion module, and the multi-scale attention feature fusion module comprises an attention feature fusion module; The attention feature fusion module is arranged between each level of the decoder of each morphing image registration network, and is configured to perform step-by-step or cross-level fusion processing on two feature maps output by each level of the decoder of the morphing image registration network.

6. The system for morphing registration of MRI brain nerve images based on dual-flow cross-network of claim 5, wherein, The step-by-step or cross-level fusion processing on the two feature map information output by each level of the decoder is performed by using a feature fusion equation as follows: Wherein, X, Y represent any two feature maps of the output of each stage of the decoder, A represents the attention mechanism, and represents point-by-point addition, represents point-by-point multiplication.

7. The dual-flow cross-network-based MRI brain nerve image morphing registration system according to claim 5, wherein, The multi-scale attention feature fusion module further comprises an up-sampling module configured to make the two feature maps output by each level of the decoder have the same size through up-sampling processing.

8. The dual-flow crossbar network based MRI brain nerve image morphing registration system of claim 1, wherein, The acquisition module is configured to pre-process the collected MRI brain nerve images to obtain an MRI brain nerve image dataset, and the pre-processing includes at least one, part or all of format conversion, steering, ACPC alignment, bias field correction, skull removal, resampling, size adjustment and label generation.

9. A computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions, when executed by the processor, implement the following steps: Obtain an MRI brain nerve image dataset; Input the MRI brain nerve image dataset into the dual-flow cross-network-based morphing image registration model to obtain a forward registration morphing field and a reverse registration morphing field, and transform the obtained forward registration morphing field and reverse registration morphing field through a spatial transformation network to generate a forward registration image and a reverse registration image using a nearest neighbor interpolation method; Output the forward registration image and the reverse registration image of the two MRI brain nerve images to be morphed and registered using the learned morphing image registration model in the training module; The dual-flow cross-network-based morphing image registration model comprises two morphing image registration networks with the same structure, and the two morphing image registration networks respectively input a first MRI brain nerve image data sample and a second MRI brain nerve image data sample of the MRI brain nerve image dataset; The two morphing image registration networks respectively comprise an encoder and a decoder; The decoders of the two morphing image registration networks are cross-connected; Each morphing image registration network takes a 3D U-Net framework of encoding-decoding as a backbone network, and the number of levels of the encoder of each morphing image registration network is equal to the number of levels of the decoder; The encoders and the decoders of each morphing image registration network are connected by jumping, and the levels of the decoder of one morphing image registration network are cross-connected with the corresponding levels of the decoder of the other morphing image registration network; An attention mechanism module is arranged between each level of the decoder of each morphing image registration network; The weights between the encoders of the two deformation image registration networks are shared.

Images