A multi-modal medical image elastic registration method and system, device, medium

Abstract

This invention provides a method, system, device, and medium for elastic registration of multimodal medical images. A dual-stream parameter-shared encoder extracts modality-independent representations, and a decoder-free network structure generates a sparse registration deformation field to accommodate large deformations between the modal images to be registered. A parameter-shared segmentation-guided contrastive learning network is used to segment the anatomical structures of the modal images to be registered. Comparative learning among the multiple hierarchical feature maps output by the encoder makes the encoded features between the modalities to be registered more consistent, improving the ability of the registration encoder to extract modality-independent representations and thus improving registration accuracy. The more aligned paired images after registration help the segmentation-guided contrastive learning network extract more consistent modality-independent representations, and the registration encoder obtained by the segmentation-guided contrastive learning network further improves registration accuracy.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a method, system, device, and medium for flexible registration of multimodal medical images. Background Technology

[0002] Multimodal medical image registration is an important prerequisite for fully integrating complementary information from multimodal medical images. However, due to limitations in imaging principles and patient movement, different modal medical images exhibit significant differences in contrast, signal-to-noise ratio, and deformation shift. Existing multimodal medical image registration methods struggle to establish accurate spatial correspondences of anatomical structures under these constraints, resulting in low registration accuracy.

[0003] The key to multimodal medical image registration lies in learning feature representations of corresponding structures in images of different modalities. Traditional registration methods based on transformations and similarity measures are computationally time-consuming and produce unstable results. In contrast, deep learning-based registration methods can automatically extract features and perform rapid inference, and have been widely used in recent years. Encoder-decoder convolutional neural networks, represented by VoxelMorph, have become the mainstream solution for image registration. However, these methods typically concatenate stationary and moving images, then feed them into the neural network to extract features and map these features to spatial transformations for matching. In this process, we cannot see explicit feature matching as in traditional image registration methods, resulting in poor performance in multimodal image registration with large deformations. In recent years, learning feature representations in a self-supervised manner through contrastive learning has received widespread attention, and has also been applied in multimodal image registration, such as using contrastive learning for joint learning of registration and image fusion.

[0004] On the other hand, based on the data type of the labels, deep learning-based registration methods can be divided into fully supervised registration, unsupervised registration, and segmentation-based weakly supervised registration. Since fully supervised registration requires enormous costs for annotating the anatomical correspondences of multimodal medical images, while unsupervised registration suffers from poor universality of similarity measures, difficulty in representing multimodal medical image features, and high network training difficulty, segmentation-based weakly supervised registration is a more feasible technical approach. Weakly supervised registration methods provide deformation field constraints through tissue segmentation labels, which reduces the difficulty of label data annotation to some extent compared to the deformation field labels required for fully supervised registration. However, weakly supervised registration methods still require a large amount of segmentation label data, and segmentation differences between different modalities can cause registration errors.

[0005] Because medical image registration and segmentation are synergistic: segmentation provides additional morphological supervision information for registration, and accurate segmentation labels can be used to improve registration accuracy; the probability maps, statistical shape models, or deformation fields generated by registration provide additional constraint information for segmentation, which can be used to guide image segmentation. Therefore, the combined application of registration and segmentation can simultaneously improve the registration and segmentation results. Summary of the Invention

[0006] To achieve the above-mentioned objectives and other advantages of the present invention, a first objective of the present invention is to provide a method for elastic registration of multimodal medical images, comprising the following steps:

[0007] A deep learning elastic registration network employing a dual-stream parameter-shared encoder and no decoder extracts modality-independent semantic representations of fixed and floating images, respectively.

[0008] The spatial differences between deep semantic representations are learned by the deformation field generation module in the registration network to obtain a sparse deformation field.

[0009] The deformation field is applied to the floating image by the sampler in the registration network to obtain a preliminarily registered floating image;

[0010] The fixed image and the initially registered floating image are input into a segmentation-guided contrastive learning network with shared parameters. The feature maps of multiple levels output by the encoders of the two segmentation-guided contrastive learning networks are compared and learned. The encoder structure of the segmentation-guided contrastive learning network is consistent with the encoder structure in the registration network. The two segmentation-guided contrastive learning networks are supervised only by the segmentation labels of the fixed image.

[0011] The registered floating image is used as the input to the segmentation-guided contrastive learning network, and the encoder network parameters obtained from the segmentation-guided contrastive learning network are used as the initialization of the encoder in the registration network for iterative training.

[0012] Furthermore, the encoder includes multiple residual convolutional modules, each residual convolutional module including multiple convolutional layers, normalization layers and activation layers, and the encoder adopts an anisotropic convolutional strategy.

[0013] Furthermore, the deformation field generation module includes multiple convolutional layers, normalization layers, and activation layers. It concatenates the deep semantic representations extracted by the encoder from the fixed image and the floating image, and inputs them into the deformation field generation module to obtain the deformation field.

[0014] Furthermore, comparative learning is performed between the feature maps of multiple layers output by the two segmentation-guided contrastive learning network encoders, specifically by outputting the feature maps of the deepest layers of the two segmentation-guided contrastive learning network encoders for comparative learning.

[0015] In the segmentation-guided contrastive learning network, the encoder is then fed into the decoder after passing through a residual convolution module.

[0016] Furthermore, the decoder includes multiple upsampling modules, each upsampling module including a transposed convolutional layer and a regular convolutional module; wherein, the regular convolutional module includes a convolutional layer, an activation layer and a normalization layer;

[0017] The transposed convolutional layer upsamples the current encoder features, concatenates them with the features of the previous encoder layer, and then inputs them into the regular convolutional module.

[0018] The number of output channels of the multiple transposed convolutional layers is the opposite of the number of output channels of the decoder.

[0019] Furthermore, the comparative learning step between the feature maps of multiple levels output by the two segmentation-guided contrastive learning network encoders includes:

[0020] Learning network parameters using the InfoNCE contrastive loss function:

[0021]

[0022] in, For the size of the batch data pairs, For similarity measurement, The temperature coefficient is adjustable for dynamic range. Greater than 0, The dataset represents paired fixed and floating image pairs. The Middle Regarding the data, The total number of image pairs. and This indicates that the images will be fixed respectively. and floating images Mapped to a modality-independent feature space at multiple scales, To determine the number of output feature layers;

[0023] Construct a loss function for floating images Thus, the total contrastive loss function is obtained as follows: The mean squared error is used as a similarity measure, i.e.:

[0024]

[0025] in, and These are modality-independent feature maps of any layer, whether paired or unpaired.

[0026] Similarity measure Convert to the following form:

[0027]

[0028] in, and For any invertible transformation, and For paired or unpaired fixed and floating images, By encoder Obtained through conversion;

[0029] Convert the similarity measure transformation results into a form that does not require inverse transformation:

[0030]

[0031] in, A random number between 0 and 1. For any spatial transformation that includes elastic deformation;

[0032] A combination of the Dice loss function and the focus loss function is used as the segmentation loss function:

[0033]

[0034] in, This refers to the number of pixels in the intersection of the segmentation label gold standard and the segmentation result. and These represent the gold standard for segmentation and the number of pixels in the segmentation result, respectively. For smoothing coefficients, The segmentation output probability for each pixel. The modulation coefficients for focus loss, This represents the number of pixels in the entire image.

[0035] A second objective of this invention is to provide a multimodal medical image elastic registration system, employing the aforementioned method, including a modality-independent semantic representation extraction module, a deep semantic representation spatial difference learning module, an image registration module, a segmentation-guided contrastive learning module, and an iterative training module; wherein,

[0036] The modality-independent semantic representation extraction module is used to extract modality-independent semantic representations of fixed and floating images respectively from two encoders that share parameters in a deep learning elastic registration network with a dual-stream parameter-shared encoder and no decoder.

[0037] The deep semantic representation spatial difference learning module is used to learn the spatial differences between deep semantic representations through the deformation field generation module in the registration network to obtain a sparse deformation field.

[0038] The image registration module is used to apply the deformation field to the floating image through the sampler in the registration network to obtain a preliminarily registered floating image;

[0039] The segmentation-guided contrastive learning module is used to input the fixed image and the initially registered floating image into a segmentation-guided contrastive learning network with shared parameters. The feature maps of multiple levels output by the encoders of the two segmentation-guided contrastive learning networks are compared and learned. The encoder structure of the segmentation-guided contrastive learning network is consistent with the encoder structure in the registration network. The two segmentation-guided contrastive learning networks are supervised only by the segmentation labels of the fixed image.

[0040] The iterative training module is used to take the registered floating image as the input of the segmentation-guided contrastive learning network, and use the encoder network parameters obtained from the segmentation-guided contrastive learning network as the initialization of the encoder in the registration network for iterative training.

[0041] A third objective of the present invention is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0042] A fourth objective of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] This invention provides a deep learning elastic registration network with a dual-stream parameter-shared encoder and no decoder. The dual-stream parameter-shared encoder is used to extract modality-independent representations, and the decoder-free network structure is used to generate a sparse registration deformation field to accommodate large deformations between modal images to be registered.

[0045] This invention provides a modality-independent representation learning method based on segmentation-guided contrastive learning. It uses a parameter-shared segmentation-guided contrastive learning network to segment the anatomical structure of the image to be registered. Only one anatomical structure label for each modality is needed. The encoder structure of the segmentation-guided contrastive learning network is the same as that of the registration network encoder. The feature maps of multiple levels output by the encoder are compared and learned, so that the encoded features of different modalities tend to be consistent and the encoded features focus on the anatomical structure. This improves the ability of the registration encoder to extract modality-independent representations, thereby improving the registration accuracy.

[0046] This invention provides an iterative training method for registration and segmentation-guided contrastive learning. The more aligned paired images obtained by registration can help segmentation-guided contrastive learning to better extract modality-independent representations. The better registration encoder obtained by segmentation-guided contrastive learning further improves the registration accuracy. Through iterative training, the adverse effects of misaligned images on segmentation-guided contrastive learning can be reduced, further improving the registration accuracy.

[0047] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it according to the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Specific embodiments of the present invention are given in detail below with reference to the accompanying drawings. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0049] Figure 1 A diagram illustrating the intermodal flexible registration framework based on a two-stream shared coding network and segmentation-guided contrastive learning;

[0050] Figure 2 A flowchart of a multimodal medical image elastic registration method;

[0051] Figure 3 To register the network structure diagram;

[0052] Figure 4 A diagram of the segmentation-guided contrastive learning network structure;

[0053] Figure 5 This is a schematic diagram of a multimodal medical image elastic registration system.

[0054] Figure 6 This is a schematic diagram of a computer device.

[0055] Figure 7 This is a schematic diagram of a computer-readable storage medium. Detailed Implementation

[0056] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0057] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention.

[0058] The drawing numbers in this application are only used to distinguish the steps in the scheme and are not used to limit the execution order of the steps. The specific execution order is as described in the specification.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0060] The method provided by this invention is as follows: Figure 1 As shown, the system mainly consists of two parts: a registration network based on two-stream parameter shared coding without a decoder and a modality-independent representation learning based on segmentation-guided contrastive learning, as well as iterative training between the two parts. In the registration network based on two-stream parameter shared coding without a decoder, let the fixed image of the i-th patient and the floating image to be registered be... and After two images are input into an encoder network with shared parameters, the deformation field generator learns the offset between encoded features to construct a registration deformation field. Then, the sampler resamples based on the deformation field to generate the registered image. In modality-independent representation learning based on segmentation-guided contrastive learning, and initial registration In the segmentation-guided contrastive learning network with shared input parameters, the encoder part of the segmentation-guided contrastive learning network maintains the same structure as the encoder in the registration network. Both networks are supervised using only the segmentation labels of fixed images, and the output encoded representations of both networks are used for contrastive learning training, promoting the encoder to learn modality-independent representations. Finally, iterative training of registration and segmentation-guided contrastive learning further enhances the ability of the registration encoder to extract modality-independent representations, thereby improving the registration accuracy between modalities. The specific scheme is as follows:

[0061] Example 1

[0062] A multimodal medical image elastic registration method, such as Figures 1-4 As shown, it includes the following steps:

[0063] Intermodal large deformation elastic registration based on dual-stream parameter-shared coding: S100, a deep learning elastic registration network with two parameter-shared encoders and no decoder, extracts modality-independent semantic representations of fixed and floating images respectively.

[0064] S200. The spatial differences between deep semantic representations are learned through the deformation field generation module in the registration network to obtain a sparse deformation field.

[0065] S300, A deformation field with the same size as the input image is obtained by upsampling through the sampler in the registration network;

[0066] In some embodiments, the encoder includes multiple residual convolutional modules, each residual convolutional module including multiple convolutional layers, normalization layers, and activation layers, and the encoder employs an anisotropic convolutional strategy. For example, as... Figure 3 As shown, the encoder in the deep learning elastic registration network with dual-stream parameter-shared encoder and no decoder includes 5 residual convolutional modules, each of which includes 3 convolutional layers, 3 normalization layers and 2 activation layers.

[0067] It should be noted that the encoder is not limited to the convolutional module described above, and other network structures can also be used. Suitable combinations, substitutions, and modifications can be made within the scope of the purpose and spirit of this invention.

[0068] Due to the high intra-layer resolution and large inter-layer spacing of MRI images, this embodiment employs an anisotropic convolution strategy: in the first two residual convolutional modules, the kernel size and stride are set to 3×3×1 and 2×2×1, respectively, making the feature map isotropic during convolution; in the latter three deeper residual convolutional modules, the kernel size and stride are set to 3×3×3 and 2×2×2, respectively. The number of output channels for each residual convolutional module is set to 32, 64, 128, 256, and 320, respectively.

[0069] The deformation field generation module includes multiple convolutional layers, normalization layers, and activation layers. It concatenates the deep semantic representations extracted from the fixed and floating images by the encoder and inputs them into the deformation field generation module to obtain the deformation field. For example, the deformation field generation module includes 4 convolutional layers, 3 normalization layers, and 3 activation layers. It concatenates the deep semantic representations extracted from the fixed and floating images by the encoder and inputs them into the deformation field generation module to obtain the deformation field. All convolutional kernels in this module are set to a size of 3×3×3 and a stride of 1×1×1. Except for the last convolutional layer, which has 3 output channels, all other convolutional layers have 512 output channels.

[0070] It should be noted that the deformation field generation module is not limited to the above structure, and can be appropriately combined, substituted, or modified within the scope of the purpose and spirit of this invention.

[0071] The registration loss function includes image similarity measure loss and deformation field smoothing constraint loss, defined as follows:

[0072]

[0073] in, For fixed image, For floating images, For deformation field, For image similarity measurement loss, The loss is a smoothing constraint loss for the deformation field.

[0074] This embodiment uses global mutual information loss as the image similarity measure. For the deformation field smoothing constraint loss, this embodiment uses the commonly used bending energy loss. This embodiment sets... and Both are set to 1 to balance the impact of image similarity metric loss and deformation field smoothing constraint loss on the training of the registration network.

[0075] Modality-independent representation learning based on segmentation-guided contrastive learning: S400, the fixed image and the initially registered floating image are input into two parameter-shared segmentation-guided contrastive learning networks, and the feature maps of multiple levels output by the encoders of the two segmentation-guided contrastive learning networks are compared and learned; wherein, the encoder structure of the segmentation-guided contrastive learning network is consistent with the encoder structure in the registration network, and the two segmentation-guided contrastive learning networks are supervised only by the segmentation labels of the fixed image.

[0076] like Figure 4 As shown, this embodiment employs two parameter-sharing segmentation-guided contrastive learning networks. The encoder structures of both networks are consistent with those of the registration network. The comparative learning between the feature maps output from the encoders of the two networks specifically involves outputting the feature maps from the deepest layers of both encoders for comparative learning; for example, the deepest three layers of feature maps from both encoders are used for comparison. The encoder in each segmentation-guided contrastive learning network is then fed into the decoder after passing through a residual convolutional module.

[0077] The decoder comprises multiple upsampling modules, each containing a transposed convolutional layer and a regular convolutional module. The regular convolutional module includes a convolutional layer, an activation layer, and a normalization layer. For example, corresponding to the encoder containing five residual convolutional modules, the decoder contains five upsampling modules. Each upsampling module contains one transposed convolutional layer and one regular convolutional module, where the regular convolutional module includes two convolutional layers, one activation layer, and two normalization layers. The transposed convolutional layer upsamples the current encoder features, concatenates them with the features from the previous encoder layer, and then inputs the result into the regular convolutional module. The number of output channels in multiple transposed convolutional modules is the opposite of the number of output channels in the decoder. Taking the five transposed convolutional layers mentioned above as examples, their output channel numbers are 320, 256, 128, 64, and 32. The kernel size and stride of the first three modules are 2×2×2, and the kernel size and stride of the last two modules are 2×2×1. Among the five regular convolutional modules, except for the last module which has a kernel size of 3×3×1, the kernel size of the other modules is 3×3×3, and the stride is 1×1×1. Finally, the number of channels is reduced to 2 by an output convolutional layer with a kernel size of 1×1×1, resulting in the segmentation result.

[0078] To fully utilize the segmentation labels, a deep supervision strategy is adopted for the segmentation output. This involves simultaneously supervising the segmentation labels using the outputs of the penultimate and third-to-last decoder layers after 1×1×1 convolution. It's important to note that the dataset used in this embodiment only contains segmentation labels for fixed images. The segmentation labels for the fixed images can only be used to supervise the learning of the floating image segmentation network after pre-registration of the floating images. Therefore, this embodiment also trains the segmentation-guided contrastive learning network based on the aforementioned preliminary registration results.

[0079] For contrastive learning of encoder features, let the dataset of paired fixed and floating image pairs be... ,in The total number of image pairs. The dataset represents paired fixed and floating image pairs. The Middle Regarding the data. The goal of this embodiment is to learn the encoder. The fixed images are respectively and fixed image Mapped to a modality-independent feature space at multiple scales, i.e. and ,in This determines the number of output feature layers. The key to contrastive learning lies in constructing positive and negative samples. For the... The first case In terms of layer feature maps, For positive samples that need to be narrowed down in the modality-independent representation space, or For negative samples that need to be far from the modality-independent characterization space.

[0080] In order to obtain The network parameters are learned using the InfoNCE contrastive loss function:

[0081]

[0082] in, For the size of the batch data pairs, For similarity measurement, The temperature coefficient is adjustable for dynamic range. Greater than 0;

[0083] Correspondingly, a loss function can also be constructed for floating images. Thus, the total contrastive loss function is obtained as follows: The mean squared error is used as a similarity measure, i.e.:

[0084]

[0085] in, and These are modality-independent feature maps of any layer, whether paired or unpaired.

[0086] To ensure that the modality-independent representations extracted by the encoder can be applied to image registration, these features should possess transformability such as spatial transformation, i.e., for the encoder... Any invertible transformation and any image , must meet encoder Convertable to For similarity measures, It can be converted into the following form:

[0087]

[0088] in, and For any invertible transformation. and For paired or unpaired fixed and floating images, however, the deformation field is not always reversible for elastic registration. To ensure that the spatial transformation isomorphic constraint is still satisfied under elastic deformation, the above equation is transformed into a form that does not require inverse transformation:

[0089]

[0090] in, A random number between 0 and 1. For any spatial transformation that includes elastic deformation;

[0091] In actual training, the random transformation can be set as a combination of a random affine transformation and an elastic deformation transformation.

[0092] It should be noted that the present invention is not limited to the above-mentioned similarity measurement, and appropriate combinations, substitutions, and modifications can be made without departing from the purpose and spirit of the present invention.

[0093] A combination of the Dice loss function and the focus loss function is used as the segmentation loss function:

[0094]

[0095] in, This refers to the number of pixels in the intersection of the segmentation label gold standard and the segmentation result. and These represent the gold standard for segmentation and the number of pixels in the segmentation result, respectively. It is a very small smoothing coefficient (with a value of 1e-5). The segmentation output probability for each pixel. The modulation coefficient for focus loss (with a value of 2). This represents the number of pixels in the entire image.

[0096] Since the segmentation adopts a deep supervision strategy, this embodiment sets the loss weight of the penultimate decoder output to 0.25, the loss weight of the penultimate decoder output to 0.5, and the loss weight of the final output layer to 1.

[0097] After joint learning training of encoder feature contrast learning and segmentation on fixed and floating images, the encoder has a good ability to extract modality-independent structural representations. Therefore, the encoder network parameters are saved and used as the initialization parameters of the encoder in the registration network to help the registration network better extract modality-independent structural representations, thereby improving the registration accuracy.

[0098] In the above steps, whether it's the contrastive learning task between features of two modality encoders or the segmentation task (especially the segmentation task of floating images, since floating images have no segmentation labels, the segmentation of floating images can only use the segmentation labels of fixed images), the alignment between input image pairs is crucial. Therefore, in the above steps, the initially registered floating images are input into the network. Ideally, if the input image pairs can be perfectly aligned, it will be most beneficial for the encoder to extract accurate modality-independent representations. However, no registration method can guarantee that the floating image and the corresponding fixed image are perfectly aligned, and this ideal situation is the target of the registration between fixed and floating images.

[0099] Therefore, this embodiment designs an iterative training framework for registration and segmentation-guided contrastive learning: S500, the registered floating image is used as the input of the segmentation-guided contrastive learning network, and the encoder network parameters obtained by the segmentation-guided contrastive learning network are used as the initialization of the encoder in the registration network for iterative training.

[0100] Through the above iterative training, the image pairs input to the segmentation-guided contrastive learning network are more aligned, minimizing the impact of unregistered floating images on modality-independent representation learning. At the same time, it provides better initialization parameters for the registration network encoder, which can further improve registration accuracy.

[0101] The evaluation metrics for this method include the Dice and Hausdorff distance (HD95) at the 95th percentile of the segmentation labels of the fixed image and the registered floating image, as well as the target registration error (TRE) calculated based on paired anatomical markers. The results in Table 1 show that when this invention is used for T2W and ADC image registration in prostate MRI, the registration results are superior to traditional affine transformation or elastic registration methods, and also superior to deep learning registration methods such as Voxelmorph, Transmorph, Vit-V-net, and TransMatch.

[0102] Table 1 Registration Results

[0103]

[0104] Example 2

[0105] A multimodal medical image elastic registration system is provided, applying the method described above. For a detailed description of the method, please refer to the corresponding descriptions in the above method embodiments; they will not be repeated here. Figure 5 As shown, the system 600 includes a modality-independent semantic representation extraction module 610, a deep semantic representation spatial difference learning module 620, an image registration module 630, a segmentation-guided contrastive learning module 640, and an iterative training module 650; wherein,

[0106] The modality-independent semantic representation extraction module is used to extract modality-independent semantic representations of fixed and floating images respectively from two encoders that share parameters in a deep learning elastic registration network with a dual-stream parameter-shared encoder and no decoder.

[0107] The deep semantic representation spatial difference learning module is used to learn the spatial differences between deep semantic representations through the deformation field generation module in the registration network to obtain a sparse deformation field.

[0108] The image registration module is used to apply the deformation field to the floating image through the sampler in the registration network to obtain a preliminarily registered floating image;

[0109] The segmentation-guided contrastive learning module is used to input the fixed image and the initially registered floating image into a segmentation-guided contrastive learning network with shared parameters. The feature maps of multiple levels output by the encoders of the two segmentation-guided contrastive learning networks are compared and learned. The encoder structure of the segmentation-guided contrastive learning network is consistent with the encoder structure in the registration network. The two segmentation-guided contrastive learning networks are supervised only by the segmentation labels of the fixed image.

[0110] The iterative training module is used to take the registered floating image as the input of the segmentation-guided contrastive learning network, and use the encoder network parameters obtained from the segmentation-guided contrastive learning network as the initialization of the encoder in the registration network for iterative training.

[0111] Based on the technical solution of the above embodiments, optionally, the encoder includes multiple residual convolution modules, each residual convolution module includes multiple convolutional layers, normalization layers and activation layers, and the encoder adopts an anisotropic convolution strategy.

[0112] Based on the technical solution of the above embodiments, optionally, the deformation field generation module includes multiple convolutional layers, normalization layers and activation layers, which concatenate the deep semantic representations extracted by the encoder from the fixed image and the floating image, and input them into the deformation field generation module to obtain the deformation field.

[0113] Based on the technical solution of the above embodiments, optionally, comparative learning is performed between the feature maps of multiple levels output by the two segmentation guided contrastive learning network encoders, specifically, the feature maps of the deepest several layers of the two segmentation guided contrastive learning network encoders are output for comparative learning.

[0114] In the segmentation-guided contrastive learning network, the encoder is then fed into the decoder after passing through a residual convolution module.

[0115] Based on the technical solutions of the above embodiments, optionally, the decoder includes multiple upsampling modules, each upsampling module including a transposed convolutional layer and a regular convolutional module; wherein, the regular convolutional module includes a convolutional layer, an activation layer and a normalization layer;

[0116] The transposed convolutional layer upsamples the current encoder features, concatenates them with the features of the previous encoder layer, and then inputs them into the regular convolutional module.

[0117] The number of output channels of the multiple transposed convolutional layers is the opposite of the number of output channels of the decoder.

[0118] Based on the technical solution of the above embodiments, optionally, the step of performing comparative learning between the feature maps of multiple levels output by the two segmentation-guided contrastive learning network encoders includes:

[0119] Learning network parameters using the InfoNCE contrastive loss function:

[0120]

[0121] in, For the size of the batch data pairs, For similarity measurement, The temperature coefficient is adjustable for dynamic range. Greater than 0, The dataset represents paired fixed and floating image pairs. The Middle Regarding the data, The total number of image pairs. and This indicates that the images will be fixed respectively. and floating images Mapped to a modality-independent feature space at multiple scales, To determine the number of output feature layers;

[0122] Construct a loss function for floating images Thus, the total contrastive loss function is obtained as follows: The mean squared error is used as a similarity measure, i.e.:

[0123]

[0124] in, and These are modality-independent feature maps of any layer, whether paired or unpaired.

[0125] Similarity measure Convert to the following form:

[0126]

[0127] in, and For any invertible transformation, and For paired or unpaired fixed and floating images, By encoder Obtained through conversion;

[0128] Convert the similarity measure transformation results into a form that does not require inverse transformation:

[0129]

[0130] in, A random number between 0 and 1. For any spatial transformation that includes elastic deformation;

[0131] A combination of the Dice loss function and the focus loss function is used as the segmentation loss function:

[0132]

[0133] in, This refers to the number of pixels in the intersection of the segmentation label gold standard and the segmentation result. and These represent the gold standard for segmentation and the number of pixels in the segmentation result, respectively. For smoothing coefficients, The segmentation output probability for each pixel. The modulation coefficients for focus loss, This represents the number of pixels in the entire image.

[0134] Example 3

[0135] A computer device 700, such as Figure 6 As shown, the system includes a memory 710, a processor 720, and a computer program 730 stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a multimodal medical image elastic registration method. For a detailed description of the method, please refer to the corresponding description in the above method embodiments; it will not be repeated here.

[0136] Example 4

[0137] A computer-readable storage medium, such as Figure 7 As shown, a computer program is stored thereon, which, when executed by a processor, implements the steps of a multimodal medical image elastic registration method. For a detailed description of the method, please refer to the corresponding description in the above method embodiments, and will not be repeated here.

[0138] The number of devices and processing scale described herein are for the purpose of simplifying the description of the invention. Applications, modifications, and variations of the invention will be readily apparent to those skilled in the art.

[0139] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

[0140] The apparatus, computer device, and non-volatile computer storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, computer device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, computer device, and non-volatile computer storage medium will not be repeated here.

[0141] Those skilled in the art will also know that, besides implementing the controller in the form of purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller take the form of logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices included within it for implementing various functions can also be considered structures within that hardware component. Alternatively, the devices for implementing various functions can be considered as both software units implementing the method and structures within a hardware component.

[0142] The systems, apparatuses, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above apparatuses are described separately by function as various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0143] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0144] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0145] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0146] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0147] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0148] This specification may be described in the general context of computer-executable instructions, such as program units, that are executed by a computer. Generally, program units include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification may also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program units may reside in local and remote computer storage media, including storage devices.

[0149] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0150] The above description is merely an embodiment of this specification and is not intended to limit the scope of one or more embodiments of this specification. Various modifications and variations can be made to one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of one or more embodiments of this specification.

Claims

1. A method for elastic registration of multimodal medical images, characterized in that, Includes the following steps: A deep learning elastic registration network employing a dual-stream parameter-shared encoder and no decoder extracts modality-independent semantic representations of fixed and floating images, respectively. The spatial differences between deep semantic representations are learned by the deformation field generation module in the registration network to obtain a sparse deformation field. The deformation field is applied to the floating image by the sampler in the registration network to obtain a preliminarily registered floating image; The fixed image and the initially registered floating image are input into a segmentation-guided contrastive learning network with shared parameters. The feature maps of multiple levels output by the encoders of the two segmentation-guided contrastive learning networks are compared and learned. The encoder structure of the segmentation-guided contrastive learning network is consistent with the encoder structure in the registration network. The two segmentation-guided contrastive learning networks are supervised only by the segmentation labels of the fixed image. The registered floating image is used as the input to the segmentation-guided contrastive learning network, and the encoder network parameters obtained from the segmentation-guided contrastive learning network are used as the initialization of the encoder in the registration network for iterative training.

2. The method for elastic registration of multimodal medical images as described in claim 1, characterized in that: The encoder includes multiple residual convolutional modules, each of which includes multiple convolutional layers, normalization layers, and activation layers. The encoder employs an anisotropic convolutional strategy.

3. The method for elastic registration of multimodal medical images as described in claim 1, characterized in that: The deformation field generation module includes multiple convolutional layers, normalization layers, and activation layers. It concatenates the deep semantic representations extracted by the encoder from the fixed image and the floating image, and inputs them into the deformation field generation module to obtain the deformation field.

4. The method for elastic registration of multimodal medical images as described in claim 1, characterized in that: The feature maps of multiple layers output by the two segmentation-guided contrastive learning network encoders are compared and learned. Specifically, the feature maps of the deepest layers of the two segmentation-guided contrastive learning network encoders are output for comparison and learning. In the segmentation-guided contrastive learning network, the encoder is then fed into the decoder after passing through a residual convolution module.

5. The method for elastic registration of multimodal medical images as described in claim 4, characterized in that: The decoder includes multiple upsampling modules, each of which includes a transposed convolutional layer and a regular convolutional module; wherein the regular convolutional module includes a convolutional layer, an activation layer, and a normalization layer; The transposed convolutional layer upsamples the current encoder features, concatenates them with the features of the previous encoder layer, and then inputs them into the regular convolutional module. The number of output channels of the multiple transposed convolutional layers is the opposite of the number of output channels of the decoder.

6. The method for elastic registration of multimodal medical images as described in claim 1, characterized in that, The comparative learning steps between the feature maps of multiple levels output by the two segmentation-guided contrastive learning network encoders include: Learning network parameters using the InfoNCE contrastive loss function: in, For the size of the batch data pairs, For similarity measurement, The temperature coefficient is adjustable for dynamic range. Greater than 0, The dataset represents paired fixed and floating image pairs. The Middle Regarding the data, The total number of image pairs. and This indicates that the images will be fixed respectively. and floating images Mapped to a modality-independent feature space at multiple scales, To determine the number of output feature layers; Construct a loss function for floating images Thus, the total contrastive loss function is obtained as follows: The mean squared error is used as a similarity measure, i.e.: in, and These are modality-independent feature maps of any layer, whether paired or unpaired. Similarity measure Convert to the following form: in, and For any invertible transformation, and For paired or unpaired fixed and floating images, By encoder Obtained through conversion; Convert the similarity measure transformation results into a form that does not require inverse transformation: in, A random number between 0 and 1. For any spatial transformation that includes elastic deformation; Using the Dice loss function With focus loss function The combination of these is used as the segmentation loss function: in, This refers to the number of pixels in the intersection of the segmentation label gold standard and the segmentation result. and These represent the gold standard for segmentation and the number of pixels in the segmentation result, respectively. For smoothing coefficients, The segmentation output probability for each pixel. The modulation coefficients for focus loss, This represents the number of pixels in the entire image.

7. A multimodal medical image elastic registration system, using the method as described in any one of claims 1 to 6, characterized in that: It includes a modality-independent semantic representation extraction module, a deep semantic representation spatial difference learning module, an image registration module, a segmentation-guided contrast learning module, and an iterative training module; among which, The modality-independent semantic representation extraction module is used to extract modality-independent semantic representations of fixed and floating images respectively from two encoders that share parameters in a deep learning elastic registration network with a dual-stream parameter-shared encoder and no decoder. The deep semantic representation spatial difference learning module is used to learn the spatial differences between deep semantic representations through the deformation field generation module in the registration network to obtain a sparse deformation field. The image registration module is used to apply the deformation field to the floating image through the sampler in the registration network to obtain a preliminarily registered floating image; The segmentation-guided contrastive learning module is used to input the fixed image and the initially registered floating image into a segmentation-guided contrastive learning network with shared parameters. The feature maps of multiple levels output by the encoders of the two segmentation-guided contrastive learning networks are compared and learned. The encoder structure of the segmentation-guided contrastive learning network is consistent with the encoder structure in the registration network. The two segmentation-guided contrastive learning networks are supervised only by the segmentation labels of the fixed image. The iterative training module is used to take the registered floating image as the input of the segmentation-guided contrastive learning network, and use the encoder network parameters obtained from the segmentation-guided contrastive learning network as the initialization of the encoder in the registration network for iterative training.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images