Construction method of multi-modal three-dimensional medical image segmentation registration model and application thereof

Abstract

The application belongs to the field of medical image registration, and particularly relates to a method for constructing a multi-modal three-dimensional medical image segmentation and registration model and application thereof, which comprises the following steps: collecting two kinds of modal medical images of each target as a reference image and a floating image respectively to obtain a training sample; simultaneously optimizing three network parameters by using the training sample to obtain a segmentation and registration model composed of a reference image segmentation model, a floating image segmentation model and a registration model; the reference image and the floating image segmentation model are respectively used for multi-scale segmentation of the corresponding images to obtain multi-scale segmentation results of the maximum scale same as the original images; the registration model is used for obtaining a multi-scale deformation field based on the reference image, the floating image, the maximum scale reference image segmentation result and the floating image segmentation result; and the segmentation loss and the registration loss are all the sum of the segmentation and registration losses at each scale, and the segmentation loss comprises a first gradient loss and / or a level set energy function loss.

Description

Technical Field

[0001] This invention belongs to the field of medical image registration, and more specifically, relates to a method for constructing a multimodal three-dimensional medical image segmentation and registration model and its application. Background Technology

[0002] In recent years, the incidence of soft tissue cancers has increased rapidly, and the proportion of malignant tumors among causes of death has also increased year by year. Cancer has become a major factor threatening human life and health, and the best strategy for dealing with cancer is early diagnosis and treatment. Taking prostate cancer as an example, biopsy is the gold standard for diagnosing prostate cancer. An accurate prostate biopsy navigation system can effectively improve the positive rate of the biopsy and reduce the trauma caused. Transrectal ultrasound is often used for prostate biopsy navigation because it is convenient, radiation-free, and has good real-time performance. However, its ability to distinguish tumors is poor, which can easily lead to inaccurate target localization. Magnetic resonance imaging (MRI) has good resolution for soft tissues and can distinguish different types of lesions in the prostate. Therefore, people are very concerned about how to synchronously display the rich information in three-dimensional MRI images in ultrasound images. The key technology is the real-time and accurate registration of three-dimensional MRI ultrasound images.

[0003] Currently, many scholars have proposed using deep learning for image registration. The mainstream approach is to directly predict pixel-level deformation fields through deep learning registration models to achieve end-to-end registration, avoiding iteration and thus accelerating the registration process. Most mainstream deep learning registration models are built on convolutional neural networks (CNNs), and some researchers combine CNNs with Transformers to improve performance. These models mostly employ encoding and decoding frameworks and only output deformation parameters or deformation fields at the end of the model. Their designs do not fully consider the characteristics of the registration task (requiring a progression from low to high resolution, and from small-image registration to large-image registration), resulting in limited registration performance.

[0004] Another problem with existing deep learning registration methods is the difficulty in obtaining realistic deformation fields for training. Early supervised registration methods trained deep learning models by labeling specific anatomical structures or randomly generating known deformation fields. This label acquisition is costly and difficult to apply effectively in real-world tasks. Therefore, unsupervised methods have been proposed for medical image registration. The principle of unsupervised registration methods is to first apply the predicted deformation field to the floating image, and then construct a loss function by evaluating the difference between the registered image and the reference image using a designed similarity metric function (such as mean squared error and normalized cross-correlation). Its performance is closely related to the similarity metric function. Clear hierarchical structures can be observed in prostate MRI images, while the internal tissue structures of the prostate are relatively blurry in ultrasound images. The similarity between the two modalities is small, therefore existing loss functions are difficult to use for training prostate MRI and ultrasound image registration models.

[0005] To address the image registration problem between modal images with low similarity, such as MRI and ultrasound images, some researchers have proposed first extracting the target region through image segmentation before performing registration. This method can construct a similarity loss based on the contour of the target region and eliminate interference from the background region. However, errors in the segmentation model can affect the subsequent registration performance. For example, training data with segmentation errors can easily lead to incorrect convergence and overfitting of the registration model. Summary of the Invention

[0006] To address the shortcomings and improvement needs of existing technologies, this invention provides a method for constructing a multimodal three-dimensional medical image segmentation and registration model and its application, with the aim of improving the image registration accuracy between multimodal three-dimensional medical images with low similarity.

[0007] To achieve the above objectives, according to one aspect of the present invention, a method for constructing a multimodal three-dimensional medical image segmentation and registration model is provided, comprising:

[0008] Two modalities of 3D medical images of multiple targets of the same type are acquired. One modality of the image is used as a reference image, and the other modality of the image is used as a floating image. The two modalities of 3D medical images corresponding to each target constitute a pair of images, which are used as a training sample to obtain a training sample set.

[0009] Using a training sample set, based on the sum of segmentation loss and registration loss, the parameters of the reference image segmentation network, the floating image segmentation network, and the registration network are trained and optimized simultaneously to obtain a segmentation and registration model consisting of the reference image segmentation model, the floating image segmentation model, and the registration model.

[0010] The reference image segmentation model and the floating image segmentation model are respectively applied to perform multi-scale segmentation on the target regions in the reference image and the floating image, resulting in multi-scale reference image segmentation results and multi-scale floating image segmentation results with the maximum scale size being the same as the original image size. The registration model is used to obtain a multi-scale deformation field based on the reference image, floating image, maximum scale reference image segmentation result, and maximum scale floating image segmentation result under each training sample, in order to achieve alignment and registration of the reference image and the floating image. The segmentation loss and the registration loss are both sums of the segmentation and registration losses at each scale, and the segmentation loss includes the first-order gradient loss and / or the level set energy function loss, used to quickly constrain the segmented region to be a uniform simply connected region.

[0011] Furthermore, the network architecture of the reference image segmentation model, the floating image segmentation model, and the registration model is the same, all being multi-scale self-attention networks.

[0012] Furthermore, during training, the segmentation loss of the reference image segmentation model and the floating image segmentation model... All are:

[0013]

[0014]

[0015]

[0016]

[0017]

[0018] In the formula, and These represent the Dice loss, cross-entropy loss, first-order gradient loss, and level set energy function loss, respectively; T represents the multi-scale segmentation label of the reference image or floating image; S represents the multi-scale reference image segmentation result or the multi-scale floating image segmentation result; I represents the reference image or floating image; i represents different resolutions; m i Ω represents the total number of pixels in an image with resolution i; j represents different pixels; Ω represents the image domain.

[0019] Furthermore, during training, the registration loss of the registration model... for:

[0020]

[0021]

[0022]

[0023]

[0024] In the formula, T represents the Dice loss, mean squared error loss, and regularization loss, respectively; m ,T f The segmentation labels represent the floating image and the reference image; φ represents the deformation field; λ and β are the coefficients, respectively; This represents the deformation field φ based on resolution i. i Transform the floating image labels of resolution i using linear interpolation. σ(·) represents an activation function with a range of [0, +∞).

[0025] Furthermore, λ takes the value of 10, and β takes the value of 0.1.

[0026] Furthermore, the multi-scale self-attention network includes: an encoding module, and n cascaded decoding modules, an encoding / decoding branch feature fusion module, and a self-attention module; n is the total number of scales corresponding to the multi-scale.

[0027] The encoding module is used to encode the received 3D image at different scales to obtain a feature map f with the same resolution as the input 3D image. e 1 and n features with resolutions lower than the feature map f e 1 Feature maps at different resolutions;

[0028] The cascaded n groups of decoding modules, encoding / decoding branch feature fusion modules, and self-attention modules are used to extract features from the lowest resolution feature map obtained by encoding, and upsample it to the next higher resolution corresponding to the current feature extraction result in the encoding, according to the encoding feature map resolution from low to high. The encoding / decoding branch feature fusion module fuses the feature map output by the decoding module in the same group with the feature map obtained by encoding at the same resolution to obtain a decoded feature map. The self-attention module calculates the self-attention principle based on the decoded feature map and the image with the same resolution obtained by resampling the stitched image, to obtain the self-attention feature map and the deformation field or segmentation result of the corresponding resolution. The self-attention feature map is used as the input of the decoding module in the second group of decoding modules, encoding / decoding branch feature fusion modules, and self-attention modules. The above process is repeated until the self-attention module in the nth group of decoding modules, encoding / decoding branch feature fusion modules, and self-attention modules outputs the deformation field or segmentation result of the original image resolution.

[0029] Furthermore, the encoding module includes convolutional layers, as well as n parallel sets of residual convolutional modules and convolutional downsampling modules; wherein, the convolutional layers are used to extract feature maps from the received 3D image to obtain feature map f. e 1 The cascaded n residual convolutional modules and convolutional downsampling modules are used to extract features and resample the feature map output from the previous stage, respectively, to obtain n feature maps with resolutions lower than the feature map f. e 1 Feature maps at different resolutions;

[0030] The decoding module includes a residual convolution module and a convolution upsampling module; wherein, the residual convolution module is used to extract features from the received 3D image; and the convolution upsampling module is used to upsample the result of the feature extraction.

[0031] Furthermore, the self-attention module performs the calculations based on the self-attention principle as follows:

[0032]

[0033]

[0034]

[0035]

[0036] In the formula, This represents the decoded feature map with resolution i received by the self-attention module. The self-attention component is obtained by decomposing the two-dimensional matrix data into multiple locally related regions on an average basis in the spatial dimension and compressing each region into a vector, and then embedding it through a fully connected layer. These represent the two self-attention components corresponding to the two-dimensional matrix data obtained after the self-attention module receives the stitched image, which is then resampled to obtain an image of resolution i. After being decomposed into multiple locally relevant regions in the spatial dimension and compressed into vectors respectively, the resulting two-dimensional matrix data is then embedded through two different fully connected layers; ps represents the side length of the locally relevant region; unflatten represents the dimension operation function for matrix decomposition into image features; LayerNorm(·) represents layer normalization calculation; FFN represents the feedforward neural network; This represents a self-attention feature map with resolution i.

[0037] This invention also provides a multimodal three-dimensional medical image segmentation and registration method, wherein the segmentation and registration model constructed using the method described above performs the following steps:

[0038] Two modalities of 3D medical images of the target are acquired, with one modality serving as a reference image and the other modality serving as a floating image;

[0039] The target region in the reference image is segmented at multiple scales using the reference image segmentation model in the segmentation and registration model to obtain a multi-scale reference image segmentation result; the target region in the floating image is segmented at multiple scales using the floating image segmentation model in the segmentation and registration model to obtain a multi-scale floating image segmentation result.

[0040] The segmentation results of the reference image, the floating image, the maximum scale reference image, and the maximum scale floating image are input into the registration model in the segmentation registration model to obtain the multi-scale deformation field.

[0041] Based on the deformation field of the largest scale in the multi-scale deformation field, each pixel in the floating image is displaced to achieve alignment and registration between the reference image and the floating image.

[0042] The present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed by a processor, it controls the device where the storage medium is located to execute a method for constructing a multimodal three-dimensional medical image segmentation and registration model as described above and / or a multimodal three-dimensional medical image segmentation and registration method as described above.

[0043] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0044] (1) This invention provides a novel method for constructing a registration model, innovatively proposing the construction of a segmentation-registration model, that is, the simultaneous training of the segmentation and registration models, collectively referred to as the segmentation-registration model. Specifically, for the segmentation and registration of medical images, a multi-scale network is proposed to perform multi-scale segmentation and registration respectively. At each resolution level in the segmentation and registration process, a deformation field or segmentation result map is generated. In the construction of the loss function, the segmentation loss and the registration loss are both the sum of the segmentation and registration losses at each scale, and the parameters of all network models are adjusted based on the sum of the segmentation loss and the registration loss. This simultaneous training method, because the parameter adjustment of the segmentation model essentially improves the registration accuracy by increasing segmentation accuracy, avoids the overfitting problem caused by independent training of segmentation and registration in existing methods, thus improving the training effect of the registration model. Secondly, compared with end-to-end registration models, the deformation field output of the final segmentation-registration model obtained by this method has undergone multiple optimizations from overall to fine-grained, resulting in higher registration accuracy. Furthermore, the inclusion of a level set energy function and gradient loss in the segmentation loss is used to quickly constrain the segmented regions to uniform simply connected domains in the early stages of training, ensuring the correctness of the early training of the registration model. Therefore, this invention improves the overall registration performance while enabling simultaneous segmentation and registration model training.

[0045] (2) The multi-scale self-attention network proposed in this invention has a self-attention module at each resolution level in the decoding process. Under the guidance of the original input, it uses the input features to generate a deformation field or segmentation map corresponding to the resolution. By introducing a cascaded update process from low resolution to high resolution, the performance of segmentation and registration is improved.

[0046] (3) The registration loss used in the method of the present invention includes constraints on the contour and constraints on deformation field regularization, which can reduce folding while improving registration accuracy. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of a method for constructing a multimodal three-dimensional medical image segmentation and registration model according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram showing the detailed structure of the segmentation and registration model provided in an embodiment of the present invention;

[0049] Figure 3 The images provided in this embodiment of the invention are visual representations of the registration results of Example 1 and Comparative Examples 1-4; wherein, (a) is the original multimodal image and the target region, and (b)-(f) are visual representations and deformation fields after registration using MSANet, Morph, TransMorph, ASNet, and Attention-Reg, respectively. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0051] Example 1

[0052] A method for constructing a multimodal 3D medical image segmentation and registration model includes:

[0053] S1. Acquire two modalities of three-dimensional medical images of multiple targets of the same type. One modal image is used as a reference image, and the other modal image is used as a floating image. The two modalities of three-dimensional medical images corresponding to each target constitute a pair of images as a training sample, thus obtaining a training sample set.

[0054] It should be noted that, considering that in actual medical image acquisition, the image may contain many areas other than the target organ, in order to improve computational efficiency, the industry usually crops the acquired 3D image and then performs pixel value normalization processing. The processed 3D image is used as the 3D medical image obtained in step S1 of this method.

[0055] S2. Using the training sample set, based on the sum of segmentation loss and registration loss, the parameters of the reference image segmentation network, floating image segmentation network and registration network are trained and optimized simultaneously to obtain a segmentation and registration model composed of the reference image segmentation model, the floating image segmentation model and the registration model.

[0056] like Figure 1As shown, the aforementioned reference image segmentation model and floating image segmentation model are applied to perform multi-scale segmentation on the target regions in the reference image and floating image, respectively, to obtain multi-scale reference image segmentation results and multi-scale floating image segmentation results with the maximum scale size being the same as the original image size. The aforementioned registration model is used to obtain a multi-scale deformation field based on the reference image, floating image, maximum scale reference image segmentation result, and maximum scale floating image segmentation result under each training sample, in order to achieve alignment and registration of the reference image and floating image. The aforementioned segmentation loss and the aforementioned registration loss are both sums of the segmentation and registration losses at each scale, and the aforementioned segmentation loss includes first-order gradient loss and / or level set energy function loss, used to quickly constrain the segmented region to be a uniform simply connected region.

[0057] It should be noted that before training, the industry typically first determines the network architecture and initializes the parameters of the reference image segmentation network, the floating image segmentation network, and the registration network. Furthermore, the aforementioned multi-scale segmentation can be achieved by first resampling the original image to obtain images at different scales (i.e., different resolutions). The segmentation network model then performs segmentation operations on the target regions in the images at different scales, resulting in multi-scale segmentation results. The resampling operation can be considered one of the functions of the segmentation network model itself. The aforementioned deformation field consists of the displacement of each pixel in the reference image relative to the corresponding point in the floating image.

[0058] This embodiment presents a novel registration model construction method. It innovatively proposes to construct a segmentation registration model, which involves simultaneously training the segmentation and registration models, collectively referred to as the segmentation registration model. This method provides a simultaneous segmentation and registration method based on a multi-scale network. The method introduces a multi-scale module and proposes a loss function for simultaneous segmentation and registration to enhance registration performance. The trained model can perform registration for target regions in multimodal medical images.

[0059] As a preferred embodiment, the network architectures of the above-mentioned reference image segmentation model, the above-mentioned floating image segmentation model, and the above-mentioned registration model are all multi-scale self-attention networks, which can achieve high accuracy.

[0060] The training process described above utilizes the multi-scale output generated by the model, the original multi-modal images, and the segmentation labels corresponding to each modality image to construct a loss function. The training objective is to minimize this loss function. Specifically, the original images and their corresponding segmentation labels are downsampled to multiple different resolutions to construct multi-scale losses. The segmentation loss is the sum of the losses corresponding to the multi-scale segmentation, and the registration loss, depending on the characteristics of the loss term, also involves the sum of the losses at multiple scales.

[0061] The first-order gradient loss mentioned above can be specifically described as follows:

[0062]

[0063] In the formula, S represents the multi-scale reference image segmentation result or the multi-scale floating image segmentation result; i represents different resolutions; m i This represents the total number of pixels in the image corresponding to resolution i.

[0064] The energy function loss of the level set mentioned above can be specifically described as follows:

[0065]

[0066] In the formula, I represents the original reference image or floating image (i.e., the original image after cropping and normalization); S represents the multi-scale reference image segmentation result or the multi-scale floating image segmentation result; i represents different resolutions; j represents different pixels; and Ω represents the image domain.

[0067] Therefore, as a preferred implementation, during the training process, the segmentation loss of the above-mentioned reference image segmentation model and the floating image segmentation model can be... All are:

[0068]

[0069]

[0070]

[0071]

[0072]

[0073] In the formula, and Let represent the Dice loss, cross-entropy loss, first-order gradient loss, and level set energy function loss, respectively; T represent the multi-scale segmentation label of the reference image or floating image; S represent the multi-scale reference image segmentation result or the multi-scale floating image segmentation result; I represent the original reference image or floating image; i represent different resolutions; m i Ω represents the total number of pixels in an image with resolution i; j represents different pixels; Ω represents the image domain.

[0074] As a preferred implementation, during the training process, the registration loss of the above registration model... for:

[0075]

[0076]

[0077]

[0078]

[0079] In the formula, T represents the Dice loss, mean squared error loss, and regularization loss, respectively; m ,T f The segmentation labels represent the floating image and the reference image; φ represents the deformation field; λ and β are coefficients, preferably λ is 10 and β is 0.1. This represents the deformation field φ based on resolution i. i Transform the floating image labels of resolution i using linear interpolation. σ(·) represents an activation function with a range of [0, +∞).

[0080] This can be used as a preferred implementation method, such as Figure 2 As shown, the multi-scale self-attention network includes: an encoding module, and n cascaded decoding modules, an encoding / decoding branch feature fusion module, and a self-attention module; n is the total number of scales corresponding to the multi-scale network.

[0081] The encoding module encodes the received 3D image at different scales to obtain a feature map f with the same resolution as the input 3D image. e 1 and n feature maps with resolutions lower than the above feature map f e 1 Feature maps at different resolutions (e.g., four feature maps with resolutions of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the original resolution) e 1 / 2 f e 1 / 4 f e 1 / 8 f e 1 / 16 It should be further explained that if a multi-scale self-attention network is used as a segmentation model, the 3D image received by the encoding module is either a reference image or a floating image. If a multi-scale self-attention network is used as a registration model, the 3D image received by the encoding module is a 3D image composed of the reference image, the floating image, the segmentation result of the largest scale reference image, and the segmentation result of the largest scale floating image.

[0082] The aforementioned cascaded n-group decoding modules, encoding / decoding branch feature fusion module, and self-attention module are used to extract features from the lowest-resolution feature map obtained from the encoding, and upsample it to the next higher resolution in the encoding, according to the encoding feature map resolution from low to high. The encoding / decoding branch feature fusion module then fuses the feature map output by the decoding module in the same group with the feature map obtained from the encoding at the same resolution to obtain a decoded feature map. Finally, the self-attention module, based on this decoded feature map and the resolution obtained by resampling the stitched image, combines the decoded feature map with the resolution obtained from the encoding. Images with the same image resolution are processed using the self-attention principle to obtain self-attention feature maps and deformation fields or segmentation results at the corresponding resolutions (if a multi-scale self-attention network is used as a segmentation model, the result obtained here is the segmentation result; if a multi-scale self-attention network is used as a registration model, the result obtained here is the deformation field). This self-attention feature map is then used as input to the second group of decoding modules, the encoding / decoding branch feature fusion module, and the decoding module within the self-attention module. This process is repeated until the nth group of decoding modules, the encoding / decoding branch feature fusion module, and the self-attention module outputs the deformation field or segmentation result at the original image resolution. For example, if n is four, the deformation fields or segmentation results output by the four groups of decoding modules, the encoding / decoding branch feature fusion module, and the self-attention module within the self-attention module are respectively Y... 1 / 8 Y 1 / 4 Y 1 / 2 Y 1 The superscripts all represent multiples of the resolution relative to the original image resolution.

[0083] In a preferred embodiment, the encoding module includes a convolutional layer, and n sets of residual convolutional modules and convolutional downsampling modules connected in parallel; wherein, the convolutional layer is used to extract feature maps from the received 3D image to obtain feature map f. e 1 The cascaded n residual convolutional modules and convolutional downsampling modules are used to extract features and resample the feature map output from the previous stage, respectively, to obtain n feature maps with resolutions lower than the feature map f. e 1 Feature maps at different resolutions;

[0084] The aforementioned decoding module includes a residual convolution module and a convolution upsampling module; wherein, the residual convolution module is used to extract features from the received 3D image; and the convolution upsampling module is used to upsample the result of the feature extraction.

[0085] As a preferred embodiment, the self-attention module performs calculations based on the self-attention principle as follows:

[0086]

[0087]

[0088]

[0089]

[0090] In the formula, This represents the decoded feature map with resolution i received by the self-attention module. The self-attention component is obtained by decomposing the two-dimensional matrix data into multiple locally related regions on an average basis in the spatial dimension and compressing each region into a vector, and then embedding it through a fully connected layer. Let represent the two self-attention components corresponding to the two-dimensional matrix data obtained after the self-attention module receives the above-mentioned stitched image, which is then resampled to obtain an image of resolution i. After being decomposed into multiple locally relevant regions in the spatial dimension and compressed into a vector, respectively, and then embedded by two different fully connected layers; ps represents the side length of the locally relevant region; unflatten represents the dimension operation function for matrix decomposition into image features; LayerNorm(·) represents the layer normalization calculation; FFN represents the feedforward neural network; This represents a self-attention feature map with resolution i.

[0091] like Figure 2 As shown in Figure c, the above calculation process can be interpreted as follows: first, the decoded feature maps with resolution i received by the self-attention module are processed respectively. The image with resolution i obtained by resampling the stitched image is then decomposed into multiple locally correlated regions in the spatial dimension, and each locally correlated region is compressed into a vector to obtain the decoded feature map. The corresponding two-dimensional matrix and the two-dimensional matrix corresponding to the image with resolution i obtained by resampling the above-mentioned concatenated image; then, the two two-dimensional matrices are embedded through a fully connected layer respectively to obtain The above three feature vector groups (i.e. feature matrices) The output vector will be generated based on the self-attention principle. Output vector After being deconstructed into image features and Adding them together gives After layer normalization, the output Y at the current resolution can be generated through two feedforward neural networks (FFNs). i (Segmentation result S) i Or register the deformation field φ i ) and the characteristic f passed down ci .

[0092] As a preferred option, the above-mentioned encoding / decoding branch feature fusion, such as Figure 2 As shown in Figure c, the two features of the encoding and decoding branches at a certain resolution are denoted as f respectively. e i and f d i First f e i Combined with f d i Attention maps f are generated through convolution and activation functions. a i , then f a i with f e i Multiply and then with f d i Combining the features f generated by convolution to produce decoded features D i The calculation process is as follows:

[0093]

[0094]

[0095] Where i represents different resolutions, w1 i w2 i w3 i w4 i w5 i The different convolution parameters are represented by σ(·), which represents the ReLU activation function, and S(·) is the Sigmoid function.

[0096] Example 2

[0097] A multimodal three-dimensional medical image segmentation and registration method, using the segmentation and registration model constructed by the method described in Example 1 above, performs the following steps:

[0098] Two modalities of 3D medical images of the target are acquired, with one modality serving as a reference image and the other modality serving as a floating image;

[0099] Using the reference image segmentation model in the above segmentation and registration model, the target region in the reference image is segmented at multiple scales to obtain the multi-scale reference image segmentation result; using the floating image segmentation model in the above segmentation and registration model, the target region in the floating image is segmented at multiple scales to obtain the multi-scale floating image segmentation result.

[0100] The segmentation results of the reference image, the floating image, the maximum scale reference image, and the maximum scale floating image are input into the registration model in the segmentation registration model to obtain the multi-scale deformation field.

[0101] Based on the largest-scale deformation field in the above multi-scale deformation field, each pixel in the floating image is displaced to achieve alignment and registration between the reference image and the floating image.

[0102] In other words, the process involves acquiring a multimodal 3D medical image to be registered, cropping the image based on the region of interest, and then normalizing the pixel values. This image is then input into a trained segmentation and registration model to obtain the deformation field, ultimately leading to the registration result.

[0103] The relevant technical solutions are the same as in Embodiment 1, and will not be repeated here.

[0104] To better illustrate the effects of this invention, the following examples are provided for verification:

[0105] Based on the above method, as Example 1, the training and testing of a multimodal 3D medical image segmentation and registration model are carried out, specifically including the following steps:

[0106] (1) Data were obtained from the Prostate-MRI-US-Biopsy dataset in the publicly available database of The Cancer Imaging Archive (TCIA). 502 data points, including MRI and ultrasound images and their corresponding prostate labels, were matched and extracted. 452 data points were randomly selected for model training, and the remaining 50 data points were used for model testing. In preprocessing, the original images were resampled to a spatial resolution of 1 mm × 1 mm × 1 mm. During training, a random, equal-sized translation was added to each pair of images to enhance data diversity.

[0107] (2) Using the above training set, the proposed segmentation and registration model (MSANet) is trained using the loss function proposed in this invention. The optimal model parameters are loaded, applied to the test set, and the registration results of the prostate region in the image are output. Finally, the registration accuracy is quantitatively evaluated.

[0108] Furthermore, to verify the method of the present invention, the following comparative examples were designed (each comparative example uses the same dataset):

[0109] Comparative Example 1

[0110] The registration task of the prostate region in multimodal 3D medical images was implemented using Morph (VoxelMorph: A Learning Framework for Deformable Medical Image Registration. IEEE Trans. Med. Imaging 38, 1788–1800, 2019), trained using the same dataset, learning rate, number of iterations, and optimizer parameters as the method of this invention.

[0111] Comparative Example 2

[0112] TransMorph (Transformer for unsupervised medical image registration. Medical Image Analysis 82, 102615, 2022) was used to perform the registration task of the prostate region in multimodal 3D medical images. It was trained using the same dataset, learning rate, number of iterations, and optimizer parameters as the method of this invention.

[0113] Comparative Example 3

[0114] The registration task of the prostate region in multimodal 3D medical images was implemented using ASNet (Adversarial learning for mono-or multi-modal registration. Medical Image Analysis 58, 101545, 2019), trained using the same dataset, learning rate, number of iterations, and optimizer parameters as the method of this invention.

[0115] Comparative Example 4

[0116] The Attention-Reg method (Cross-modal attention for multi-modal image registration. Medical Image Analysis 82, 102612, 2022) was used to perform the registration task of the prostate region in multimodal 3D medical images. It was trained using the same dataset, learning rate, number of iterations, and optimizer parameters as the method of this invention.

[0117] Results analysis:

[0118] To demonstrate the advantages of the present invention, the registration performance of Example 1 is compared with that of Comparative Examples 1-4. Quantitative comparisons are made using the Dice similarity coefficient (DSC), 95% Hausdorff distance (HD95), and the proportion of pixels less than 0 in the Jacobian determinant matrix of the deformation field (|J).φ The evaluation is performed if |≤0), defined as follows:

[0119]

[0120] HD95 = max k95％ [d(X,Y),d(Y,X)];

[0121]

[0122]

[0123] Where maxk95% represents taking the maximum value of 95%, and X and Y represent the set of two boundary points of the region used to calculate the index.

[0124] Table 1 lists the quantitative evaluation results of the registration results of Example 1 and Comparative Examples 1-4. It can be seen that the MSANet proposed in this invention achieves the highest average DSC and the lowest average HD95, with the lowest variance for both. Furthermore, the average folding ratio of the deformation field predicted by MSANet is 0.001, and the variance is 0.0003, both lower than the results of Comparative Examples 1-4. These results demonstrate that this invention achieves more accurate registration of the prostate in three-dimensional MRI and ultrasound images.

[0125] Table 1 shows the DSC, HD95, and folding ratio of each registration method on the selected dataset.

[0126]

[0127] To more intuitively demonstrate the superiority of the present invention, visual effect diagrams of the registration results corresponding to Example 1 and Comparative Examples 1-4, and visualization diagrams of their corresponding deformation fields are provided, as follows: Figure 3 As shown, (a) displays the original medical image and its corresponding prostate region label. (b)-(f) show the registration results of MSANet, Morph, TransMorph, ASNet, and Attention-Reg, respectively, indicating the degree of overlap of the prostate region after multimodal image registration. Clearly, the method of this invention demonstrates good consistency between the prostate region in ultrasound and MRI images after registration.

[0128] The above embodiments, taking the registration of the prostate region in ultrasound and MRI images as an example, fully illustrate that the present invention has higher multimodal three-dimensional medical image registration accuracy. Embodiment 1 is based on deep learning and can be used for multimodal three-dimensional medical image registration tasks.

[0129] The above embodiments are merely examples. Besides ultrasound and MRI images, the method of the present invention is also applicable to the registration of three-dimensional medical images of other modalities such as ultrasound and CT. Furthermore, it is also applicable to the registration of medical images of other regions such as the kidneys.

[0130] Example 3

[0131] A computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, it controls the device where the storage medium is located to perform a method for constructing a multimodal three-dimensional medical image segmentation and registration model as described in Embodiment 1 above and / or a multimodal three-dimensional medical image segmentation and registration method as described in Embodiment 2.

[0132] The relevant technical solutions are the same as in Embodiment 1, and will not be repeated here.

[0133] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing a multimodal three-dimensional medical image segmentation and registration model, characterized in that, include: Two modalities of 3D medical images of multiple targets of the same type are acquired. One modality of the image is used as a reference image, and the other modality of the image is used as a floating image. The two modalities of 3D medical images corresponding to each target constitute a pair of images, which are used as a training sample to obtain a training sample set. Using a training sample set, based on the sum of segmentation loss and registration loss, the parameters of the reference image segmentation network, the floating image segmentation network, and the registration network are trained and optimized simultaneously to obtain a segmentation and registration model consisting of the reference image segmentation model, the floating image segmentation model, and the registration model. The reference image segmentation model and the floating image segmentation model are respectively used to perform multi-scale segmentation on the target regions in the reference image and the floating image, respectively, to obtain multi-scale reference image segmentation results and multi-scale floating image segmentation results with the same maximum scale size as the original image size; the registration model is used to obtain a multi-scale deformation field based on the reference image, floating image, maximum scale reference image segmentation result, and maximum scale floating image segmentation result under each training sample, so as to achieve alignment and registration of the reference image and the floating image; the segmentation loss and the registration loss are both the sum of the segmentation and registration losses at each scale, and the segmentation loss includes the first-order gradient loss and the level set energy function loss, which are used to quickly constrain the segmented region to be a uniform simply connected region.

2. The construction method according to claim 1, characterized in that, The reference image segmentation model, the floating image segmentation model, and the registration model have the same network architecture, which is a multi-scale self-attention network.

3. The construction method according to claim 1, characterized in that, During training, the segmentation loss of the reference image segmentation model and the floating image segmentation model are... All are: ； ； ； ； ； In the formula, , , and These represent the Dice loss, cross-entropy loss, first-order gradient loss, and level set energy function loss, respectively. Multi-scale segmentation labels representing reference images or floating images; This indicates the segmentation result of a multi-scale reference image or a multi-scale floating image. Indicates a reference image or a floating image; Indicates different resolutions; m i Indicates resolution as i The total number of pixels in the image; Ω represents different pixels; Ω represents the image domain.

4. The construction method according to claim 1, characterized in that, During training, the registration loss of the registration model... for: ； ； ； ； In the formula, These represent Dice loss, mean squared error loss, and regularization loss, respectively. Segmentation labels representing the floating image and the reference image; Represents the deformation field; , They are coefficients, respectively. Indicates based on resolution i Deformation field The resolution is transformed using linear interpolation. i Floating image labels ; Indicates the range is Activation function; Represents the image domain; Indicates resolution as i The total number of pixels in the image.

5. The construction method according to claim 4, characterized in that, The value is 10. The value is 0.

1.

6. The construction method according to claim 2, characterized in that, The multi-scale self-attention network corresponding to the registration model includes: an encoding module, and n cascaded decoding modules, an encoding / decoding branch feature fusion module, and a self-attention module; n is the total number of scales corresponding to the multi-scale. The encoding module is used to encode the received 3D image at different scales to obtain a feature map with the same resolution as the input 3D image. f e 1 and n features with resolutions lower than the aforementioned feature map f e 1 Feature maps of different resolutions; the three-dimensional image received by the encoding module is a three-dimensional image obtained by stitching together the reference image, the floating image, the segmentation result of the maximum scale reference image, and the segmentation result of the maximum scale floating image. The cascaded n groups of decoding modules, encoding / decoding branch feature fusion modules, and self-attention modules are used to extract features from the lowest resolution feature map obtained by encoding, and upsample it to the next higher resolution corresponding to the current feature extraction result in the encoding, according to the encoding feature map resolution from low to high. The encoding / decoding branch feature fusion module fuses the feature map output by the decoding module in the same group with the feature map obtained by encoding at the same resolution to obtain a decoded feature map. The self-attention module calculates the self-attention principle based on the decoded feature map and the image with the same resolution obtained by resampling the stitched image, to obtain the self-attention feature map and the deformation field or segmentation result of the corresponding resolution. The self-attention feature map is used as the input of the decoding module in the second group of decoding modules, encoding / decoding branch feature fusion modules, and self-attention modules. The above process is repeated until the self-attention module in the nth group of decoding modules, encoding / decoding branch feature fusion modules, and self-attention modules outputs the deformation field or segmentation result of the original image resolution.

7. The construction method according to claim 6, characterized in that, The encoding module includes convolutional layers, as well as n cascaded residual convolutional modules and convolutional downsampling modules; wherein, the convolutional layers are used to extract feature maps from the received 3D image to obtain feature maps. f e 1 The cascaded n residual convolutional modules and convolutional downsampling modules are used to extract features and resample the feature map output from the previous stage, respectively, to obtain n feature maps with resolutions lower than the original feature map. f e 1 Feature maps at different resolutions; The decoding module includes a residual convolution module and a convolution upsampling module; wherein, the residual convolution module is used to extract features from the received 3D image; and the convolution upsampling module is used to upsample the result of the feature extraction.

8. The construction method according to claim 6, characterized in that, The self-attention module performs calculations based on the self-attention principle as follows: ； ； ； ； In the formula, This indicates that the resolution received by the self-attention module is... i Decoding feature map The self-attention component is obtained by decomposing the two-dimensional matrix data into multiple locally related regions on an average basis in the spatial dimension and compressing each region into a vector, and then embedding it through a fully connected layer. These represent the resolutions obtained by resampling the stitched image received by the self-attention module. i The image is decomposed into multiple locally related regions in the spatial dimension and compressed into a vector respectively. The resulting two-dimensional matrix data is then embedded through two different fully connected layers, resulting in two self-attention components. This represents the side length of the locally relevant region; `unflatten` represents the dimension manipulation function for decomposing the matrix into image features. Representation layer normalization calculation; This represents a feedforward neural network; Indicates resolution as i Self-attention feature map.

9. A multimodal three-dimensional medical image segmentation and registration method, characterized in that, The segmentation registration model constructed using the construction method described in any one of claims 1 to 8 shall undergo the following steps: Two modalities of 3D medical images of the target are acquired, with one modality serving as a reference image and the other modality serving as a floating image; The target region in the reference image is segmented at multiple scales using the reference image segmentation model in the segmentation and registration model to obtain a multi-scale reference image segmentation result; the target region in the floating image is segmented at multiple scales using the floating image segmentation model in the segmentation and registration model to obtain a multi-scale floating image segmentation result. The segmentation results of the reference image, the floating image, the maximum scale reference image, and the maximum scale floating image are input into the registration model in the segmentation registration model to obtain the multi-scale deformation field. Based on the deformation field of the largest scale in the multi-scale deformation field, each pixel in the floating image is displaced to achieve alignment and registration between the reference image and the floating image.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed by a processor, it controls the device where the storage medium is located to perform a method for constructing a multimodal three-dimensional medical image segmentation and registration model as described in any one of claims 1 to 8 and / or a multimodal three-dimensional medical image segmentation and registration method as described in claim 9.

Images