Transrectal ultrasound and MR image registration method

Abstract

The invention provides a transrectal ultrasound and MR image registration method, and relates to the technical field of image processing. The method comprises the following steps: firstly, acquiring an MR image and a TRUS image to be registered, and marks corresponding to the images to form a multi-modal image data set; the MR image is regarded as a floating image, and the TRUS image is regarded as a fixed image; the two images and the corresponding marks are preprocessed; then, constructing a feature extraction module, and carrying out image-level feature extraction on the two preprocessed images; respectively extracting the features corresponding to the two images under five layers of different resolutions; carrying out fusion decoding on two groups of features under five layers of different resolutions layer by layer from low resolution to high resolution, and generating a deformation field; and obtaining a deformed image, namely an MR image which is finally aligned with a fixed image space, by adopting a spatial transformation network according to a feature pair generated deformation field under the highest resolution. According to the method, accurate registration of the TRUS image and the MR image can be realized.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for registering transrectal ultrasound and MR images. Background Technology

[0002] Cancer has become a major public health issue. For patients with advanced cancer or those with contraindications to surgery, combined radiotherapy and chemotherapy is the preferred treatment option. Brachytherapy, in particular, delivers high-dose radiation precisely via an intracavitary applicator, maximizing the local dose to the tumor while protecting surrounding healthy tissues. The treatment process includes key steps such as pre-brachytherapy MR scanning (starting only before the first intracavitary radiotherapy), transrectal ultrasound imaging (TRUS), applicator implantation, planning CT scan, target delineation using a three-dimensional treatment planning system, and dose planning. MR images acquired before the first intracavitary radiotherapy can be used to assess the initial location and size of the tumor. However, the morphology and location of the tumor may change as treatment progresses. Furthermore, to more accurately guide the implantation of the insertion needle, it is necessary to register real-time transrectal ultrasound images with pre-brachytherapy MR images to integrate image information from different time points and modalities, providing reliable visual evidence for intraoperative decisions.

[0003] In the current field of automated medical image registration, registration networks such as VoxelMorph, TransMorph, and MambaMorph are mainly used to achieve fast and automated image registration. These methods typically stitch the image to be registered with a reference image together and feed them into a network model. Through extensive data learning, the network captures features of the image pair. Finally, these high-level features are fused and decoded to convert them into the deformation field required for registration. Although existing registration methods have made significant progress, they face unique challenges in registering TRUS and MR images: TRUS, as a two-dimensional imaging technique, has a narrow field of view and is dependent on the operating angle. The lesion area in the image lacks reference to normal tissue structures, and there is also interference from applicator artifacts; MR images, on the other hand, have complete three-dimensional spatial information and clear anatomical structural references. This modal difference leads to difficulties in feature extraction, complex spatial mapping, significant artifact interference, and a lack of reliable spatial references in TRUS-MR registration. Existing registration methods struggle to effectively overcome these technical challenges. Furthermore, the data features extracted by existing methods are limited, resulting in insufficient expressive power of the model and thus restricting the accuracy of registration. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for registering transrectal ultrasound and MR images in order to overcome the shortcomings of the prior art and achieve accurate registration of transrectal ultrasound and MR images.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] On one hand, the present invention provides a method for registering transrectal ultrasound and MR images, comprising:

[0007] The MR and TRUS images to be registered, along with their corresponding annotations, are acquired to form a multimodal image dataset. The dataset is then divided into training, validation, and test sets based on the patient.

[0008] Treat MR images as floating images The corresponding label is denoted as TRUS images are considered fixed images. The corresponding label is And for the two images and corresponding markers Preprocessing is required;

[0009] A feature extraction module is constructed to extract features from the two preprocessed images. and Image-level feature extraction is performed, and the extracted features are denoted as features. and ;

[0010] Extract features separately and Features at 5 different resolutions and ;

[0011] Two sets of features at different resolutions across 5 layers and The deformation field is generated by fusing and decoding layer by layer from low to high resolution.

[0012] A spatial transformation network is used to apply the generated deformation field to the floating image based on the features at the highest resolution. Interpolation is performed, and then the original size is sampled back to obtain the deformed image. ,image That is, the final and fixed image Spatially aligned MR images.

[0013] Furthermore, the preprocessing of the two images and their corresponding markers includes:

[0014] For the two images and Cropping was performed separately to remove the black background areas from the images. Similarly, corresponding annotations were applied to both images. and Each image was cropped according to its own cropping criterion; finally, linear interpolation was used to adjust the resolution of the two images and annotations to be consistent.

[0015] Furthermore, the feature extraction module is built based on the Unet feature extraction network. The feature extraction module uses the same Unet feature extraction network structure for both images and shares parameters during training.

[0016] Furthermore, the method uses the MMC module to extract features respectively. and Features at 5 different resolutions and ;

[0017] Feature extraction using the MMC module Features at 5 different resolutions The specific method is as follows:

[0018] The first convolutional block of the MMC module uses convolution + batch normalization + activation layer to process features. Feature extraction is performed to obtain the features of the first layer. ;

[0019] The second layer of the MMC module first uses average pooling to analyze the features. The pooled features are then reduced in frequency, and then convolution + batch normalization + activation layer is used to extract features from the pooled features to obtain the features of the second layer. ;

[0020] The third layer of the MMC module first uses Mamba convolutional blocks to process the features. Transform dimensional slices into features , for features Perform Mamba block operations and concatenate them back to their original dimensions according to the slice order to obtain the features. Then the features of the second layer Perform downsampling and convolution to expand the dimensions and obtain a feature. The two sets of features obtained and By accumulating element by element, the features of the third layer are obtained. ;

[0021] Features of the third layer Features are obtained by merging blocks. and The two are fed into a Mamba convolution block to obtain the features of the fourth layer. ;

[0022] Features of the fourth layer The features are fed into the block merge block and obtained. and Then, the two are fed into a Mamba convolution block to obtain features. ;

[0023] Feature extraction using the MMC module Features at 5 different resolutions Methods and Feature Extraction Features at 5 different resolutions The method is the same.

[0024] Furthermore, the features of the third layer Features are obtained by merging blocks. and The two are fed into a Mamba convolution block to obtain the features of the fourth layer. The specific method is as follows:

[0025] First, the features of the third layer... The algorithm scans along alternating rows and columns (0, 0; 1, 0; 0, 1; and 1, 1), concatenates the data along the dimensional channels, and then performs dimensionality reduction using a convolutional layer to obtain the features. Then the features Expand into features The final output of merging blocks is the feature. and ; Features and Feeding into the Mamba convolution block, for features After Mamba layer operations, the slices are pieced back to their original size according to the slice order and combined with the features. The fourth layer features are obtained by element-wise accumulation. .

[0026] Furthermore, the method employs an FFA module to analyze two sets of features at five different resolutions. and The FFA module performs layer-by-layer fusion decoding from low to high resolution to generate a deformation field; specifically, it uses feature fusion blocks and convolutional decoding blocks to fuse and decode two sets of features layer by layer from low to high resolution to generate a deformation field.

[0027] feature pairs and The data are fed separately into a feature fusion block for fusion to obtain the fused features. and ; for features Upsampling and features After feature concatenation, the decoded features are obtained by feeding them into the convolutional decoding block. Decoding features Deformation field prediction and generation ;

[0028] For deformation fields Perform upsampling operation and apply it to the features. Deformation is performed to obtain features , feature pair The fused features are fed into the feature fusion block to obtain the fused features. ;Decoding features After upsampling and fusion features After feature concatenation, the decoded features are obtained by feeding them into the convolutional decoding block. Decoding features Deformation field prediction and generation ;

[0029] Similarly, for feature pairs { , and{ , Execution and Feature Pair The same operation generates a deformation field. .

[0030] Furthermore, during the training process, the method uses the Dice loss before and after registration, as well as the smoothing loss for generating the deformation field, as constraints to train the feature extraction module, MMC module, and FFA module, and update the parameters therein; and implements an early stopping strategy on the test set to achieve optimal parameter selection for all modules.

[0031] On the other hand, the present invention also provides a transrectal ultrasound and MR image registration system, comprising:

[0032] The image acquisition module acquires the MR images and TRUS images to be registered, as well as the corresponding annotations, to form a multimodal image dataset. The dataset is then divided into training, validation, and test sets based on the patient.

[0033] The preprocessing module treats MR images as floating images. The corresponding label is denoted as TRUS images are considered fixed images. The corresponding label is And for the two images and corresponding markers Preprocessing is required;

[0034] The feature extraction module is used to construct the feature extraction module for the two preprocessed images. and Image-level feature extraction is performed, and the extracted features are denoted as features. and ; and extract features respectively and Features at 5 different resolutions and ;

[0035] The deformation field generation module generates two sets of features at five different resolutions. and The deformation field is generated by fusing and decoding layer by layer from low to high resolution.

[0036] The image alignment module uses a spatial transformation network to apply the generated deformation field to the floating image based on the features at the highest resolution. Interpolation is performed, and then the original size is sampled back to obtain the deformed image. ,image That is, the final and fixed image Spatially aligned MR images.

[0037] Thirdly, this application proposes a computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the transrectal ultrasound and MR image registration method.

[0038] Fourthly, this application proposes a computer program product, including a computer program or instructions that, when executed by a processor, implement the transrectal ultrasound and MR image registration method.

[0039] The beneficial effects of the above technical solution are as follows: The transrectal ultrasound and MR image registration method provided by this invention employs a multi-scale feature-aware fusion and attention-guided convolutional neural network to achieve accurate registration of transrectal ultrasound (TRUS) and pre-radiotherapy MR images. A multi-resolution Mamba convolutional module (MMC) extracts multi-level features from both modal images and deeply fuses their local and regional correlation information to address the difficulties in feature extraction and large modal differences. Based on this, a feature-aware fusion module (FFA) is introduced to fuse multi-scale features layer by layer from low resolution to high resolution and gradually generate deformation fields, achieving coarse-to-fine spatial alignment and effectively overcoming the problems of large structural deformation and complex spatial mapping between images. Finally, the annotation constraints of lesion sites enhance the anatomical rationality of the registration, significantly improving the robustness and accuracy of registration in cases where ultrasound images lack clear structural references. Attached Figure Description

[0040] Figure 1 A flowchart of a method for registering transrectal ultrasound and MR images provided in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the U-shaped feature extraction network structure provided in an embodiment of the present invention;

[0042] Figure 3This is a schematic diagram of the structure of the MMC module and FFA module provided in an embodiment of the present invention;

[0043] Figure 4 This is a visualization of the registration results of different registration methods provided in the embodiments of the present invention. Detailed Implementation

[0044] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0045] Example 1:

[0046] This embodiment uses MR and TRUS images of a cervical cancer patient as an example, and adopts the transrectal ultrasound and MR image registration method of the present invention to achieve the registration of TRUS and MR images of the cervix.

[0047] In this embodiment, a method for registering transrectal ultrasound and MR images is described, such as... Figure 1 As shown, it includes the following steps:

[0048] Step 1: Extract the MR and TRUS images belonging to the cervix to be registered from the MR and TRUS sequences of cervical cancer patients, along with the corresponding annotations, to form a multimodal cervical cancer image dataset, and divide the dataset into data units based on patients.

[0049] In this embodiment, TRUS and MR images of 53 cervical cancer patients were obtained and divided into a training set (31 cases), a validation set (11 cases), and a test set (11 cases) in a ratio of 3:1:1.

[0050] Step 2: Treat MR images as floating images The corresponding label is denoted as TRUS images are considered fixed images. The corresponding label is The two images and their corresponding markers were preprocessed.

[0051] For the two images and Cropping was performed separately to remove the black background areas from the images. Similarly, corresponding annotations were applied to both images. and Each image was cropped according to its own cropping criterion; finally, linear interpolation was used to adjust the resolution of the two images and annotations to be consistent.

[0052] Step 3: Construct a feature extraction module based on the Unet feature extraction network to extract features from the two preprocessed images. and As input, these are fed into the feature extraction module for image-level feature extraction. The extracted features are denoted as features. and ;

[0053] Feature extraction was performed on the preprocessed TRUS and MR images using a 3-layer Unet feature extraction network. The network structure is as follows: Figure 2 As shown; the Unet feature extraction network structure used for both images is the same, and the parameters are shared during training;

[0054] This embodiment uses MR images as an example to illustrate the feature extraction process of the Unet feature extraction network: Preprocessed image The features are first obtained by sequentially passing them through the first layer of the encoder (convolution + batch normalization + activation layer). ;

[0055] For features The feature is obtained by downsampling and passing it through the second layer of the encoder (convolution + batch normalization + activation layer). ;

[0056] For features The downsampling operation is performed, and the features are obtained by passing them through the third layer of the encoder (convolution + batch normalization + activation layer). ;

[0057] For features Perform upsampling to double the resolution, then compare it with the features. The features are concatenated and fed into the second layer of the decoder (convolution + batch normalization + activation layer) for feature decoding to obtain the features. ;

[0058] For features Perform upsampling to double the resolution, then compare it with the features. The features are concatenated and fed into the first layer of the decoder (convolution + batch normalization + activation layer) for feature decoding to obtain the features. ;

[0059] For features Perform dimensional transformation to obtain features This feature is the feature extracted by the feature extraction module. .

[0060] For TRUS images The feature extraction process using the Unet feature extraction network is the same as the feature extraction process for MR images, resulting in the following features. This feature is the feature extracted by the feature extraction module. ;

[0061] Step 4: Process the features extracted by the feature extraction module and Send them to, respectively Figure 3 In the MMC module shown, features are extracted. and Features at 5 different resolutions and ;

[0062] In this embodiment, the resolution decreases progressively from 1 to 5:

[0063] In this embodiment, the feature and The features extracted in the MMC module are divided into 5 resolutions. Both feature extraction and feature extraction are performed separately in the MMC module, using the same network structure and sharing parameters.

[0064] This embodiment uses features For example, the process of the MMC module extracting features at 5 different resolutions is given as follows:

[0065] The first convolutional block of the MMC module uses convolution + batch normalization + activation layer to process features. Feature extraction is performed to obtain the features of the first layer. .

[0066] The second layer of the MMC module first uses average pooling to analyze the features. The pooled features are then reduced in frequency, and then convolution + batch normalization + activation layer is used to extract features from the pooled features to obtain the features of the second layer. .

[0067] The third layer of the MMC module first uses Mamba convolutional blocks to process the features. Transform dimensional slice projection Features , for features Perform Mamba block operations and then concatenate the slices back to their original dimensions according to the slice order. Features; then the features of the second layer Perform downsampling and convolution to expand the dimensions to obtain a... Features The two sets obtained The features are accumulated element by element to obtain the features of the third layer. .

[0068] Features of the third layer The features of the fourth layer are obtained by merging blocks and using Mamba convolutional blocks. ;

[0069] The specific implementation in the block merging block is as follows: First, the features of the third layer are... Scan along row 0, column 0; row 1, column 0; row 0, column 1; and row 1, column 1, alternating rows and columns, and stitch the results along the dimensional channels to obtain... The characteristics will The features are reduced in dimension by convolutional layers to become Features Next, the features Expand into Features The final output of merging blocks is the feature. and ; Features and Feeding into the Mamba convolution block, for features After Mamba layer operations, the slices are pieced back to their original size according to the slice order. Features, and features The fourth layer features are obtained by element-wise accumulation. .

[0070] Features of the fourth layer The features are fed into the block merge block and obtained. and Then, the two are fed into a Mamba convolution block to obtain features. At this point, the features of the five different resolutions of the MMC module have all been extracted.

[0071] Step 5: Extract the features from the 5 layers at different resolutions using the MMC module. and Send in Figure 3 In the FFA module shown, a feature fusion block and a convolutional decoding block are used to fuse and decode two sets of features layer by layer from low to high resolution and generate a deformation field; the convolutional decoding block consists of two layers of convolution + batch normalization + activation layer.

[0072] feature pairs and The data are fed separately into a feature fusion block for fusion to obtain the fused features. and ; for features Upsampling and features After feature concatenation, the decoded features are obtained by feeding them into the convolutional decoding block. Decoding features Deformation field prediction and generation ;

[0073] For deformation fields Perform upsampling operation and apply it to the features. Deformation is performed to obtain features , feature pair The fused features are fed into the feature fusion block to obtain the fused features. ;Decoding features After upsampling and fusion features After feature concatenation, the decoded features are obtained by feeding them into the convolutional decoding block. Decoding features Deformation field prediction and generation ;

[0074] Similarly, for feature pairs { , and{ , Execution and Feature Pair The same operation generates a deformation field. ;

[0075] Step 5.1: The fusion and decoding process of the FFA module starts from the lowest resolution, that is, for feature pairs. First, the two channels are concatenated, and then processed through convolution + batch normalization + activation layers to obtain fused features. ; to integrate features Projection as The features are used as query vectors, and the features are... The projection is two The features are used as key vectors and value vectors, respectively; depthwise separable convolution combined with the Sigmoid function is used to calculate the features. Query vectors and features Cross attention of the key vectors and features The feature is obtained by multiplying the value vector element by element. ; to integrate features and characteristics Channel concatenation is performed, followed by convolutional dimensionality reduction, and then activation is achieved using the softmax function to obtain a result. Attention vector; fuse features The feature is multiplied element-wise with the value of channel 1 in the attention vector to obtain the feature. The values ​​of channel 2 in the attention vector are multiplied element-wise and then added together to obtain the fused decoded features. .

[0076] Step 5.2: For feature pairs First, the two channels are concatenated, and then processed through convolution + batch normalization + activation layers to obtain fused features. Features Projection as The features are used as query vectors, and the features are... The projection is two The features are used as key vectors and value vectors, respectively. Depthwise separable convolution combined with the Sigmoid function is employed to calculate the features. Query vectors and features Cross attention of the key vectors and features The feature is obtained by multiplying the value vector element by element. . Fusing features and characteristics Channel concatenation is performed, followed by convolutional dimensionality reduction, and then activation is achieved using the softmax function to obtain a result. The attention vector. The fused features. The feature is multiplied element-wise with the value of channel 1 in the attention vector to obtain the feature. The values ​​of channel 2 in the attention vector are multiplied element-wise and then added together to obtain the fused features. . Fuse low-resolution decoding features Upsampling and fusion features The features are obtained by concatenating the data and then decoding it through a convolutional decoding block. Then the features The data is fed into a deformation field generation head composed of convolutional blocks to generate a deformation field. .

[0077] Step 5.3: For feature pairs First, the deformation field Upsampling is performed to expand the partial variation to Then, based on the sampled deformation field, the features are... Perform deformation. Then, the deformed feature... and characteristics Channel concatenation is performed, followed by convolution, batch normalization, and activation layers to obtain fused features. Features Projection as The features are used as query vectors, and the features are... The projection is two The features are used as key vectors and value vectors, respectively. Depthwise separable convolution combined with the Sigmoid function is employed to calculate the features. query vector and Cross attention of the key vectors and features The feature is obtained by multiplying the value vector element by element. . Fusing features and characteristics Channel concatenation is performed, followed by convolutional dimensionality reduction, and then activation is achieved using the softmax function to obtain a result. The attention vector. The fused features. The feature is multiplied element-wise with the value of channel 1 in the attention vector to obtain the feature. The values ​​of channel 2 in the attention vector are multiplied element-wise and then added together to obtain the fused features. . Fuse low-resolution decoding features Upsampling and fusion features The features are obtained by concatenating the data and then decoding it through a convolutional decoding block. Then the features The data is fed into a deformation field generation head composed of convolutional blocks to generate a deformation field. .

[0078] Step 5.4: For feature pairs First, the deformation field Upsampling is performed to expand the partial variation to Then, based on the sampled deformation field, the features are... Perform deformation. Then, the deformed feature... and characteristics Channel concatenation is performed, followed by convolution, batch normalization, and activation layers to obtain fused features. Features Perform block division and projection as The features are used as query vectors, and the features are... Divide into blocks and project into two The features are used as key vectors and value vectors, respectively. Depthwise separable convolution combined with the Sigmoid function is employed to calculate the features. Query vectors and features Cross attention of the key vectors and features The feature is obtained by multiplying the value vector element by element. , will integrate features and characteristics Channel concatenation is performed, followed by convolutional dimensionality reduction, and then activation is achieved using the softmax function to obtain a result. The attention vector. The fused features. The feature is multiplied element-wise with the value of channel 1 in the attention vector to obtain the feature. The values ​​of channel 2 in the attention vector are multiplied element-wise and then added together to obtain the fused features. . Fuse low-resolution decoding features Upsampling and fusion features The features are obtained by concatenating the data and then decoding it through a convolutional decoding block. Then the features The data is fed into a deformation field generation head composed of convolutional blocks to generate a deformation field. .

[0079] Step 5.5: Targeting Features First, the deformation field Upsampling is performed to expand the partial variation to Then, based on the sampled deformation field, the features are... Perform deformation. Then, the deformed feature... and characteristics Channel concatenation is performed, followed by convolution, batch normalization, and activation layers to obtain fused features. Features Perform block division and projection as The features are used as query vectors, and the features are... Divide into blocks and project into two The features are used as key vectors and value vectors, respectively. Depthwise separable convolution combined with the Sigmoid function is employed to calculate the features. Query vectors and features Cross attention of the key vectors and features The feature is obtained by multiplying the value vector element by element. , will integrate features and characteristics Channel concatenation is performed, followed by convolutional dimensionality reduction, and then activation is achieved using the softmax function to obtain a result. The attention vector. The fused features. The feature is multiplied element-wise with the value of channel 1 in the attention vector to obtain the feature. The values ​​of channel 2 in the attention vector are multiplied element-wise and then added together to obtain the fused features. . Fuse low-resolution decoding features Upsampling and fusion features The features are obtained by concatenating the data and then decoding it through a convolutional decoding block. Then the features The data is fed into a deformation field generation head composed of convolutional blocks to generate a deformation field. .

[0080] Step 6: Using a Spatial Transformation Network (STN), based on the deformation field generated in Step 5... For floating images Interpolation is performed, and then the original size is sampled back to obtain the deformed image. ,image That is, the final and fixed image Spatially aligned MR images.

[0081] Step 7: Use Dice loss and deformation field smoothing loss to perform constrained training on the Unet feature extraction module, MMC module and FFA module involved in the method, update the parameters in them; and implement an early stopping strategy on the test set to achieve the optimal parameter selection for all modules.

[0082] Step 7.1: To enable each module to better learn the knowledge within the data, Dice loss before and after registration, as well as smoothing loss for generating the deformation field, are used as constraints during training.

[0083] The specific loss function is as follows:

[0084] (1);

[0085] in, The loss is calculated based on the deformation field generated in the i-th layer of the FFA module; p represents the number of pixels in the image. It is the set of all pixels in the image. For images Scale to the deformation field The value of pixel p in an image at the same resolution. For images Scale to the deformation field Images at the same resolution and after deformation field The value of pixel p in the transformed image. To find the gradient function, This is for L2 regularization.

[0086] Step 7.2: The overall loss formula during training is a weighted sum of the losses of the deformation fields output from layers 1 to 4 of the FFA module, as follows:

[0087] (2);

[0088] in, Let be the weight of the i-th layer in the FFA module. During implementation, the weights of layers 1 to 4 are set to 1 / 2, 1 / 6, 1 / 6 and 1 / 6 respectively.

[0089] The goal of this loss function is to minimize the smoothness of the deformation field in the horizontal and vertical directions of the four-layer prediction output of the FFA module, and to maximize the overlap between the deformed labels as constraints, thereby updating the parameters of each module. By introducing a test set, the optimal parameters of the model are obtained by determining whether the loss value on the test set is optimal. These optimal parameters are then saved.

[0090] This embodiment also provides a comparison of the registration results of the method of the present invention with existing medical image registration methods VoxelMorph, Transmorph, MambaMorph, and Vmambamorph, as shown in Table 1 and... Figure 4 As shown, the method of this invention performs better for registering multimodal TRUS and MR images that are difficult to extract features, have complex spatial mapping, significant artifact interference, and lack reliable spatial references. This is further demonstrated by the proportion of negative values ​​in the Dice similarity coefficient (DSC), 95% Hausdorff distance (HD95), and Jacobian determinant. They showed superior results in all evaluation metrics.

[0091] As shown in Table 1, the method of this invention significantly improves upon other methods in terms of DSC and HD95 indices. Specifically, the DSC index reaches 0.7489, and the HD95 index reaches 23.7529. Furthermore, the method of this invention exhibits the smallest amplification in the proportion of negative values ​​in the Jacobian determinant, indicating fewer folds in the predicted deformation field and suggesting a more reasonable generated deformation field.

[0092] Next, according to Figure 4 Qualitative analysis was performed. The first row shows the TRUS image, MR image, and image data after deformation fields generated by various methods were applied to the MR image for patient 1. The second row shows the TRUS image, MR image, and image data after deformation fields generated by various methods were applied to the MR image for patient 2. The first column shows the TRUS image; the second column shows the MR image before registration; the third column shows the registration result of the VoxelMorph method; the fourth column shows the registration result of the TransMorph method; the fifth column shows the registration result of the MambaMorph method; the sixth column shows the registration result of the VmambaMorph method; and the seventh column shows the registration result of the method of this invention. In the figure, the area inside the yellow line represents the cervical region in the TRUS image, while the area inside the blue line represents the cervical region in the MR image. Figure 4 As can be seen from this, the registration results of the method of the present invention have a higher degree of overlap with the annotation of the TRUS image.

[0093] Table 1. DSC, HD95, and values ​​of different methods on the test set.

[0094] Example 2:

[0095] This embodiment provides a transrectal ultrasound and MR image registration system, including:

[0096] The image acquisition module acquires the MR images and TRUS images to be registered, as well as the corresponding annotations, to form a multimodal image dataset. The dataset is then divided into training, validation, and test sets based on the patient.

[0097] The preprocessing module treats MR images as floating images. The corresponding label is denoted as TRUS images are considered fixed images. The corresponding label is And for the two images and corresponding markers Preprocessing is required;

[0098] The feature extraction module is used to construct the feature extraction module for the two preprocessed images. and Image-level feature extraction is performed, and the extracted features are denoted as features. and ; and extract features respectively and Features at 5 different resolutions and ;

[0099] The deformation field generation module generates two sets of features at five different resolutions. and The deformation field is generated by fusing and decoding layer by layer from low to high resolution.

[0100] The image alignment module uses a spatial transformation network to apply the generated deformation field to the floating image based on the features at the highest resolution. Interpolation is performed, and then the original size is sampled back to obtain the deformed image. ,image That is, the final and fixed image Spatially aligned MR images.

[0101] Example 3:

[0102] This embodiment proposes an electronic device, including: one or more processors, and a memory, the memory being used to store instructions, which, when executed by the one or more processors, cause the one or more processors to perform the transrectal ultrasound and MR image registration method.

[0103] The electronic device may be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the transrectal ultrasound and MR image registration method as described in the embodiments. It is understood that the electronic device may also include an input / output (I / O) interface and communication components.

[0104] The processor is used to execute all or part of the steps in the transrectal ultrasound and MR image registration method as described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as application-related data.

[0105] The processor can be implemented as an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic components, and is used to execute the transrectal ultrasound and MR image registration method described in the above embodiments.

[0106] Example 4:

[0107] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0108] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the transrectal ultrasound and MR image registration method described in the various embodiments of this application.

[0109] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory), random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disk, optical disk, server, APP (Application) application store, and other media capable of storing program verification codes, on which computer programs are stored. When the computer programs are executed by the processor, they can implement the various steps of the transrectal ultrasound and MR image registration method described above.

[0110] Example 5:

[0111] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the transrectal ultrasound and MR image registration method.

[0112] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0113] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0114] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of this application and its equivalents, then the intent of this disclosure also includes such modifications and variations.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the present invention.

Claims

1. A method of transrectal ultrasound and MR image registration, characterized by, The method comprises the following steps: An image acquisition module is configured to acquire MR images and TRUS images to be registered, and corresponding labels of the images, to form a multi-modal image dataset, and to divide the dataset into a training set, a validation set and a test set according to patients; MR images are regarded as floating images The corresponding label is denoted as TRUS images are regarded as fixed images The corresponding label is denoted as The two images and the corresponding labels are preprocessed; Constructing a feature extraction module, two images after pre-processing and Carrying out image-level feature extraction, the extracted features are respectively denoted as features and ; extract features respectively and features at 5 different resolutions and ; Two groups of features at 5 layers with different resolutions and Fusion decoding is performed layer by layer from low to high resolution and a morphing field is generated; The spatial transformation network generates a deformation field according to the features at the highest resolution, interpolates the floating image according to the deformation field, and then samples back to the original size to obtain a deformed image, which is the final MR image spatially aligned with the fixed image. ​​​​ 2. A transrectal ultrasound and MR image registration method according to claim 1, wherein, The pre-processing of the two images and the corresponding labels comprises the following steps: For two images and are cropped respectively to remove the black background region in the images, and the labels corresponding to the two images and are also cropped respectively with the respective cropping bases; finally, the resolutions of the two images and the labels are adjusted to be consistent by linear interpolation.

3. A transrectal ultrasound and MR image registration method according to claim 2, wherein, The feature extraction module is constructed based on an Unet feature extraction network, and the Unet feature extraction network structure used by the feature extraction module for the two images is the same, and the parameters are shared during training.

4. A transrectal ultrasound and MR image registration method according to claim 3, wherein, The method extracts features respectively by MMC modules and Features under 5 layers of different resolutions and ; Extracting features using MMC modules Features at 5 different resolutions The specific method is: The first layer convolution block of the MMC module adopts convolution + batch normalization + activation layer to the features The features of the first layer are obtained by feature extraction ; The second layer of the MMC module, first uses average pooling to reduce the variance of the features , and then uses convolution + batch normalization + activation layer to extract features from the pooled features to obtain the features of the second layer ; The third layer of the MMC module is obtained by first performing feature dimensional slicing projection to transform the feature , then performing Mamba block operation and splicing it back to the original size according to the order of the slices to obtain the feature ; then performing downsampling operation and convolution operation to expand the dimensions to obtain a feature ; then performing element-wise accumulation on the two groups of features obtained from the second layer and the feature obtained from the third layer to obtain the feature of the third layer ; and ​ Features of the third layer Features of the third layer are obtained by merging the blocks and Features of the fourth layer are obtained by feeding both into a Mamba convolutional block ; Features of the fourth layer The merged block is fed into a block to get features and Then both are fed into a Mamba convolution block to get features ; Extracting features using MMC modules Features at 5 different resolutions The method and extracting features Features at 5 different resolutions The method is the same.

5. A transrectal ultrasound and MR image registration method according to claim 4, wherein, Features of the third layer Features of the third layer are obtained by merging the blocks and Features of the fourth layer are obtained by feeding both into a Mamba convolutional block The specific method is as follows: First, the features of the third layer are obtained by scanning along the 0th row and 0th column, the 1st row and 0th column, the 0th row and 1st column and the 1st row and 1st column respectively, splicing on the dimension channel, and then dimension reduction through a convolution layer. ; then the features are unfolded into features ; the final output of the block merging block is features and ; the features and are sent into the Mamba convolution block, and after the Mamba layer operation, the features are spliced back to the original size according to the order of the slices, and are element-wise accumulated with the features to obtain the features of the fourth layer . ​ 6. A transrectal ultrasound and MR image registration method according to claim 5, wherein, The method adopts an FFA module to perform fusion decoding on two groups of features at five layers of different resolutions and The FFA module adopts a feature fusion block and a convolution decoding block to perform fusion decoding on the two groups of features at the low-to-high resolutions layer by layer and generate a deformation field, specifically: The feature pairs are respectively sent into a feature fusion block for fusion to obtain fused features and respectively, and the fused features and are obtained. on the feature after upsampling and on the feature perform feature stitching to obtain a decoded feature after inputting into a convolutional decoding block , on the decoded feature generate a deformation field ; For deformation field Perform upsampling operation, and deform the feature , to obtain the feature , and send the feature pair into the feature fusion block to obtain the fused feature ; after upsampling the decoding feature , perform feature splicing with the fused feature , and send it into the convolutional decoding block to obtain the decoding feature , perform deformation field prediction on the decoding feature to generate the deformation field ; The same is done for feature pair , and , feature pair , generating morphing field .

7. A transrectal ultrasound and MR image registration method according to claim 6, wherein, In the training process, the Dice loss before and after registration and the smoothing loss of the generated deformation field are used as constraints for the generated deformation field, and the feature extraction module, the MMC module and the FFA module are trained to update the parameters therein; and early stopping strategy is performed on the test set to select optimal parameters of all modules.

8. A transrectal ultrasound and MR image registration system, characterized by The method comprises the following steps: An image acquisition module is configured to acquire MR images and TRUS images to be registered, and corresponding labels of the images, to form a multi-modal image dataset, and to divide the dataset into a training set, a validation set and a test set according to patients; A preprocessing module, the MR image is regarded as a floating image , the corresponding label is , the TRUS image is regarded as a fixed image , the corresponding label is ; and the two images and the corresponding label are preprocessed; The feature extraction module is constructed to perform feature extraction on the two preprocessed images and at the image level, and the extracted features are denoted as features and respectively; and features and at 5 different resolutions are extracted respectively and ; The deformation field generation module generates a deformation field for the two groups of features at 5 different resolutions and performs fusion decoding and generates a deformation field layer by layer from low to high resolution. An image alignment module generates a deformation field using a spatial transformation network in accordance with the features of the highest resolution image pair , interpolates the floating image using the deformation field, and then resamples the interpolated image to the original size to obtain a deformed image , and the deformed image is the final spatially aligned MR image with the fixed image .

9. A computer readable storage medium storing executable instructions for performing the method of transrectal ultrasound and MR image registration according to any one of claims 1 to 7, characterized in that, The instructions, when executed, cause the processor to perform the transrectal ultrasound and MR image registration method.

10. A computer program product for performing the method of transrectal ultrasound and MR image registration according to any one of claims 1-7, characterized in that, The computer program or instructions are executed by the processor to implement the transrectal ultrasound and MR image registration method.

Images