Cross-modal cross-dimension prostate nuclear magnetic ultrasound image registration method

Abstract

A cross-modal cross-dimension prostate nuclear magnetic ultrasound image registration method belongs to the technical field of medical image processing and artificial intelligence auxiliary diagnosis and treatment, and comprises the following steps: firstly, equivalently reconstructing a registration task into two sub-tasks; providing a cross-modal double-branch attention mechanism to realize semantic alignment according to modal differences; and aiming at the dimension difference, designing a lightweight rigid positioning network to carry out affine slice transformation. According to the method, through equivalent problem reconstruction, the modeling difficulty and the calculation complexity are remarkably reduced; a dual-module structure combining elastic and rigid registration is designed, and the registration precision and clinical adaptability are comprehensively improved; a cross-modal double-branch attention mechanism is introduced, and the overall transformation effect is optimized while the precision of the local anatomical structure is kept; a three-stage combined training strategy is adopted, the three-stage combined training strategy comprises contrast learning pre-training, module-level training and whole-network fine tuning, optimization is carried out in combination with weak supervision signals (segmentation labels), a high-quality registration result can be obtained under the condition of no registration labels or a small number of labels, and data dependence and training cost are reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing and artificial intelligence-assisted diagnosis and treatment, and relates to a cross-modal and cross-dimensional prostate magnetic resonance imaging and ultrasound image registration method. Background Art

[0002] Prostate cancer is the second most common malignant tumor in men worldwide. Over the past decade, the incidence of prostate cancer in my country has shown a significant upward trend, with an average annual growth rate of 7%. This rapid increase places greater demands on early diagnosis technologies. Currently, the clinical diagnosis of prostate cancer relies primarily on prostate-specific antigen (PSA) testing and transrectal ultrasound (TRUS)-guided tissue biopsy. However, due to the limited specificity of PSA, false-positive results are common, and tissue biopsy remains the gold standard for prostate cancer diagnosis.

[0003] In clinical practice, transrectal ultrasound (TRUS)-guided prostate biopsy is widely adopted due to its advantages, including real-time imaging, ease of use, and low cost. However, the limited resolution of TRUS images makes it difficult to accurately identify and locate malignant tumor areas, resulting in a false-negative rate of up to 30% with the traditional six-needle biopsy protocol. In contrast, multi-parametric magnetic resonance imaging (mpMRI) can clearly visualize prostate tissue structure and suspicious lesions, significantly improving tumor localization accuracy. However, its long imaging time limits its application in clinical intraoperative imaging.

[0004] To improve prostate cancer detection rates and overcome the limitations of a single imaging modality, Magnetic Resonance / Transrectal Ultrasound Fusion-Guided Biopsy (MR / TRUS) has emerged. This technology aligns preoperative 3D MR images with real-time 2D TRUS images acquired during surgery, providing physicians with more precise puncture positioning.

[0005] However, due to the significant differences between MR imaging (such as T2-weighted images, ADC images, and DCE sequences) and TRUS imaging (such as B-ultrasound and elastography) in imaging principles (proton density and acoustic impedance), spatial resolution (for example, MR is 0.5×0.5×2 mm³, TRUS is 0.2×0.2 mm²), and tissue contrast, achieving cross-dimensional, non-rigid, precise registration between these two modalities remains a significant challenge.

[0006] Currently, prostate MR / TRUS registration methods are mainly divided into two categories: traditional registration methods and deep learning-based registration methods. In traditional methods, external tracking devices (such as electromagnetic sensors) are often relied on to establish the spatial relationship between images of different dimensions, such as (Huang, Q. et al. (2015) 'Bezier Interpolation for 3-D Freehand Ultrasound', IEEE transactions on human-machine systems, 45(3), pp. 385–392.). However, such solutions have complex system structures and high computational costs, making them difficult to meet clinical real-time navigation requirements. Some non-tracking algorithms assume linear probe motion and match 2D images by 3D image slices, but their accuracy is limited by slice gap errors and axial motion assumptions.

[0007] Learning-based registration methods (such as convolutional neural networks, adversarial networks, and visual transformers) have shown promise in improving registration accuracy. For example, (A. Bhardwaj, J.-S. Park, S. Mukhopadhyay, S. Sharda, Y.Son, B. Ajani, and SR Kudavelly, “Rigid and deformable corrections in real-time using deep learning for prostate fusion biopsy,” in Medical Imaging: Image-Guided Procedures, 2020) proposed a deep learning method that combines rigid and non-rigid corrections to support real-time registration. However, these methods still have several limitations: supervised deep learning relies heavily on large amounts of annotated data; unsupervised deep learning places high demands on network performance; and visual transformers are computationally complex, making them difficult to meet the requirements of real-time intraoperative navigation.

[0008] In terms of cross-modal registration, traditional methods rely on biomechanical or statistical deformation models, requiring rigid registration and accurate prostate segmentation before registration, which limits their clinical application. In recent years, some studies have attempted to use weakly supervised / unsupervised deep learning methods to predict transformation matrices or deformation fields. However, due to the lack of real registration deformation data as supervision, it is still difficult to achieve stable and reliable results in practical applications. Some work has attempted to combine biomechanical properties with deep learning methods to improve registration accuracy, but end-to-end real-time cross-modal and cross-dimensional registration has not yet been achieved.

[0009] In summary, current MR / TRUS registration technology still faces the following core problems: (1) There are large imaging differences between cross-modal images, and traditional methods are difficult to achieve high-precision alignment; (2) Deep learning methods are highly dependent on high-quality annotated data or have a heavy computational burden, making them difficult to apply to intraoperative navigation scenarios; (3) There is still a lack of real-time, end-to-end non-rigid registration methods for 3D MR and 2D TRUS. Summary of the Invention

[0010] In response to the problems existing in the existing technology, the present invention aims to solve the problems of modality and dimensionality differences in the registration process between preoperative 3D MR images and intraoperative 2D transrectal TRUS images, and provide an end-to-end, cross-modality and cross-dimensional joint registration method to improve the positioning accuracy, positive detection rate and clinical operation efficiency of prostate puncture biopsy.

[0011] In order to achieve the above object, the present invention adopts the following technical solutions:

[0012] A cross-modal and cross-dimensional prostate magnetic resonance imaging (MRI) registration method comprises the following steps:

[0013] Step S1: Problem Equivalent Reconstruction. In view of the great technical challenges in directly achieving cross-modal and cross-dimensional registration between preoperative 3D MR images and intraoperative 2D transrectal TRUS images, the present invention proposes a two-stage registration strategy with intermediate modal connection. Specifically, first, the 2D TRUS sequence acquired before surgery is Perform three-dimensional reconstruction to obtain 3D TRUS images , and 3D TRUS images As an intermediate modality, it is used to connect the subsequent two-stage registration process. Based on the above strategy, the original complex cross-modal and cross-dimensional registration problem is reconstructed into the following two subtasks:

[0014] Subtask 1: Same-dimensional cross-modality elastic registration between 3D MR and reconstructed 3D TRUS; 3D TRUS is denoted as , three-dimensional TRUS is recorded as .

[0015] Subtask 2: Homomodal cross-dimensional rigid registration between 3D TRUS and 2D TRUS acquired during surgery; 2D TRUS is denoted as .

[0016] By introducing 3D TRUS volumetric images As an intermediate modality, it effectively realizes the staged decoupling of the registration problem between 3D MR and 2D TRUS images, thereby reducing the computational complexity and implementation difficulty of the overall registration task and improving the accuracy and robustness of the registration.

[0017] Step S2: Feature extraction of images of different modalities. In order to achieve effective registration between 3D MR and 3D TRUS, a feature extraction mechanism that can handle the information differences between the modalities needs to be designed for subsequent spatial correspondence construction and deformation field prediction.

[0018] Step S21: Construct a modality-independent feature extraction sub-network. As input, a 3D convolutional neural network 1 is fed into the The image is used as input and fed into a 3D convolutional neural network 2. The 3D convolutional neural network 1 and the 3D convolutional neural network 2 have the same structure but do not share weights. This is to achieve independent feature extraction between different modalities. Each feature extraction subnetwork (3D convolutional neural network 1, 3D convolutional neural network 2) contains multiple layers of 3D convolution, batch normalization, and activation function (ReLU) operations. It uses a multi-level step-by-step downsampling strategy to extract features layer by layer. and The local structural information of the image at different spatial resolutions. Through the above processing, we can get the local structural information of the image at different spatial resolutions. and The corresponding feature tensor and .

[0019] Step S22: Introduce a cross-modal dual-branch attention mechanism.

[0020] For the sake of and The semantic correspondence between features is fully modeled. The present invention adopts a cross-modal dual-branch attention mechanism to the feature tensor extracted in step S21. and Perform interaction modeling.

[0021] set up ,set up , where L, W, H represent the length, width and height of the feature map respectively, and C represents the number of channels.

[0022] The feature tensor 、 Expanding to a two-dimensional representation, we get: query vector ;Key vector ; Value vector .in ; is the dimension of the key / query vector; is the dimension of the value vector. 、 、 is a learnable linear mapping module.

[0023] Using MR features as the query and TRUS features as the key-value pairs, the attention weight matrix is ​​calculated as follows:

[0024]

[0025] in, Indicates the The query position (MR feature) and the Attention weights between key positions (TRUS features); represents the exponential function, which is used to convert similarity into non-negative weight; Indicates the first The original feature vector extracted from the voxel position; Represents the learnable functions that linearly transform the query vector (from MR) and the key vector (from TRUS), and the output dimension is ; represents the original feature vector extracted from the j-th voxel position in the TRUS image; Represents a dimension value.

[0026] Then calculate the attention fusion output:

[0027]

[0028] in, represents the position of the jth key in TRUS; Indicates the upper limit of operation; Represents the fusion output of the cross-modal dual-branch attention mechanism; Represents the TRUS feature tensor through the linear mapping module After processing, A vector of values ​​at each position.

[0029] A reverse attention operation is then performed, using TRUS as the query and MR as the key, enabling bidirectional cross-modal information exchange and enhancing the stability and expressiveness of structural alignment. This mechanism enables modeling and explicit alignment of the correspondence between different modal structures at a semantic level.

[0030] Step S23: Use cross-scale skip connections to enhance feature semantic expression.

[0031] The features output by each layer of the encoder (the 3D convolutional neural network in step S21) are passed to the decoder network through jump connections; the spatial resolution is gradually restored through transposed convolution, and the 3D deformation field is finally output. .

[0032] Apply it to Get the elastically registered image:

[0033]

[0034] in, represents the image domain, Indicates the spatial position of image pixels; represents the pixel value at spatial position x of the three-dimensional preoperative MR image obtained after registration transformation; Indicates a spatial transformation of the spatial position x; Represents the original 3D preoperative MR image after spatial transformation After the transformation The pixel value at .

[0035] Step S3: Cross-dimensional image rigid registration. To achieve rigid registration between 3D TRUS and 2D TRUS, a lightweight network structure is designed to meet the real-time reasoning requirements during surgery.

[0036] Step S31: 2D TRUS image feature extraction and channel expansion. As input, import the multi-layer two-dimensional convolutional neural network to obtain the feature tensor Its dimension is ; In order to bridge the dimensionality difference between two-dimensional and three-dimensional images, the feature channel expansion method is adopted. Specifically, The channel is expanded along the depth direction by adding a single dimension in the depth dimension ( ) Construct a pseudo 3D volume feature tensor , laying the foundation for subsequent registration with three-dimensional features.

[0037] Step S32: 3D TRUS volume feature extraction. Re-input 3D convolutional neural network to extract feature tensors Its dimension is , and the one obtained in step S31 Splicing along the channel dimension to form a joint feature tensor .

[0038] Step S33: Rigid parameter regression and affine slice resampling. Use the Localization Net based on the ResNeXt structure to regress the six-degree-of-freedom rigid transformation parameters of translation and rotation (For details, see S. Xie, RBGirshick, P. Doll´ar, Z. Tu, and K. He, “Aggregated residualtransformations for deep neural networks,” 2017 IEEE Conference on Computer Vision and PatternRecognition (CVPR), pp. 5987–5995, 2016.):

[0039]

[0040] in, represents the translation amount, Represents the rotation angle around three axes.

[0041] according to Building the Reflection Mesh , applied to The obtained slice image :

[0042]

[0043] in, Indicates original image; represents a composite operation of a representation function (here it represents a registration operation); By adjusting the parameters The transformation can be changed; Represents a geometric transformation (here, a deformation field).

[0044] And slice image Perform bilinear interpolation to obtain the registered predicted image.

[0045] Step S4: Position correction and improving model robustness.

[0046] Step S41: Image similarity evaluation mechanism. Use the Pearson correlation coefficient (formula below) to register the slice images after step S33. With the original intraoperative Image similarity measurement.

[0047]

[0048] in, is the covariance, is the standard deviation; and Represent the images after registration and the original 2D image The standard deviation of Two images ( ), the value range is [-1,1], and the closer the value is to 1, the higher the similarity.

[0049] If the calculated similarity score is lower than the preset threshold (usually set to a value range of [0, 1]), then trigger the subsequent iterative optimization process (step S42) to further improve the registration accuracy.

[0050] Step S42: Iterative optimization based on affine parameters.

[0051] If the similarity obtained in step S41 Less than the preset threshold , start the iterative optimization mechanism: after the transformation generated in step S33 As a new input, replace the original input image in step S32 , re-extract features and construct joint features and construct joint feature tensors ; The rigid initial parameters estimated in step S33 As initial value And re-predict the transformation until the similarity improves or the termination condition is met:

[0052]

[0053] in, Represents a weighted fusion strategy, where the weights are dynamically determined by the similarity improvement gradient.

[0054] Finally, stable rigidity parameters are obtained The 3D image after the rigid parameter adjustment transformation is used as the new input of step S41 to re-predict the rigid parameters; if the new slice Bianyuan output If the similarity is higher, the current rigid parameters are merged With new parameters ;

[0055] Repeat the above operation until the termination condition is met to ensure that the similarity between the registered slice image and the original two-dimensional TRUS image is maximized.

[0056] Step S5: A three-stage joint training strategy is used to design a progressive training process to improve the overall network performance and stability.

[0057] Step S51: Contrastive learning pre-training (stage 1). and Randomly select For matching points, construct the cosine similarity matrix , define contrastive loss:

[0058]

[0059]

[0060]

[0061] in, Represents the loss in the direction of the i-th row, represents the loss in the direction of column i, represents the overall contrast loss; Representative The MR feature point and the corresponding The cosine similarity matrix between TRUS feature points (i.e., positive sample pairs); Indicates the representative The MR feature point and the corresponding The cosine similarity matrix between TRUS feature points; .

[0062] By maximizing the similarity of features at the same position (i.e., the positive sample pair ), minimize the similarity of other positions and optimize cross-modal feature extraction.

[0063] Step S52: Module independent training (Phase 2). Use the cross-modal dual-branch attention mechanism registration module in step S22 for unsupervised training (NMI loss + deformation field smoothness constraint). The cross-modal module loss is:

[0064]

[0065] in, , is the weight factor, represents the deformation field gradient, Represents the squared L2 norm (Euclidean norm) which is often used to ensure spatial transformation Smooth and physically correct, preventing unrealistic distortions; represents the normalized mutual information loss between two images; After spatial transformation Three-dimensional preoperative MR images after surgery; Represents the reconstructed three-dimensional TRUS image;.

[0066] The 3D-2D rigid registration module in step S3 is trained under supervision (regression MSE loss, using synthetic slices and true parameters).

[0067] Cross-dimensional module loss (supervised):

[0068]

[0069] Among them, N is the number of sample pairs in each round of training, Used to average the errors of all sample pairs; represents the true transformation label (known dataset); represents the prediction transformation parameters; represents the squared L2 norm.

[0070] Step S53: End-to-end fine-tuning (stage 3). Use 3D MR and 3D TRUS segmentation labels (known datasets) for weakly supervised training;

[0071] Based on the Dice similarity coefficient (in Formula 13 ) measures the overlap accuracy between the predicted results and the true two-dimensional labels;

[0072]

[0073] in, Represents set A, which here refers to the prediction results; represents the set B, which refers to the real two-dimensional labels; Represents the size of the intersection of set A and set B (that is, the number of elements they contain in common or the number of voxels / pixels in the overlapping area).

[0074] Only the decoder in step S23 and the Localization Net weight in step S33 are optimized to improve the final overall performance.

[0075] Step S6: Integrate registration image generation with clinical application, and use the trained cross-modal and cross-dimensional prostate magnetic resonance ultrasound image registration method in the prostate puncture clinical system; achieve real-time registration of prostate tumors and tissue structures in 3D MR images to intraoperative two-dimensional TRUS images; assist doctors in accurately locating the puncture path, improving the puncture positive rate and safety.

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] (1) Most existing image registration methods can only handle registration problems of the same modality or the same dimension. The present invention proposes a cross-modal and cross-dimensional prostate magnetic resonance imaging and ultrasound image registration method for the first time. By reconstructing the equivalent problem, the complex registration task of three-dimensional MR and two-dimensional TRUS images is split into two sub-modules that can be independently optimized, which significantly reduces the modeling difficulty and computational complexity. (2) Most traditional registration methods rely on a single rigid or elastic registration model and lack real-time support. The present invention designs a dual-module structure that combines elastic and rigid registration, in which elastic registration is used between three-dimensional MR and three-dimensional TRUS to capture tissue deformation, and lightweight rigid registration is used between three-dimensional TRUS and two-dimensional TRUS to meet real-time intraoperative guidance. (3) To improve the cross-modal feature matching capability, the present invention introduces a cross-modal dual-branch attention mechanism, establishes alignment channels for queries in different modalities, effectively constructs cross-domain semantic associations, enhances spatial correspondence accuracy, and optimizes the overall transformation effect while maintaining the accuracy of local anatomical structures; (4) Compared with the supervised learning method that requires large-scale manual registration labels, the present invention adopts a three-stage joint training strategy, including contrastive learning pre-training, module-level training and full-network fine-tuning, and combines it with weak supervisory signals (segmentation labels) for optimization. High-quality registration results can be obtained even without registration labels or with a small number of labels, which greatly reduces data dependence and training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 Schematic diagram for decoupling cross-modal and cross-dimensional tasks;

[0079] Figure 2 The internal structure of the dual-branch attention network across channels;

[0080] Figure 3 A neural network for cross-modal single-dimension registration;

[0081] Figure 4 Neural networks for cross-dimensional unimodal registration;

[0082] Figure 5 The following is a comparison diagram of the registration diagram and the results visualization.

[0083] Figure 6 Flowchart of the overall method for cross-modality and cross-dimensional prostate MRI-Ultrasound registration. DETAILED DESCRIPTION

[0084] The technical solution of the present invention will be further described below in conjunction with specific embodiments and drawings.

[0085] A cross-modal and cross-dimensional prostate MRI registration method was developed. The specific implementation method was experimentally verified on a public dataset (µ-RegPro), and trained and inferred on an NVIDIA RTX 3090 GPU. The steps are as follows:

[0086] Step S1: Problem Equivalence Reconstruction

[0087] The complex cross-modality and cross-dimensional registration problem between 3D MR and 2D TRUS is restructured into two subtasks:

[0088] Subtask 1: 3D MR (denoted as ) and 3D TRUS (denoted as ) between the same dimension cross-modal elastic registration;

[0089] Subtask 2: 3D TRUS ( ) and two-dimensional TRUS (denoted as ) between homomodal cross-dimensional rigid registration.

[0090] In the specific implementation, firstly the two-dimensional TRUS image sequence Perform 3D volume reconstruction and generate 3D intermediate modality images using traditional image registration and voxel interpolation methods .

[0091] Figure 1 Schematic diagram of cross-modal and cross-dimensional task decoupling, showing how to connect MR and 2D TRUS registration tasks through the 3D TRUS intermediate modality.

[0092] Step S2: Feature extraction of different modality images. In order to achieve effective registration between 3D MR and 3D TRUS, a feature extraction mechanism that can handle modality differences is designed:

[0093] Step S21: Construct a modality-independent feature extraction sub-network. and Input the 3D convolution encoder respectively to extract the feature tensor and The network consists of 4 convolutional layers, each of which contains 3D convolution, BN and ReLU activation, and obtains multi-scale features through downsampling.

[0094] Step S22: Introduce the cross-modal dual-branch attention mechanism. and Mapped into high-dimensional Query, Key and Value vectors through the linear transformation module, and calculate the attention matrix and fusion output . Perform bidirectional matching and build semantic alignment relationships.

[0095] Figure 2 This is a diagram of the internal structure of the cross-channel dual-branch attention network, illustrating how the attention mechanism achieves explicit alignment of features from different modalities.

[0096] Step S23: Cross-scale jump connection and decoding. The features after attention fusion are jump-connected with the corresponding scale features of the encoder, and input into the decoder to upsample layer by layer to restore the resolution and output the 3D deformation field. .

[0097] Step S24: Apply deformation field Elastic registration of MR images. Using ϕ Get the registered image .

[0098] Figure 3 This is a diagram of the neural network structure for cross-modal single-dimensional registration, showing the visualization effect of deformation field generation and application to MR images, indicating that the boundary overlap is significantly improved after registration.

[0099] Step S3: Cross-dimensional image rigid registration

[0100] Step S31: 2D TRUS image feature extraction and channel expansion. Extraction using 2D convolutional network After the feature, copy its channel dimension to construct a pseudo 3D tensor .

[0101] Step S32: 3D TRUS image feature extraction. Input 3D convolutional network to extract spatial features, and Splicing to form joint features .

[0102] Step S33: Rigid parameter regression and slice generation. Input Localization Net regression six degrees of freedom parameters , using an affine sampler to generate slice images , and perform rigid registration compared with the original image.

[0103] Figure 4 This is a diagram of the neural network structure for cross-dimensional single-modality registration, showing the rigid transformation prediction process and slice generation method.

[0104] Step S4: Position correction and robustness improvement.

[0105] Step S41: Image similarity evaluation. Calculation and Pearson correlation coefficient If it is below the threshold , enter the subsequent optimization.

[0106] Step S42: Iteratively optimize the affine parameters. Adjust the transformation parameters until the similarity no longer improves. In practice, the average number of iterations is 1.6, and the processing time is less than 0.05 seconds.

[0107] Step S5: Three-stage joint training strategy

[0108] Step S51: Contrastive learning pre-training. Use K=64 pairs of cross-modal point pairs to construct a similarity matrix , training cosine contrastive loss To enhance and Semantic consistency.

[0109] Step S52: Module-level training. Unsupervised loss is used for the elastic module: Smoothing, using MSE supervision for rigid modules.

[0110] Step S53: End-to-end fine-tuning. Dice supervision is performed using segmentation labels. In the validation set, the Dice coefficient is improved from 0.78 for the unregistered dataset to 0.849 for the proposed method.

[0111] Figure 5 The comparison of prostate region segmentation before and after MR / US image registration is shown.

[0112] Step S6: System deployment and application integration

[0113] This method is integrated into the puncture guidance system to achieve to Real-time registration with inference latency less than 0.26 seconds per image satisfies intraoperative navigation requirements.

[0114] Figure 6 This is a flowchart of the overall method for cross-modal and cross-dimensional prostate MRI-ultrasound registration, showing the execution process and data interaction between the various modules of the system.

[0115] The above-described embodiments merely express the implementation methods of the present invention, but should not be understood as limiting the scope of the present invention. It should be pointed out that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. A cross-modal and cross-dimensional prostate magnetic resonance imaging and ultrasound image registration method, characterized in that: The following steps are involved: Step S1: Use a two-stage registration strategy with intermediate modal connection to perform equivalent reconstruction of the problem; The registration task is equivalently reconstructed into two subtasks: cross-modal elastic registration between 3D MR and 3D TRUS, and cross-dimensional rigid registration between 3D TRUS and 2D TRUS. Step S2: Feature extraction of different modal images. To achieve registration between 3D MR and 3D TRUS, a cross-modal dual-branch attention mechanism is designed. Step S3: Cross-dimensional image rigid registration. To achieve rigid registration between 3D TRUS and 2D TRUS, a 3D-2D rigid registration module is designed to meet the real-time reasoning requirements. Step S4: Position correction and improving model robustness; Step S41: image similarity evaluation mechanism; Step S42: iterative optimization based on affine parameters; Step S5: Three-stage joint training strategy, designing a progressive training process; Step S51: Contrastive learning pre-training, as stage 1: optimizing cross-modal feature extraction by maximizing the similarity of features at the same position and minimizing the similarity of features at other positions; Step S52: Module independent training, as stage 2: Use the cross-modal dual-branch attention mechanism registration module in step S2 for unsupervised training, and the 3D-2D rigid registration module in step S3 for supervised training; Step S53: end-to-end fine-tuning, as stage three: weakly supervised training using 3D MR and 3D TRUS segmentation labels for optimization; Step S6: Generate and integrate the registration image with the application.

2. The cross-modal and cross-dimensional prostate magnetic resonance imaging registration method according to claim 1, characterized in that: In step S1, the acquired 2D TRUS sequence Perform three-dimensional reconstruction to obtain 3D TRUS images , and 3D TRUS images As an intermediate modality, it is used to connect the subsequent two-stage registration process, and reconstructs two subtasks, including: Subtask 1: Same-dimensional cross-modality elastic registration between 3D MR and reconstructed 3D TRUS; 3D TRUS is denoted as , three-dimensional TRUS is recorded as ; Subtask 2: Homomodal cross-dimensional rigid registration between 3D TRUS and 2D TRUS acquired during surgery; 2D TRUS is denoted as .

3. The cross-modal and cross-dimensional prostate magnetic resonance ultrasound image registration method according to claim 2, characterized in that: The step S2 is specifically as follows: Step S21: constructing a modality-independent feature extraction subnetwork; Will As input, a 3D convolutional neural network is fed into it; The image is used as input and then fed into another 3D convolutional neural network. The two 3D convolutional neural networks have the same structure but do not share weights, thus achieving independent feature extraction between different modalities. and The corresponding feature tensor and ; Step S22: Use the cross-modal dual-branch attention mechanism to extract the feature tensor obtained in step S21 and Conduct interaction modeling; Step S23: using cross-scale skip connections to enhance feature semantic expression; The features output by each layer of the encoder of the 3D convolutional neural network in step S21 are transferred to the decoder network through jump connections; the spatial resolution is gradually restored through transposed convolution, and the 3D deformation field is finally output. ; Apply it to Get the elastically registered image: ,in, represents the image domain, Indicates the spatial position of image pixels; represents the pixel value at spatial position x of the three-dimensional preoperative MR image obtained after registration transformation; Indicates a spatial transformation of the spatial position x; Represents the original 3D preoperative MR image after spatial transformation After the transformation The pixel value at .

4. The cross-modal and cross-dimensional prostate magnetic resonance imaging registration method according to claim 3, characterized in that: In the step S2: In step S21, each three-dimensional convolutional neural network includes multiple layers of 3D convolution, batch normalization and activation function operations, and adopts a multi-level step-by-step downsampling strategy to extract the and Local structural information of images at different spatial resolutions; The step S22 is specifically as follows: set up ,set up , where L, W, H represent the length, width and height of the feature map respectively, and C represents the number of channels; The feature tensor 、 Expanding to a two-dimensional representation, we get: query vector ;Key vector ; Value vector ;in ; is the dimension of the key / query vector; is the dimension of the value vector; 、 、 It is a learnable linear mapping module; Using MR features as the query and TRUS features as the key-value pairs, the attention weight matrix is ​​calculated as follows: ,in, Indicates the The query position (MR feature) and the Attention weights between key positions (TRUS features); represents the exponential function, which is used to convert similarity into non-negative weight; Indicates the first The original feature vector extracted from the voxel position; Represents the learnable functions that linearly transform the query vector and key vector, respectively, and the output dimension is ; represents the original feature vector extracted from the j-th voxel position in the TRUS image; Represents dimension value; Then calculate the attention fusion output: ,in, represents the position of the jth key in TRUS; Indicates the upper limit of operation; Represents the fusion output of the cross-modal dual-branch attention mechanism; Represents the TRUS feature tensor through the linear mapping module After processing, A vector of values ​​at each position; Then, a reverse attention operation is performed to achieve bidirectional cross-modal information interaction and enhance the stability and expressiveness of structural alignment.

5. The cross-modal and cross-dimensional prostate magnetic resonance ultrasound image registration method according to claim 3, characterized in that: The step S3 is specifically as follows: Step S31: 2D TRUS image feature extraction and channel expansion; The image As input, import the multi-layer two-dimensional convolutional neural network to obtain the feature tensor Its dimension is ; And the feature channel expansion method is used to obtain the pseudo 3D volume feature tensor ; Step S32: 3D TRUS volume feature extraction; Re-input 3D convolutional neural network to extract feature tensors Its dimension is , and the value obtained in step S31 Splicing along the channel dimension to form a joint feature tensor ; Step S33: Rigid parameter regression and affine slice resampling; and slice image Perform bilinear interpolation to obtain the registered predicted image.

6. The cross-modal and cross-dimensional prostate magnetic resonance ultrasound image registration method according to claim 3, characterized in that: In the step S3: The characteristic channel expansion method in step S31 is: Will The channel is expanded along the depth direction by adding a single dimension in the depth dimension ( ) Construct a pseudo 3D volume feature tensor ; The step S33 is specifically as follows: Use the Localization Net based on the ResNeXt structure to regress the six-degree-of-freedom rigid transformation parameters of translation and rotation : ,in, represents the translation amount, Represents the rotation angle around three axes; according to Building the Reflection Mesh , applied to The obtained slice image : ,in, Indicates original image; Represents a composite operation of a function; By adjusting the parameters The transformation can be changed; Represents a geometric transformation.

7. The cross-modal and cross-dimensional prostate magnetic resonance ultrasound image registration method according to claim 6, characterized in that: In the step S4: Step S41 is specifically as follows: The slice images after step S33 registration are registered using the Pearson correlation coefficient. With the original intraoperative Image similarity measurement; ,in, is the covariance, is the standard deviation; and Represent the images after registration and the original 2D image The standard deviation of Two images ( ), the similarity score between them ranges from [-1,1], and the closer the value is to 1, the higher the similarity; If the calculated similarity score is lower than the preset threshold , then the subsequent step S42 is triggered to further improve the registration accuracy; the preset threshold The value range of is [0,1]; Step S42 is specifically as follows: If the similarity obtained in step S41 Less than the preset threshold , start the iterative optimization mechanism: after the transformation generated in step S33 As a new input, replace the original input image in step S32 , re-extract features and construct joint features and construct joint feature tensors ; The rigid initial parameters estimated in step S33 As initial value And re-predict the transformation until the similarity improves or the termination condition is met: ,in, Represents a weighted fusion strategy, where the weight is dynamically determined by the similarity improvement gradient; Finally, stable rigidity parameters are obtained ; Use the rigid parameters to adjust the transformed 3D image as the new input of step S41 to re-predict the rigid parameters; if the new slice Bianyuan output If the similarity is higher, the current rigid parameters are merged With new parameters ; Repeat the above steps until the termination condition is met.

8. The cross-modal and cross-dimensional prostate magnetic resonance ultrasound image registration method according to claim 7, characterized in that: In the step S5: Step S51 is specifically as follows: exist and Randomly select For matching points, construct the cosine similarity matrix , define contrastive loss: ,in, Represents the loss in the direction of the i-th row, represents the loss in the direction of column i, represents the overall contrast loss; Representative The MR feature point and the corresponding The cosine similarity matrix between TRUS feature points, that is, the positive sample pair; Indicates the representative The MR feature point and the corresponding The cosine similarity matrix between TRUS feature points; ; By maximizing the similarity of features at the same position, that is, the positive sample pair , minimize the similarity of other positions and optimize cross-modal feature extraction; Step S52 is specifically as follows: Use the cross-modal dual-branch attention mechanism registration module in step S22 for unsupervised training, and the cross-modal module loss is: ,in, , is the weight factor, represents the deformation field gradient, Represents the squared L2 norm (Euclidean norm) which is often used to ensure spatial transformation Smooth and physically correct, preventing unrealistic distortions; represents the normalized mutual information loss between two images; After spatial transformation Three-dimensional preoperative MR images after surgery; represents the reconstructed three-dimensional TRUS image; Supervised training of the 3D-2D rigid registration module in step S3; cross-dimensional module loss: , where N is the number of sample pairs in each round of training, Used to average the errors of all sample pairs; represents the true transformation label; represents the prediction transformation parameters; represents the squared L2 norm; Step S53 is specifically as follows: The overlap accuracy between the predicted result and the true two-dimensional label is measured based on the Dice similarity coefficient; ,in, Represents set A, which here refers to the prediction results; represents the set B, which refers to the real two-dimensional labels; Indicates the size of the intersection of set A and set B; Only the decoder in step S23 and the Localization Net weight in step S33 are optimized to improve the final overall performance.

Images