A structure-preserving weakly supervised multimodal prostate image registration method

By introducing multi-level self-attention convolutional neural network in weakly supervised multimodal prostate image registration, the dependence between prostate feature maps is established, and the problem of maintaining the fineness of the internal organ structure of the prostate image is solved, and high precision is achieved image registration and real-time image guidance.

CN114897948BActive Publication Date: 2024-11-05NANJING TECH UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210553346.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-20
Publication Date
2024-11-05
Estimated Expiration
2042-05-20

AI Technical Summary

Technical Problem

In weakly supervised multimodal prostate image registration, the prior art is difficult to maintain the fineness of internal organ structures in prostate images after registration without damaging the accuracy of prostate transformation, especially in the unsupervised registration framework Distortion problems caused by the generated unconstrained displacement field.

Method used

Design a weakly supervised multimodal prostate image registration method with structure-maintained structure, adopt multi-level self-attention convolution neural network, and establish prostate feature maps at different scales by constructing sample sets and training prostate image transformation models using self-attention mechanism. The long-term dependence relationship between them is reflected in the displacement field, thereby achieving more accurate information transmission of spatial displacement.

Benefits of technology

Without damaging the accuracy of prostate transformation, the fineness of the internal organ structure of the prostate image after registration is significantly improved, the requirements of real-time image guidance surgery are met, and the registration effect of prostate MRI images and TRUS images is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114897948B_ABST
    Figure CN114897948B_ABST
Patent Text Reader

Abstract

The present invention relates to a structure-preserving weakly supervised multimodal prostate image registration method. Through the designed steps i to ii, a prostate image transformation model is obtained; then steps A to B are executed, and the prostate image transformation model is applied to realize the registration between a three-dimensional prostate magnetic resonance (MRI) image and a prostate transrectal ultrasound (TRUS) image of the prostate organ region in the known images corresponding to the target object; the scheme design introduces a multi-level self-attention mechanism in the weakly supervised multimodal prostate registration network to establish a long-range feature dependency relationship for prostate feature maps of different scales, so as to ensure that the prostate feature maps of different scales can carry more accurate spatial displacement information and reflect it on the displacement field, so as to realize that the internal organ structure in the registered prostate image is more finely maintained without damaging the prostate transformation accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a structure-preserving weakly supervised multimodal prostate image registration method, and belongs to the technical field of image processing. Background Art

[0002] Transrectal ultrasound (TRUS) is a standard imaging modality for cancer diagnosis and image-guided interventions, such as biopsy, brachytherapy, and cryotherapy. Prostate brachytherapy is often used to treat prostate cancer using high dose rates and whole prostate irradiation. However, it is difficult to accurately delineate tumor and lesion areas on TRUS images due to the lack of soft tissue contrast. Magnetic resonance imaging (MRI) provides detailed anatomical and functional information that is not available on TRUS images, and due to its superior soft tissue contrast, MRI has higher sensitivity and specificity than TRUS for the detection and characterization of prostate cancer. However, MRI is expensive, not as widely available as TRUS, and most importantly, it is impractical to provide real-time image guidance for prostate interventions. Therefore, it is necessary to register TRUS and MRI images to provide real-time image guidance with improved soft tissue contrast.

[0003] The emergence of deep learning has greatly influenced the hotspots of image registration research. Initially, deep learning technology was mainly used to enhance iterative, grayscale-based medical image registration, but the registration speed was slow. The demand for fast registration algorithms promoted the development of end-to-end supervised registration methods. Since it is difficult to obtain the true displacement field through supervised learning, many scholars have developed an unsupervised registration framework for end-to-end. At present, the important thing in the unsupervised registration framework is the quantification of image similarity. For the registration of transrectal ultrasound images and magnetic resonance images of the prostate, due to the huge differences in image grayscale and field of view between the two, it is very difficult to rely solely on the quantification of image similarity in the unsupervised registration framework.

[0004] The weakly supervised framework uses the segmented overlap of the prostate anatomical structures segmented from transrectal ultrasound images and magnetic resonance images to design the loss function, that is, the label similarity measure is used as the objective function, and the weights of the network are updated in reverse iteration. The anatomical structure labels are used to highlight the same organs and their boundaries, pathological areas and other anatomical structures, and morphological or physiological features that appear in the two images in the paired prostate images, and are trained as weak labels to predict the low-level voxel correspondence. However, the unconstrained displacement field generated in the weakly supervised framework will cause inappropriate distortion of the moving image, resulting in an unrealistic organ appearance. Although some regularization terms (such as bending energy, L1 or L2 norm of displacement gradient) are added to the weakly supervised framework, the displacement field is restricted to a certain extent, and the unrealistic organ appearance of the distorted moving image is improved, but it leads to a decrease in registration accuracy. Therefore, how to effectively balance the appearance preservation and accuracy of the internal organs of the prostate image after registration in the weakly supervised multimodal prostate image registration has become an urgent problem to be solved in the research. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a structure-preserving weakly supervised multimodal prostate image registration method, which adopts a new application framework to ensure that prostate feature maps of different scales can carry more accurate spatial displacement information and reflect it on the displacement field, thereby achieving a more precise preservation of the internal organ structure in the prostate image after registration without compromising the accuracy of prostate transformation.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions: the present invention designs a structure-preserving weakly supervised multimodal prostate image registration method, through steps i to ii, a prostate image transformation model is obtained; then steps A to B are performed, and the prostate image transformation model is applied to achieve registration between the three-dimensional prostate MRI image and the prostate TRUS image of the prostate organ area in the known images corresponding to the target object;

[0007] Step i. Based on a preset number of three-dimensional prostate MRI sample images, and the binary three-dimensional prostate MRI sample images corresponding to the prostate organ region in the distinguished image, the prostate TRUS sample images belonging to the same object as the three-dimensional prostate MRI sample images, and the binary prostate TRUS sample images corresponding to the prostate organ region in the distinguished image, in combination with the displacement fields of the method for describing the transformation of the prostate organ region in each three-dimensional prostate MRI sample image to the contour shape of the prostate organ region in the prostate TRUS sample image of the same object, a sample set is constructed, and then step ii is entered;

[0008] Step ii. Based on the sample set, the three-dimensional prostate MRI sample image and the prostate organ region in the corresponding binary three-dimensional prostate MRI sample image, as well as the prostate TRUS sample image and the prostate organ region in the corresponding binary prostate TRUS sample image belonging to the same object are used as input, and the displacement field describing the method of transforming the prostate organ region in the three-dimensional prostate MRI sample image to the contour shape of the prostate organ region in the prostate TRUS sample image is used as output, and the network to be trained is trained to obtain a prostate image transformation model;

[0009] Step A. applying a prostate image transformation model to the three-dimensional prostate MRI image and the prostate TRUS image corresponding to the target object, obtaining a displacement field describing a method of transforming a prostate organ region in the three-dimensional prostate MRI image to a contour shape of the prostate organ region in the prostate TRUS image, and then proceeding to step B;

[0010] Step B. From the three-dimensional prostate MRI image and according to the method described by the displacement field, a three-dimensional prostate MRI target image is obtained after the prostate organ region in the three-dimensional prostate MRI image is transformed to the contour shape of the prostate organ region in the prostate TRUS sample image, so as to achieve registration between the three-dimensional prostate MRI image and the prostate TRUS image corresponding to the target object.

[0011] The network to be trained in step ii is a multi-level self-attention convolutional neural network, including a first downsampling module, a second downsampling module, a third downsampling module, a first upsampling module, a second upsampling module, a third upsampling module, and four trilinear upsampling modules;

[0012] The first downsampling module includes a self-attention mechanism block, three feature extraction modules, a redundant feature reduction module, and an addition module, wherein the input end of the self-attention mechanism block constitutes the input end of the first downsampling module; the three feature extraction modules are sequentially connected in series, and the input end of the feature extraction module at the first position constitutes the input end of the series structure, and the output end of the feature extraction module at the first position and the output end of the feature extraction module at the third position are connected to the addition module, and the output end of the addition module constitutes the output end of the series structure; the output end of the self-attention mechanism block is connected to the input end of the series structure, the output end of the series structure is connected to the input end of the redundant feature reduction module, and the output end of the redundant feature reduction module constitutes the output end of the first downsampling module;

[0013] The structure of the second downsampling module is the same as that of the third downsampling module. The second downsampling module and the third downsampling module respectively include a self-attention mechanism block, two feature extraction modules, a redundant feature reduction module, and an addition module, wherein the input end of the self-attention mechanism block constitutes the input end of the downsampling module, the two feature extraction modules are sequentially connected in series, the output end of the self-attention mechanism block is connected to the input end of the series structure, and at the same time, the output end of the self-attention mechanism block and the output end of the series structure are connected to the addition module, the output end of the addition module is connected to the input end of the redundant feature reduction module, and the output end of the redundant feature reduction module constitutes the output end of the downsampling module;

[0014] The structures of the first upsampling module, the second upsampling module, and the third upsampling module are the same, and each includes a scaling module, two feature extraction modules, an addition module, and a trilinear addition upsampling module, wherein the input end of the scaling module and the input end of the trilinear addition upsampling module constitute the input end of the upsampling module, the two feature extraction modules are sequentially connected in series, and the input end of the series structure and the output end of the series structure are connected to the addition module, the output end of the addition module constitutes the output end of the series structure, and the output end of the series structure constitutes the output end of the upsampling module, and the output end of the scaling module and the output end of the trilinear addition upsampling module converge with each other to form a convergence end;

[0015] The input end of the first down-sampling module constitutes the input end of the multi-level self-attention neural network, the first down-sampling module, the second down-sampling module, and the third down-sampling module are connected in series in sequence, the output end of the third down-sampling module is connected to the input end of the first up-sampling module, the output end of the self-attention mechanism block in the third down-sampling module and the convergence end in the first up-sampling module are connected to the addition module, and the output end of the addition module is connected to the input end of the series structure in the first up-sampling module; the output end of the first up-sampling module is connected to the input end of the second up-sampling module, the output end of the self-attention mechanism block in the second down-sampling module and the convergence end in the second up-sampling module are connected to the addition module, and the output end of the addition module is connected to the input end of the series structure in the second up-sampling module; the output end of the second up-sampling module and the output end of the feature extraction module at the first position in the series structure in the first down-sampling module are connected to the addition module, the output end of the addition module is connected to the input end of the third up-sampling module, and the convergence end in the third up-sampling module is connected to the input end of its series structure;

[0016] The output end of the third down-sampling module, the output end of the first up-sampling module, the output end of the second up-sampling module, and the output end of the third up-sampling module are respectively connected to the input end of each trilinear up-sampling module in a one-to-one correspondence, and the output ends of each trilinear up-sampling module are connected to an addition module, which constitutes the output end of the multi-level self-attention neural network.

[0017] As a preferred technical solution of the present invention: the self-attention mechanism block includes three 1x1x1 convolutional layers θ connected in parallel, Unary function g, softmax function module, matrix multiplication module, element-by-element summation module, maximum pooling layer, deconvolution layer, residual connection.

[0018] As a preferred technical solution of the present invention: the mathematical expression of the self-attention mechanism block is defined as:

[0019]

[0020] Where i represents the index of the output position in space, time, or spatiotemporal where the response is computed, j represents the index enumerating all possible positions, x represents the input image signal, m represents the downsampled version of x, y represents the output signal of the same size as x, the pairwise function f(·) computes the scalar dependency between i and all j, the unary function g(·) computes the representation of the input signal at position j, and the response is normalized by the factor C(m);

[0021] The self-attention mechanism block calculates the response of the target position in the feature map as the weighted average of the features at all positions in the input feature map, thereby establishing dependencies between long-distance features.

[0022] As a preferred technical solution of the present invention: the feature extraction module includes a convolution layer, a normalization layer, and an activation layer connected in series in sequence; the redundant feature reduction module includes a maximum pooling layer, a convolution layer, a normalization layer, and an activation layer connected in series in sequence; the scale expansion module includes a deconvolution layer, a normalization layer, and an activation layer connected in series in sequence; the trilinear upsampling module includes a convolution layer, a bias layer, and a cubic linear interpolation layer connected in series in sequence.

[0023] As a preferred technical solution of the present invention: according to the multi-scale dice function that describes the overlap between the binary prostate organ target image and the prostate organ area in the corresponding binary prostate TRUS image, combined with the bending energy loss function that penalizes the displacement field, the overall loss function of the multi-level self-attention neural network is constructed; and in the training process of the multi-level self-attention neural network in the step ii, the Adam optimizer and the standard random K-minibatch gradient descent method are used to update the parameters in the multi-level self-attention convolutional neural network structure until the multi-level self-attention neural network converges.

[0024] As a preferred technical solution of the present invention: the overall loss function of the multi-level self-attention neural network is as follows:

[0025]

[0026] Where J(·) represents the multi-scale dice loss function, represents interpolation, I A ,I B They represent the prostate organ region in the binary 3D prostate MRI image and the prostate organ region in the binary prostate TRUS image, respectively. α and μ represent the weight and displacement field between the similarity measure and the regularization term of the binary prostate organ region, respectively. C smooth (·) represents the bending energy loss function.

[0027] As a preferred technical solution of the present invention: the multi-scale dice loss function J(·) is as follows:

[0028]

[0029] Among them, Z represents the number of scales, f σ is a 3D Gaussian filter with isotropic standard deviation σ;

[0030] S Dice Defined as:

[0031]

[0032] Among them, two labels a={a t} and b={b t},a t , b t ∈[0,1], t=1,…,I, t represents a voxel in image I;

[0033] The bending energy loss function C smooth as follows:

[0034]

[0035] Among them, V represents the volume size of the displacement field, μ represents the displacement field, and X, Y, and Z represent the voxel sizes of the displacement field in the x, y, and z directions in three-dimensional space, respectively.

[0036] As a preferred technical solution of the present invention: the standard random K-minibatch gradient descent method requires an unbiased evaluation of the cumulative batch gradient for each minibatch into which the sample set is evenly divided:

[0037]

[0038] K represents the number of 3D prostate image pairs in each subset, J represents the label similarity measure, and θ represents the learnable weight parameter related to the label similarity measure in the multi-level self-attention convolutional neural network.

[0039] The structure-preserving weakly supervised multimodal prostate image registration method described in the present invention adopts the above technical solution and has the following technical effects compared with the prior art:

[0040] (1) The present invention designs a structure-preserving weakly supervised multimodal prostate image registration method, which obtains a prostate image transformation model through the designed steps i to ii; then executes steps A to B, applies the prostate image transformation model, and realizes the registration between the three-dimensional prostate MRI image and the prostate TRUS image of the prostate organ area in the known images corresponding to the target object; the scheme design introduces a multi-level self-attention mechanism in the weakly supervised cross-modal prostate registration network to establish long-range feature dependency relationships for prostate feature maps of different scales, ensures that prostate feature maps of different scales can carry more accurate spatial displacement information and reflect it on the displacement field, so as to achieve a more precise preservation of the internal organ structure in the registered prostate image without compromising the accuracy of prostate transformation.

[0041] (2) The present invention designs a structure-preserving weakly supervised multimodal prostate image registration method, which can, in real time, register the prostate MRI image with the prostate organ region in the corresponding prostate TRUS image, meeting the requirements of real-time image-guided surgery; the prostate organ quality and accuracy in the registered prostate MRI image are better, and important detail information can be preserved more finely; and the end-to-end weakly supervised registration technology is used, and a good registration effect can be achieved by simply inputting the prostate MRI image and the prostate TRUS image. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the weakly supervised multimodal prostate image registration method designed by the present invention with structure preservation;

[0043] Figure 2 It is a structural diagram of the self-attention mechanism block in the design of the present invention;

[0044] Figure 3 It is a structural diagram of a multi-level self-attention convolutional neural network designed in the present invention;

[0045] Figure 4 It is a schematic diagram of the weakly supervised registration training process architecture in the design of the present invention. DETAILED DESCRIPTION

[0046] The specific implementation modes of the present invention will be further described in detail below in conjunction with the accompanying drawings.

[0047] The present invention designs a structure-preserving weakly supervised multimodal prostate image registration method. In practical applications, such as Figure 4As shown, by executing the following steps i to ii, a prostate image transformation model is obtained.

[0048] Step i. Based on a preset number of three-dimensional prostate MRI sample images, and the binary three-dimensional prostate MRI sample images corresponding to the prostate organ area in the distinguished image, each three-dimensional prostate MRI sample image belongs to the same object as each three-dimensional prostate MRI sample image, and each binary prostate TRUS sample image corresponding to the prostate organ area in the distinguished image, combined with each displacement field of the method used to describe the transformation of the prostate organ area in each three-dimensional prostate MRI sample image to the contour shape of the prostate organ area in the prostate TRUS sample image of the same object, a sample set is constructed, and then step ii is entered.

[0049] In the application, the prostate organ region in each three-dimensional prostate MRI sample image and the prostate organ region in each three-dimensional prostate TRUS sample image are determined in advance by the following method before the scheme is executed.

[0050] First, the sample image is cropped outward from the center point to obtain an image with a resolution of 128*128*28; then the prostate organ area in the above image is manually outlined, and then binarization is performed to obtain the prostate organ area in each binary three-dimensional prostate MRI sample image and each binary three-dimensional prostate TRUS sample image.

[0051] Step ii. Based on the sample set, 70% is selected as the training set, 20% as the validation set and 10% as the test set. The three-dimensional prostate MRI sample image and the prostate organ region in the corresponding binarized three-dimensional prostate MRI sample image, as well as the prostate TRUS sample image belonging to the same object and the prostate organ region in the corresponding binarized prostate TRUS sample image are used as input. The displacement field of the method describing the transformation of the prostate organ region in the three-dimensional prostate MRI sample image to the contour shape of the prostate organ region in the prostate TRUS sample image is used as output. The network to be trained is trained to obtain a prostate image transformation model.

[0052] In practical applications, the network to be trained here is a multi-level self-attention convolutional neural network, such as Figure 3 As shown, it includes a first down-sampling module, a second down-sampling module, a third down-sampling module, a first up-sampling module, a second up-sampling module, a third up-sampling module, and four trilinear up-sampling modules.

[0053] Among them, the first downsampling module includes a self-attention mechanism block, three feature extraction modules, a redundant feature reduction module, and an addition module. The input end of the self-attention mechanism block constitutes the input end of the first downsampling module; the three feature extraction modules are connected in series in sequence, and the input end of the feature extraction module at the first position constitutes the input end of the series structure, and the output end of the feature extraction module at the first position and the output end of the feature extraction module at the third position are connected to the addition module, and the output end of the addition module constitutes the output end of the series structure; the output end of the self-attention mechanism block is connected to the input end of the series structure, the output end of the series structure is connected to the input end of the redundant feature reduction module, and the output end of the redundant feature reduction module constitutes the output end of the first downsampling module.

[0054] The structure of the second downsampling module is the same as that of the third downsampling module. The second downsampling module and the third downsampling module respectively include a self-attention mechanism block, two feature extraction modules, a redundant feature reduction module, and an addition module. The input end of the self-attention mechanism block constitutes the input end of the downsampling module, the two feature extraction modules are connected in series in sequence, and the output end of the self-attention mechanism block is connected to the input end of the series structure. At the same time, the output end of the self-attention mechanism block and the output end of the series structure are connected to the addition module, and the output end of the addition module is connected to the input end of the redundant feature reduction module, and the output end of the redundant feature reduction module constitutes the output end of the downsampling module.

[0055] The structures of the first upsampling module, the second upsampling module and the third upsampling module are the same, and each includes a scaling module, two feature extraction modules, an addition module and a trilinear addition upsampling module, wherein the input end of the scaling module and the input end of the trilinear addition upsampling module constitute the input end of the upsampling module, the two feature extraction modules are sequentially connected in series, and the input end of the series structure and the output end of the series structure are connected to the addition module, the output end of the series structure constitutes the output end of the series structure, and the output end of the series structure constitutes the output end of the upsampling module, and the output end of the scaling module and the output end of the trilinear addition upsampling module converge with each other to form a convergence end.

[0056] The input end of the first down-sampling module constitutes the input end of the multi-level self-attention neural network. The first down-sampling module, the second down-sampling module, and the third down-sampling module are connected in series in sequence. The output end of the third down-sampling module is connected to the input end of the first up-sampling module. The output end of the self-attention mechanism block in the third down-sampling module and the convergence end in the first up-sampling module are connected to the addition module, and the output end of the addition module is connected to the input end of the series structure in the first up-sampling module; the output end of the first up-sampling module is connected to the input end of the second up-sampling module, the output end of the self-attention mechanism block in the second down-sampling module and the convergence end in the second up-sampling module are connected to the addition module, and the output end of the addition module is connected to the input end of the series structure in the second up-sampling module; the output end of the second up-sampling module and the output end of the feature extraction module at the first position in the series structure in the first down-sampling module are connected to the addition module, and the output end of the addition module is connected to the input end of the third up-sampling module, and the convergence end in the third up-sampling module is connected to the input end of its series structure.

[0057] The output end of the third down-sampling module, the output end of the first up-sampling module, the output end of the second up-sampling module, and the output end of the third up-sampling module are respectively connected to the input end of each trilinear up-sampling module in a one-to-one correspondence, and the output ends of each trilinear up-sampling module are connected to an addition module, which constitutes the output end of the multi-level self-attention neural network.

[0058] In practical applications, such as Figure 2 As shown in Figure 2, the specific design of the self-attention mechanism block includes three 1x1x1 convolutional layers θ connected in series. Unary function g, softmax function module, matrix multiplication module, element-by-element summation module, maximum pooling layer, deconvolution layer, residual connection; and the mathematical expression of the self-attention mechanism block is defined as:

[0059]

[0060] Where i represents the index of the output position in space, time, or spatiotemporal where the response is computed, j represents the index enumerating all possible positions, x represents the input image signal, m represents the downsampled version of x, y represents the output signal of the same size as x, the pairwise function f(·) computes the scalar dependency between i and all j, the unary function g(·) computes the representation of the input signal at position j, and the response is normalized by the factor C(m);

[0061] The self-attention mechanism block calculates the response of the target position in the feature map as the weighted average of the features at all positions in the input feature map, thereby establishing dependencies between long-distance features.

[0062] And in the application, regarding the feature extraction module, the design specifically includes a convolution layer, a normalization layer, and an activation layer connected in series in sequence; the redundant feature reduction module includes a maximum pooling layer, a convolution layer, a normalization layer, and an activation layer connected in series in sequence; the scale expansion module includes a deconvolution layer, a normalization layer, and an activation layer connected in series in sequence; the trilinear upsampling module includes a convolution layer, a bias layer, and a cubic linear interpolation layer connected in series in sequence.

[0063] During network training, the overall loss function of the multi-level self-attention neural network is constructed based on the multi-scale dice function that describes the overlap between the binary prostate organ target image and the prostate organ region in the corresponding binary prostate TRUS image, combined with the bending energy loss function that penalizes the displacement field, as follows:

[0064]

[0065] Where J(·) represents the multi-scale dice loss function, represents interpolation, I A ,I B They represent the prostate organ region in the binary 3D prostate MRI image and the prostate organ region in the binary prostate TRUS image, respectively. α and μ represent the weight and displacement field between the similarity measure and the regularization term of the binary prostate organ region, respectively. C smooth (·) represents the bending energy loss function.

[0066] The multi-scale dice loss function J(·) is as follows:

[0067]

[0068] Among them, Z represents the number of scales, f σ is a 3D Gaussian filter with isotropic standard deviation σ.

[0069] Among them, S Dice Defined as:

[0070]

[0071] Among them, two labels a={a t} and b={b t},a t 、b t ∈[0,1], t=1,…,I, t represents a voxel in image I.

[0072] The bending energy loss function C smooth as follows:

[0073]

[0074] Among them, V represents the volume size of the displacement field, μ represents the displacement field, and X, Y, and Z represent the voxel sizes of the displacement field in the x, y, and z directions in three-dimensional space, respectively.

[0075] Based on the overall loss function of the above-mentioned multi-level self-attention neural network, during the training process of the multi-level self-attention neural network in step ii, the Adam optimizer and the standard random K-minibatch gradient descent method are used to update the parameters in the multi-level self-attention convolutional neural network structure until the multi-level self-attention neural network converges.

[0076] Among them, the standard random K-minibatch gradient descent method requires an unbiased evaluation of the cumulative batch gradient for each minibatch into which the sample set is evenly divided:

[0077]

[0078] K represents the number of 3D prostate image pairs in each subset, J represents the label similarity measure, and θ represents the learnable weight parameter related to the label similarity measure in the multi-level self-attention convolutional neural network.

[0079] Based on the prostate image transformation model obtained above, step A to step B are performed, such as Figure 1 As shown, the prostate image transformation model is applied to achieve the registration between the three-dimensional prostate MRI image and the prostate TRUS image of the prostate organ area in the known images corresponding to the target object.

[0080] Step A. For the three-dimensional prostate MRI image and prostate TRUS image corresponding to the target object, apply the prostate image transformation model to obtain a displacement field describing the method of transforming the prostate organ region in the three-dimensional prostate MRI image to the contour shape of the prostate organ region in the prostate TRUS image, and then proceed to step B.

[0081] Step B. From the three-dimensional prostate MRI image and according to the method described by the displacement field, a three-dimensional prostate MRI target image is obtained after the prostate organ region in the three-dimensional prostate MRI image is transformed to the contour shape of the prostate organ region in the prostate TRUS sample image, so as to achieve registration between the three-dimensional prostate MRI image and the prostate TRUS image corresponding to the target object.

[0082] The above technical solution designs a structure-preserving weakly supervised multimodal prostate image registration method, which introduces a multi-level self-attention mechanism in the weakly supervised multimodal prostate registration network to ensure that prostate feature maps of different scales can carry more accurate spatial displacement information and reflect it in the displacement field, thereby achieving a more precise preservation of the internal organ structure in the registered prostate image without compromising the accuracy of prostate transformation.

[0083] In the application, the prostate MRI image can be registered with the prostate organ in the corresponding prostate TRUS image in real time to meet the requirements of real-time image-guided surgery; the quality and accuracy of the prostate organ in the registered prostate MRI image are better, and important detail information can be preserved more finely; and the end-to-end weakly supervised registration technology is used, and a good registration effect can be achieved by simply inputting the prostate MRI image and the prostate TRUS image.

[0084] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A structure-preserving weakly supervised multimodal prostate image registration method, Features: Through step i to step ii, a prostate image transformation model is obtained; then steps A to B are performed, and the prostate image transformation model is applied to achieve registration between the three-dimensional prostate magnetic resonance MRI image and the prostate transrectal ultrasound TRUS image corresponding to the prostate organ area in the known images of the target object; Step i. Based on a preset number of three-dimensional prostate MRI sample images, and the binary three-dimensional prostate MRI sample images corresponding to the prostate organ region in the distinguished image, the prostate TRUS sample images belonging to the same object as the three-dimensional prostate MRI sample images, and the binary prostate TRUS sample images corresponding to the prostate organ region in the distinguished image, in combination with the displacement fields of the method for describing the transformation of the prostate organ region in each three-dimensional prostate MRI sample image to the contour shape of the prostate organ region in the prostate TRUS sample image of the same object, a sample set is constructed, and then step ii is entered; Step ii. Based on the sample set, the three-dimensional prostate MRI sample image and the prostate organ region in the corresponding binary three-dimensional prostate MRI sample image, as well as the prostate TRUS sample image and the prostate organ region in the corresponding binary prostate TRUS sample image belonging to the same object are used as input, and the displacement field describing the method of transforming the prostate organ region in the three-dimensional prostate MRI sample image to the contour shape of the prostate organ region in the prostate TRUS sample image is used as output, and the network to be trained is trained to obtain a prostate image transformation model; Step A. applying a prostate image transformation model to the three-dimensional prostate MRI image and the prostate TRUS image corresponding to the target object, obtaining a displacement field describing a method of transforming a prostate organ region in the three-dimensional prostate MRI image to a contour shape of the prostate organ region in the prostate TRUS image, and then proceeding to step B; Step B. Obtaining, from the three-dimensional prostate MRI image and according to the method described by the displacement field, a three-dimensional prostate MRI target image after the prostate organ region in the three-dimensional prostate MRI image is transformed into the contour shape of the prostate organ region in the prostate TRUS sample image, so as to achieve registration between the three-dimensional prostate MRI image and the prostate TRUS image corresponding to the target object; The network to be trained in step ii is a multi-level self-attention convolutional neural network, including a first downsampling module, a second downsampling module, a third downsampling module, a first upsampling module, a second upsampling module, a third upsampling module, and four trilinear upsampling modules; The first downsampling module includes a self-attention mechanism block, three feature extraction modules, a redundant feature reduction module, and an addition module, wherein the input end of the self-attention mechanism block constitutes the input end of the first downsampling module; the three feature extraction modules are sequentially connected in series, and the input end of the feature extraction module at the first position constitutes the input end of the series structure, and the output end of the feature extraction module at the first position and the output end of the feature extraction module at the third position are connected to the addition module, and the output end of the addition module constitutes the output end of the series structure; the output end of the self-attention mechanism block is connected to the input end of the series structure, the output end of the series structure is connected to the input end of the redundant feature reduction module, and the output end of the redundant feature reduction module constitutes the output end of the first downsampling module; The structure of the second downsampling module is the same as that of the third downsampling module. The second downsampling module and the third downsampling module respectively include a self-attention mechanism block, two feature extraction modules, a redundant feature reduction module, and an addition module, wherein the input end of the self-attention mechanism block constitutes the input end of the downsampling module, the two feature extraction modules are sequentially connected in series, the output end of the self-attention mechanism block is connected to the input end of the series structure, and at the same time, the output end of the self-attention mechanism block and the output end of the series structure are connected to the addition module, the output end of the addition module is connected to the input end of the redundant feature reduction module, and the output end of the redundant feature reduction module constitutes the output end of the downsampling module; The structures of the first upsampling module, the second upsampling module, and the third upsampling module are the same, and each includes a scaling module, two feature extraction modules, an addition module, and a trilinear addition upsampling module, wherein the input end of the scaling module and the input end of the trilinear addition upsampling module constitute the input end of the upsampling module, the two feature extraction modules are sequentially connected in series, and the input end of the series structure and the output end of the series structure are connected to the addition module, the output end of the addition module constitutes the output end of the series structure, and the output end of the series structure constitutes the output end of the upsampling module, and the output end of the scaling module and the output end of the trilinear addition upsampling module converge with each other to form a convergence end; The input end of the first down-sampling module constitutes the input end of the multi-level self-attention convolutional neural network, the first down-sampling module, the second down-sampling module, and the third down-sampling module are connected in series in sequence, the output end of the third down-sampling module is connected to the input end of the first up-sampling module, the output end of the self-attention mechanism block in the third down-sampling module and the convergence end in the first up-sampling module are connected to the addition module, and the output end of the addition module is connected to the input end of the series structure in the first up-sampling module; the output end of the first up-sampling module is connected to the input end of the second up-sampling module, the output end of the self-attention mechanism block in the second down-sampling module and the convergence end in the second up-sampling module are connected to the addition module, and the output end of the addition module is connected to the input end of the series structure in the second up-sampling module; the output end of the second up-sampling module and the output end of the feature extraction module at the first position in the series structure in the first down-sampling module are connected to the addition module, the output end of the addition module is connected to the input end of the third up-sampling module, and the convergence end in the third up-sampling module is connected to the input end of its series structure; The output end of the third down-sampling module, the output end of the first up-sampling module, the output end of the second up-sampling module, and the output end of the third up-sampling module are respectively connected to the input end of each trilinear up-sampling module in a one-to-one correspondence, and the output ends of each trilinear up-sampling module are connected to an addition module, which constitutes the output end of a multi-level self-attention convolutional neural network.

2. According to the structure-preserving weakly supervised multimodal prostate image registration method of claim 1, Features: The self-attention mechanism block includes three 1x1x1 convolutional layers connected in parallel, a unary function g, a softmax function module, a matrix multiplication module, an element-by-element summation module, a maximum pooling layer, a deconvolution layer, and a residual connection.

3. According to the structure-preserving weakly supervised multimodal prostate image registration method of claim 2, Features: The mathematical expression of the self-attention mechanism block is defined as: where i is the index of the output location in space, time, or spatiotemporal where the response is computed, j is the index enumerating all possible locations, x is the input image signal, m is a downsampled version of x, y is the output signal of the same size as x, the pairwise function f(·) computes a scalar dependency between i and all j, the unary function g(·) computes the representation of the input signal at location j, and the response is normalized by the factor Cm.

4. According to claim 1, a structure-preserving weakly supervised multimodal prostate image registration method, Features: The feature extraction module includes a convolution layer, a normalization layer, and an activation layer connected in series in sequence; the redundant feature reduction module includes a maximum pooling layer, a convolution layer, a normalization layer, and an activation layer connected in series in sequence; the scale expansion module includes a deconvolution layer, a normalization layer, and an activation layer connected in series in sequence; the trilinear upsampling module includes a convolution layer, a bias layer, and a trilinear interpolation layer connected in series in sequence.

5. According to the structure-preserving weakly supervised multimodal prostate image registration method of claim 1, Features: According to the multi-scale dice function describing the overlap between the binary prostate organ target image and the prostate organ area in the corresponding binary prostate TRUS image, combined with the bending energy loss function of the penalty displacement field, the overall loss function of the multi-level self-attention convolutional neural network is constructed; and in the training process of the multi-level self-attention convolutional neural network in the step ii, the Adam optimizer and the standard random K-minibatch gradient descent method are used to update the parameters in the multi-level self-attention convolutional neural network structure until the multi-level self-attention convolutional neural network converges.

6. A structure-preserving weakly supervised multimodal prostate image registration method according to claim 5, Features: The overall loss function of the multi-level self-attention convolutional neural network is as follows: Where J· represents the multi-scale dice loss function, represents interpolation, I A ,I B They represent the prostate organ region in the binary 3D prostate MRI image and the prostate organ region in the binary 3D prostate TRUS image, respectively. α and μ represent the weight and displacement field between the similarity measure and the regularization term of the binary prostate organ region, respectively. smooth · represents the bending energy loss function.

7. A structure-preserving weakly supervised multimodal prostate image registration method according to claim 6, Features: The multi-scale dice loss function J· is as follows: Among them, Z represents the number of scales, f σ is a 3D Gaussian filter with isotropic standard deviation σ; S Dice Defined as: Among them, two labels a={a t } and b={b t },a t 、b t ∈[0,1], t=1,…,I, t represents a voxel in image I; the bending energy loss function C smooth as follows: Among them, V represents the volume size of the displacement field, μ represents the displacement field, and X, Y, and Z represent the voxel sizes of the displacement field in the x, y, and z directions in three-dimensional space, respectively.

8. According to claim 5, a structure-preserving weakly supervised multimodal prostate image registration method, Features: The standard stochastic K-minibatch gradient descent method requires an unbiased evaluation of the cumulative batch gradient for each minibatch into which the sample set is evenly divided: K represents the number of 3D prostate image pairs in each subset, J represents the label similarity measure, and θ represents the learnable weight parameter related to the label similarity measure in the multi-level self-attention convolutional neural network.