A GAN-based method for motion compensation during TRUS prostate biopsy

Abstract

This invention provides a GAN-based method for motion compensation in prostate biopsy via TRUS, comprising: acquiring intraoperative 2D TRUS images and preoperative 3D TRUS images; inputting the intraoperative 2D TRUS images into a pre-trained registration network, which is trained as a generator in a generative adversarial network; the registration network predicting and outputting spatial transformation parameters between the intraoperative 2D TRUS images and the preoperative 3D TRUS images based on the features of the intraoperative 2D TRUS images; and applying the spatial transformation parameters to the preoperative 3D TRUS images to achieve motion compensation of the prostate. This invention integrates a Self-Attention Feature Reassignment (SAFR) module to achieve accurate extraction of the overall structure and boundaries of the prostate, and designs a physical decoupling regression head at the output of the registration network to achieve separation of physical dimensions, gradient balancing, and output range control.

Description

Technical Field

[0001] This invention belongs to the field of deep learning image processing technology, and particularly relates to medical 2D / 3D TRUS image registration technology, specifically a GAN-based TRUS prostate puncture motion compensation method. Background Technology

[0002] Prostate cancer is extremely common among men. Early detection, prevention, and treatment in high-risk groups can improve treatment outcomes and, to some extent, increase survival rates. Prostate biopsy is an important method for diagnosing prostate cancer, and transrectal ultrasound (TRUS) is widely used due to its short imaging time. During the procedure, clinicians use a fine needle attached to a biopsy gun to collect suspicious tissue samples from the prostate for testing, based on real-time TRUS imaging. However, due to the relatively poor imaging quality and low resolution of TRUS, cancers in the anterior part of the prostate are often overlooked, and approximately 30% of cancers are located in this area, resulting in a 30% false negative rate with TRUS prostate biopsy.

[0003] In recent years, magnetic resonance imaging (MRI) has been widely used in the identification of prostate lesions due to its high image quality. However, MRI has the disadvantage of long imaging time, making it impossible to guide targeted prostate puncture in real time.

[0004] With the development of modern medical technology, MRI and TRUS fusion biopsy technology has been applied to the identification of prostate cancer lesions. Clinicians obtain 3D MRI and 3D TRUS images of the prostate preoperatively and then register them. This involves establishing a spatial coordinate system to map the location information of the lesion on the 3D MRI image onto the 3D TRUS image. Then, the intraoperative 2D TRUS image is aligned with the preoperative 3D TRUS image. The registered lesion location information is displayed in real-time on the 3D TRUS image, guiding precise prostate biopsy. By combining the high-definition imaging characteristics of MRI with the real-time imaging efficiency of TRUS, MRI and TRUS fusion biopsy technology helps doctors more accurately locate suspicious tissue and perform sampling and testing, significantly improving the diagnostic rate of prostate cancer.

[0005] However, during prostate biopsy, patient discomfort causing body tremors and the pressure of the ultrasound probe inevitably lead to overall prostate movement. This results in a discrepancy between the preoperative lesion location coordinates established by 3D TRUS images and the actual intraoperative location coordinates. This is a major problem in current MRI and TRUS-guided prostate biopsies. Therefore, selecting an appropriate motion compensation method to compensate for the prostate's movement, ensuring that the compensated results accurately reflect the coordinates of lesions during the procedure, and guiding clinicians to perform precise biopsies of suspicious lesions, is of great significance for the clinical diagnosis of prostate cancer.

[0006] In existing technologies, motion compensation methods typically employ Generative Adversarial Networks (GANs), including CGAN / WGAN / SAGAN / SGAN, etc. These methods differ significantly in terms of task objectives, input-output results, generator structure, discriminator structure, and training methods, resulting in compensated image results that fail to accurately reflect the location of lesions and thus cannot meet clinical needs. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to provide a GAN-based TRUS prostate biopsy motion compensation method. This method can accurately acquire the spatial positional relationship between intraoperative 2D TRUS images and preoperative 3D TRUS images, thereby compensating for motion during prostate targeted biopsy and facilitating precise biopsy of suspicious lesions by clinicians. To achieve the above objective, the specific technical solution adopted by the present invention is as follows:

[0008] A GAN-based TRUS prostate biopsy motion compensation method specifically includes the following steps:

[0009] Acquire intraoperative 2D TRUS images and preoperative 3D TRUS images;

[0010] The intraoperative 2D TRUS image is input into a pre-trained registration network, which is trained as a generator in a generative adversarial network.

[0011] The registration network predicts and outputs the spatial transformation parameters between the intraoperative 2D TRUS image and the preoperative 3D TRUS image based on the features of the intraoperative 2D TRUS image.

[0012] The spatial transformation parameters are applied to the preoperative 3D TRUS image to achieve motion compensation for the prostate.

[0013] The registration network includes a multi-scale feature extraction module, which consists of a global feature extraction module, a local detail enhancement module, and a multi-scale feature fusion module.

[0014] The global feature extraction module integrates a self-attention feature redistribution (SAFR) module. The SAFR module generates query vectors, key vectors, and value vectors through 1×1 convolution, calculates the global correlation matrix between spatial locations, and adaptively controls the intensity of attention mapping using a learnable parameter to fuse the weighted features with the original features.

[0015] The steps for predicting spatial transformation parameters are to use a physical decoupling regression head to divide the spatial transformation parameters into at least three subgroups: large-angle rotation, small-angle rotation, and translation distance. The parameters of each subgroup are predicted by regression through an independent fully connected layer to decouple the gradients between parameters of different physical dimensions.

[0016] Preferably, the formula for the SAFR module to calculate the global correlation matrix between spatial locations is:

[0017] β = softmax(QK ),

[0018] Where Q is the query vector, K is the key vector, and T is the matrix transpose;

[0019] The intensity of the attention mapping is controlled by a learnable parameter γ1, and the final output feature Y is:

[0020] Y = γ1·(βV) + X,

[0021] Where X is the input feature before mapping, and V is the value vector.

[0022] Preferably, the dataset is further enriched by performing translation, flipping, and rotation operations on the 2D TRUS images before training the registration network.

[0023] Preferably, the local detail enhancement module adopts a residual dense block structure, which contains multiple densely connected convolutional layers.

[0024] Preferably, the training process of the registration network includes:

[0025] The registered images generated by the registration network are used as negative examples, and the intraoperative 2D TRUS images are used as positive examples.

[0026] Positive and negative samples are input together into the discrimination network in the generative adversarial network for authenticity determination;

[0027] Based on the discrimination results of the discrimination network, the parameters of the registration network and the discrimination network are optimized through backpropagation.

[0028] Preferably, the training process further includes: the discrimination network determining the geometric consistency between the registered image and the intraoperative 2D TRUS image, and the error signal being transmitted in reverse to the registration network.

[0029] Preferably, the training process is guided by a joint loss function, which is a weighted sum of the registration network loss function and the discriminant network loss function.

[0030] Preferably, the registration network loss function consists of rotation angle loss and displacement distance loss calculated based on the mean square error loss function, and is expressed as: ,in Let the rotation angle loss function be... The displacement distance loss function, For hyperparameters;

[0031] Both loss functions are calculated using the mean squared error loss function, as shown in the equation:

[0032]

[0033] in, It is the true label value of the i-th sample. This is the predicted value of the i-th sample, expressed as a rotation angle or displacement distance. Indicates the first in the data One sample;

[0034] The loss function for the discrimination network is calculated using the binary classification cross-entropy loss function, expressed as:

[0035]

[0036] in The label value of the i-th image is represented by the set of values. , This represents the probability that the label value of the i-th image is predicted to be 1, where n is the sample size.

[0037] Preferably, the discrimination network includes five convolutional layers and one fully connected layer. The fully connected layer uses patch-level output and scores each position on a 4×4×512 feature. The convolutional layers use a 3×3 kernel with a stride of 2. The fully connected layer is followed by a Sigmoid activation function.

[0038] Preferably, the global feature extraction module consists of 4 convolutional layers, 2 max pooling layers, an upsampling layer, a batch normalization (BN) layer, and a self-attention mechanism module.

[0039] The beneficial effects of this invention are as follows: 1) A registration network based on Generative Adversarial Network (GAN) is selected, and the spatial transformation parameters are optimized so that the spatial positions determined on the preoperative 3D TRUS image can be accurately and in real time matched with the positions in the intraoperative 2D TRUS image after compensation; 2) A self-attention mechanism is introduced, and a self-attention feature redistribution (SAFR) module is integrated to achieve accurate extraction of the overall structure and boundaries of the prostate; 3) The registration network contains a physical decoupling regression head to decouple the gradients between parameters of different physical dimensions, thereby achieving separation of physical dimensions, gradient balancing, and output range control; 4) The adversarial training mechanism of Generative Adversarial Network (GAN) is used to force the registration network (as a generator) to continuously optimize, striving to generate a registration image that is indistinguishable from the real intraoperative image, thereby improving the prediction accuracy and robustness of the spatial transformation parameters; 5) A geometric adversarial learning mechanism is proposed, and a loss function is designed: a combination of rotation + displacement + binary cross-entropy to ensure registration accuracy and network stability. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the process for identifying adversarial training networks according to the present invention;

[0041] Figure 2 This is a schematic diagram of the multi-scale feature extraction module of the present invention;

[0042] Figure 3 This is a schematic diagram of the motion compensation method of the present invention;

[0043] Figure 4 This is a flowchart of a GAN-based TRUS prostate puncture motion compensation method according to the present invention.

[0044] Figure 5 This is a schematic diagram of a 3D TRUS image;

[0045] Figure 6 This is a schematic diagram of the generation of a 2D sliced ​​TRUS image sequence;

[0046] Figure 7 It is a prostate motion compensation framework with rigid registration of 2D / 3D TRUS images;

[0047] Figure 8 It is a prostate motion compensation network with rigid registration of 2D / 3D TRUS images;

[0048] Figure 9 This is a schematic diagram of the registration network structure;

[0049] Figure 10 This is a schematic diagram of the global feature extraction module structure;

[0050] Figure 11 This is a schematic diagram of the self-attention mechanism structure;

[0051] Figure 12 This is a schematic diagram of the local detail enhancement module structure;

[0052] Figure 13 This is a schematic diagram of the identification network structure;

[0053] Figure 14 This is a comparison chart of the registration results of the method of this invention and the methods of CGAN, SAGAN, and WGAN. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative effort are all within the scope of protection of the present invention.

[0055] like Figures 1-14 As shown, this invention proposes a GAN-based TRUS prostate puncture motion compensation method, the specific implementation steps of which are described below:

[0056] First, such as Figure 1 , Figure 2 , Figure 3 As shown, a GAN-based TRUS prostate biopsy motion compensation method is proposed. Its core innovation lies in the network structure design and adversarial training method, specifically reflected in:

[0057] Based on the GAN framework and parameter prediction, the core of this method is a pre-trained registration network. This registration network is functionally designed as a generator within a Generative Adversarial Network (GAN). Its main task is, after receiving an input intraoperative 2D TRUS image, to predict and output the spatial transformation parameters between the 2D image and the preoperative 3D TRUS image by analyzing its image features. These parameters are then applied to the preoperative 3D TRUS image to achieve motion compensation for the prostate.

[0058] To ensure the prediction accuracy of spatial transformation parameters, the registration network includes a specially designed multi-scale feature extraction module. This module, a key innovation of this invention, consists of two cooperating sub-modules:

[0059] The global feature extraction module is responsible for capturing key information about the prostate region based on the overall structure of the image. To effectively expand the network's receptive field and perform global information interaction, this module also integrates a self-attention mechanism.

[0060] The local detail enhancement module compensates for image detail that may be lost during the global feature extraction process. Ultimately, the global features and local detail information are combined to generate information-rich multi-scale features for predicting transformation parameters.

[0061] The training process of the registration network (i.e., the generator) employs an adversarial training mechanism, specifically:

[0062] Real intraoperative 2D TRUS images are defined as positive examples.

[0063] The registered images generated by the registration network based on the prediction parameters are defined as negative examples.

[0064] Both types of samples are fed into the discrimination network in the GAN.

[0065] The discriminant network is responsible for distinguishing between genuine and fake images. Based on its judgment, it optimizes the parameters of both the registration network and the discriminant network through backpropagation. This adversarial mechanism forces the registration network to continuously improve its predictive ability to generate a registered image that the discriminant network cannot distinguish, thereby greatly improving the prediction accuracy of spatial transformation parameters.

[0066] The GAN in this invention is not structurally based on the encoder-decoder generative framework of existing generative adversarial networks (including CGAN / SAGAN / WGAN / SGAN). Instead, it redefines the roles of the generator and discriminator, and modifies the training method to be tailored for registration tasks of prostate ultrasound images. A comparison overview is shown in Table 1 below:

[0067] Table 1. Comparison between the present invention and the prior art

[0068]

[0069] To further illustrate the technical solution of the present invention, such as... Figure 4 As shown, the compensation method specifically includes the following steps:

[0070] S1. Obtain the preoperative 3D TRUS image dataset and the intraoperative 2D TRUS image dataset respectively;

[0071] The 2D slice TRUS image is obtained by sampling the slice from the 3D TRUS image;

[0072] like Figure 5 The image shown is an example of 3D TRUS image data;

[0073] like Figure 6 The image shown is an example of generating a 2D slice TRUS image sequence.

[0074] S2. Divide the patient's 3D TRUS images into training and testing sets in an 8:2 ratio.

[0075] S3. Within the spatial transformation parameter distribution range, generate 3D TRUS images of the cross-section in the training set according to a uniform distribution to complete sampling, and generate 2D slice TRUS images for use in training the network model. Intraoperative 2D TRUS images are used to test the network model. Training is performed in a motion-compensated network.

[0076] S4. To avoid overfitting during training, data augmentation is used to enrich the dataset.

[0077] S5. Input the intraoperative 2D TRUS image into the prostate motion compensation network based on the rigid registration of 2D / 3D TRUS images using generative adversarial network (GAN) to predict the spatial transformation parameters of the 2D / 3D TRUS image.

[0078] like Figure 7 As shown, this is the prostate motion compensation framework for rigid registration of 2D / 3D TRUS images.

[0079] S6. First, the intraoperative 2D TRUS image is input into the prostate motion compensation network for 2D / 3D TRUS image registration to predict the spatial transformation parameters of the 2D / 3D TRUS image. These parameters consist of three rotation angle parameters and three displacement distance parameters, represented by α, β, and γ, and x, y, and z, respectively. Next, motion compensation is performed based on the obtained spatial transformation parameters. The spatial transformation parameters are then transformed using rigid body transformation to generate a cross-section of the 3D TRUS image. This cross-section is used to sample and generate the registration image from the 3D TRUS image. The correspondence between the pixel coordinates of the registration image and the intraoperative 2D TRUS image at this point represents the spatial positional relationship after motion compensation.

[0080] During the intraoperative puncture process, clinicians can locate the corresponding position coordinates on the 3D TRUS image based on the registration results of the 2D / 3D TRUS images, thereby guiding the precise puncture of the lesion during the operation.

[0081] like Figure 8 As shown, the prostate motion compensation network for rigid registration of 2D / 3D TRUS images designed in this invention includes a generative adversarial network and a discriminant network, and uses the generative adversarial network as the registration network to generate spatial transformation parameters.

[0082] First, the intraoperative 2D TRUS image is input into the registration network. Features of the intraoperative 2D TRUS image are extracted to obtain spatial transformation parameters. After rigid body transformation using the spatial transformation parameters, the 3D TRUS image is cross-sampled to obtain the registered image. Then, the registered image is used as a negative sample, and the intraoperative 2D TRUS image is used as a positive sample. Together, they are input into the discrimination network for positive and negative binary classification to distinguish whether the input image is a real image or a fake image.

[0083] S7. The spatial transformation parameters between the intraoperative 2D TRUS image and the preoperative 3D TRUS image are predicted by training the registration network. The 3D TRUS image is then sampled using the spatial transformation parameters after rigid body transformation to generate a registration image similar to the intraoperative 2D TRUS image. The discrimination network is then trained using the registration image to improve its ability to distinguish between the registered images.

[0084] S8. During training, calculate the loss functions of the registration network and the discriminator network respectively, and optimize the parameters of the registration network and the discriminator network through backpropagation.

[0085] The registration network is an important part of the prostate motion compensation network for rigid registration of 2D / 3D TRUS images. It consists of convolutional layers, multi-scale extraction modules, and fully connected layers.

[0086] like Figure 9 As shown, it first consists of a convolution kernel with a size of 7. 7. A convolutional layer with a stride of 1 extracts shallow features from the input image and uses them as input to the multi-scale feature extraction module. The multi-scale feature extraction module consists of a global feature extraction module and a local detail enhancement module. The input features are processed by the global feature extraction module and the local detail enhancement module, which output deep-level global and local feature information. Then, residual connections are used to combine the global and local detail features to obtain deep-level multi-scale features.

[0087] S9. The present invention designs a global feature extraction module to capture key information of the prostate region from the overall structure of the intraoperative 2D TRUS image, and uses the preoperative 3D TRUS image information to grasp the overall shape, size and positional relationship of the prostate boundary, thereby improving the prediction accuracy of spatial transformation parameters.

[0088] like Figure 10 As shown, the global feature extraction module consists of 4 convolutional kernels with a size of 3. 3. A convolutional layer with a stride of 1, and two pooling windows with a size of 2. 2. It consists of a max pooling layer with a stride of 2, an upsampling layer, a batch normalization (BN) layer, and a self-attention mechanism (SAM) module. Each convolutional layer is followed by a LeakyReLU activation function, which enables the network to handle nonlinear problems.

[0089] The self-attention mechanism is a core component in the global feature extraction module, which plays a role in expanding the network's receptive field and enabling global information interaction.

[0090] like Figure 11 As shown, the specific process is as follows: the input feature map is compared with three learnable matrices. , as well as Multiplication is performed to achieve a linear mapping, outputting a value vector V and a query vector. and key vector First, the query vector With key vector The attention score matrix is ​​obtained through multiplication, reflecting the correlation between various positions in the input sequence. This matrix is ​​then normalized using the Softmax function, transforming it into a probability distribution and outputting the attention weight matrix. Finally, it is multiplied with the value vector V to generate the final output sequence.

[0091] S10. The present invention also designs a local detail enhancement module to compensate for the detail information lost in the global feature extraction module, and combines it with the global features to obtain multi-scale features.

[0092] like Figure 12 As shown, the local detail enhancement module uses a Residual Dense Block (RDB), which includes a dense connection module and a feature fusion module. The dense connection module consists of 5 convolutional layers, with each convolutional kernel having a size of 3. 3. The step size is 1.

[0093] S11. The discrimination network is responsible for discriminating the input image. The intraoperative 2D TRUS image is used as the reference image, i.e., the positive sample, and the registration image is used as the negative sample. The discrimination network determines whether the input image is the reference image or the registration image.

[0094] like Figure 13 As shown, the discrimination network designed in this invention comprises five convolutional layers and one fully connected layer. The convolutional kernels used in the convolutional layers have a size of 3. 3. The stride is 2, and the number of convolutional kernels is 32, 64, 128, 256, and 512 respectively. After the first convolutional layer, only the LeakyReLU activation function is added to increase the network's non-linearity while preserving the original data distribution. After the second to fifth convolutional layers, LeakyReLU activation functions and Batch Normalization (BN) layers are added to increase non-linearity and accelerate network convergence. The sixth layer is a fully connected layer with 1 neuron, followed by a Sigmoid activation function to distribute the predicted values... Within the range, the probability that the output registered image is identified as the reference image.

[0095] S12. The loss function of the prostate motion compensation network based on rigid registration of 2D / 3D TRUS images is used to guide the improvement of the network's performance.

[0096] The loss function designed in this invention consists of a registration network loss function and a discrimination network loss function, and can be expressed as:

[0097] ;

[0098] In the formula, To register the network loss function, To identify the network loss function, These are hyperparameters used to balance the loss function of the registration network and the loss function of the discrimination network;

[0099] The registration network loss function consists of a rotation angle loss function and a displacement distance loss function, and can be expressed as:

[0100] ;

[0101] In the formula, Let the rotation angle loss function be... The displacement distance loss function, For hyperparameters;

[0102] Both loss functions are calculated using the mean squared error loss function:

[0103]

[0104] In the formula, It is the true label value of the i-th sample. This is the predicted value of the i-th sample, expressed as a rotation angle or displacement distance. Indicates the first in the data One sample;

[0105] The loss function for the discriminative network uses binary cross-entropy, which can be expressed as:

[0106] ;

[0107] in The label value of the i-th image is represented by the set of values. , This represents the probability that the label value of the i-th image is predicted to be 1, where n is the sample size.

[0108] S13. To verify the effectiveness of the invention, a comparative experiment was designed to analyze the registration results of different methods and compare them with existing CGAN, SAGAN and WGAN methods.

[0109] like Figure 14 As shown, this paper presents four sets of experimental comparisons between the registered images of the present invention and existing CGAN, SAGAN, WGAN, and reference image methods.

[0110] from Figure 14 As can be seen, the CGAN method produces poor registration results, exhibiting significant differences not only in overall structure but also in local morphology. While the SAGAN and WGAN methods produce registration results that are relatively close to the reference image in overall structure, their local morphology still deviates to varying degrees. In contrast, the registration image produced by the method of this invention shows smaller discrepancies with the reference image in both overall structure and local morphology. The qualitative analysis above demonstrates that the registration effect of the method of this invention is superior to the comparative methods mentioned above.

[0111] The technical solution of the present invention, a transrectal ultrasound (TRUS) prostate biopsy motion compensation method based on generative adversarial networks (GAN), will be described more clearly and completely below with reference to the accompanying drawings.

[0112] Step 1: Data Preparation and Preprocessing

[0113] First, preoperative 3D TRUS images and intraoperative 2D TRUS images were acquired. The patient's 3D TRUS image data were divided into training and testing sets in an 8:2 ratio.

[0114] To generate 2D slice images for training the network model, simulated intraoperative 2D slice images are generated by uniformly sampling within a preset spatial transformation parameter distribution range. These generated 2D slice images are used as training data for the network model. Real intraoperative 2D TRUS images are then used to test the trained model.

[0115] To prevent overfitting during training and to enhance the model's generalization ability, data augmentation techniques were employed to expand the training dataset. Specific operations included translating, horizontally or vertically flipping, and randomly rotating the 2D TRUS images, thus expanding a single original image into multiple new images. Figure 5 Examples of 3D TRUS images are shown. Figure 6 This illustrates the process of generating a 2D slice sequence from a 3D image.

[0116] Step 2: GAN-based motion compensation network framework

[0117] This invention proposes a rigid registration framework for 2D / 3D TRUS images based on Generative Adversarial Networks (GANs) for achieving motion compensation of the prostate. For example... Figure 9 and Figure 13 As shown, the framework mainly consists of two parts: a registration network (as the generator in GAN) and a discriminator network (discriminator).

[0118] Its core process is:

[0119] Parameter prediction: An intraoperative 2D TRUS image is input into a pre-trained registration network. The registration network extracts features from the image and predicts the spatial transformation parameters between the intraoperative 2D TRUS image and the preoperative 3D TRUS image. These parameters consist of six components: three rotation angles (α, β, γ) and three displacement distances (x, y, z).

[0120] Motion compensation: The preoperative 3D TRUS image is rigidly transformed and sectioned using predicted spatial transformation parameters. This generates a new 2D image, called the "registered image." This registered image corresponds spatially to the input intraoperative 2D TRUS image, thus compensating for prostate motion.

[0121] Adversarial Training: This invention proposes a Geometric Adversarial Learning (GAL) mechanism. Through alternating training of the generator and discriminator, the network simultaneously possesses numerical supervision constraints and geometric structure feedback during pose parameter prediction. The discriminator determines the geometric consistency between the registered image under the predicted pose and the intraoperative 2D TRUS image, and propagates the error signal back to the registration network. Under the constraint of the joint loss function, the registration network simultaneously minimizes the adversarial loss through the registration network loss function, achieving joint optimization of supervision accuracy and adversarial consistency. This mechanism can be regarded as "Adversarial Geometry Regularization," enabling the generator to approximate the real geometric relationship within a physically feasible pose distribution. Compared with traditional GANs, GAL shows significant advantages in training stability, pose distribution generalization, and local structure awareness. Traditional GANs are adversarial in the image pixel domain, while our adversarial training occurs on the path of "pose parameters → rendered geometry → discriminant result"; the discriminator forces the generator to learn pose parameters that conform to the real geometric distribution by feeding back gradients. With the backpropagated gradient from the discriminator, the generator learns to focus on the "regions most sensitive to the discriminator's judgment"—that is, areas with local structural misalignment, edge shift, etc.; this is equivalent to an adaptive weighting mechanism: the model automatically strengthens the accuracy of geometrically critical regions; enabling the generator to evolve from "globally minimizing the mean square error" to "focusing on correcting structures deemed incorrect by the discriminator." The effect of this is that pose prediction is more robust in ultrasound images with speckle noise and low contrast.

[0122] Step 3: Detailed Network Structure Design

[0123] 1. Registration Network (Generator)

[0124] Registration network (generator) such as Figure 9 As shown, the registration network aims to accurately predict spatial transformation parameters, and its structure integrates multi-scale information. It mainly consists of a shallow feature extraction module and a multi-scale feature extraction module, specifically including:

[0125] a. Global Feature Extraction Module (GCM), such as Figure 10As shown, this module is responsible for capturing key information such as the prostate's contour, size, and location from the overall structure of the image. It consists of convolutional layers, max pooling layers, and a self-attention mechanism (SAM). To enhance the generator's ability to model global dependencies in rigid registration, this invention introduces a self-attention feature redistribution (SAFR) module on the mid-layer features (128×128×64). This module generates Query, Key, and Value features through three 1×1 convolutions, respectively, and reduces computational complexity with a channel compression ratio of 1 / 8; it also uses β = softmax(QK) to achieve the desired effect. The global correlation matrix between spatial locations is calculated, and the intensity of the attention mapping is controlled by a learnable parameter γ1. The final output is Y = γ1·(βV) + X, which achieves adaptive enhancement of key structural regions while maintaining the original resolution. Key improvements in the SAFR module structure: 1. Compared with traditional methods (such as SGAN) that apply the self-attention mechanism to high-level / full-image features, the SAFR module acts on mid-level (128×128×64) features, preserving sufficient spatial resolution while already having high semantic features, thus balancing resolution and network receptive field; 2. From the perspective of weighting strategy, traditional self-attention mechanisms usually adopt a strategy of fixed residual summation, while SAFR innovatively uses adaptive attention weight control, using a learnable parameter γ1 to control the attention intensity.

[0126] Table 2 shows a comparative analysis of the structure of the self-attention mechanism module of this invention with other methods:

[0127] Table 2. Comparison of the structure of the self-attention mechanism module of this invention with other methods.

[0128]

[0129] As shown in Table 3, compared with attention methods such as SENet, CBAM, and GAM, the SAFR module reduces rotation RMSE, translation RMSE, and MAE of the six rigid body pose parameters by 50.9%, 44.1%, and 47.7%, respectively, while the inference latency remains basically unchanged.

[0130] Table 3. Comparison of registration results between this invention and other network structures

[0131]

[0132] The results above show that the SAFR module increases the network's receptive field by calculating the attention weights between any two locations in the image sequence, enhancing its ability to capture global features. Combined with the Local Detail Enhancement Network (LDF), it effectively reduces registration errors and improves registration accuracy. Furthermore, the SAFR module is a lightweight network, not introducing significant additional registration execution time, and exhibits good execution efficiency. In contrast, GAM, CBAM, and SENet rely on convolution and pooling operations to acquire global information, failing to effectively capture long-distance dependencies in the image. This limitation in global information acquisition results in a less significant improvement in model registration performance.

[0133] b. Local Detail Enhancement Module (LDEM), based on residual dense connections, such as... Figure 12 As shown, this module is used to compensate for details such as texture that may be lost during global feature extraction (downsampling). It adopts a Residual Dense Block (RDB) structure, which contains multiple densely connected convolutional layers, and can effectively extract and fuse local features, enhancing texture, edge, and local gradient information.

[0134] c. Multi-Scale Feature Fusion (MSFF) module: This module fuses LDEM and GCM features, taking into account both global structure and local details. It is the key to improving the accuracy of parameter prediction.

[0135] Ultimately, global features and local detail features are combined through residual connections to generate multi-scale features for predicting transformation parameters.

[0136] Physically-Decoupled Regression Head (PDRH) (Grouped Regression)

[0137] To address the differences in dimensions, gradients, and distributions of rotation and translation parameters, this invention designs a Physical Decoupling Regression Head (PDRH) at the generator output.

[0138] Unlike traditional single-head regression structures, PDRH divides the six rigid body attitude parameters into three subgroups: large-angle rotation, small-angle rotation, and translation. Each subgroup uses an independent fully connected layer and a different activation function (Pitanh series), thus explicitly embedding physical scale constraints in the output layer. This structure achieves separation of physical dimensions, gradient balancing, and output range control at the network layer level, significantly improving the stability and accuracy of attitude prediction.

[0139] Design Principles and Advantages Analysis

[0140] Physical scale decoupling

[0141] The numerical dimensions and gradient distributions of rotation and translation differ greatly:

[0142] Rotation: typically within ±π / 4 (approximately 45°);

[0143] Translation: within ±10~20 pixels or millimeters;

[0144] If a uniform regression is used, the rotation dominates the gradient, and the translation is almost completely masked.

[0145] Grouped regression, through hierarchical output and independent weight matrices, ensures that the loss gradient of each group is backpropagated independently, preventing mutual interference. To further illustrate the differences between the grouped regression of this invention and existing technologies, Table 4 is provided:

[0146] Table 4 System Comparison of the Invention with Common Regression Methods

[0147]

[0148] Existing generative adversarial networks (GANs) generate images that are essentially just mappings between image pixels. In contrast, the generator of this invention, which generates parameters, is essentially a geometric generator, learning a mapping from the image domain to the pose space.

[0149] 2. Discriminator Network

[0150] like Figure 13 As shown, the goal of the discrimination network is to distinguish between real and fake images. It consists of five convolutional layers and one fully connected layer. The network takes a real intraoperative 2D TRUS image or an image generated by the registration network as input. After feature extraction through multiple convolutional layers, a fully connected layer with a sigmoid activation function outputs a probability value in the range [0,1]. This value represents the probability that the input image is judged as a real image.

[0151] To address the registration problem in prostate ultrasound images, the discriminator of this invention employs five convolutional layers and one fully connected layer. The convolutional layers use a 3×3 kernel with a stride of 2. The fully connected layer is followed by a Sigmoid activation function. Multi-level convolutional stacking extracts features from the input image, which is gradually downsampled from 128×128 to 4×4, constructing a pyramid-shaped feature hierarchy to capture multi-scale differences and take into account both overall alignment and local misalignment features.

[0152] Unlike traditional GAN ​​discriminators:

[0153] A. Conventional discriminators rely solely on shallow convolutions to determine whether an image is real, primarily focusing on texture.

[0154] B. The discriminator of this invention encodes the "hierarchical nature of geometric structure" into the feature space by stacking five layers of pyramids.

[0155] C. Enables the model to distinguish between different types of errors such as "global deflection", "local offset", and "overall misalignment".

[0156] The fully connected layer uses patch-level output (the fully connected layer's role is related to the feature map), scoring each position on a 4×4×512 feature map. The discriminator independently evaluates the authenticity of each local region → the "PatchGAN" concept. The advantage of this approach is that it can not only determine the "true or false" of the entire image (using traditional methods), but also pinpoint "where it is aligned and where it is offset," representing an innovative transfer of GAN discriminative thinking to "spatial geometry tasks."

[0157] The fundamental difference between the discriminator in this invention and other GAN networks lies in that it does not aim for "the realism of the generated image," but rather for the structural rationality and alignment accuracy of the generated pose parameters. It does not judge whether the texture is realistic, but rather determines whether the geometric structure of the predicted image is consistent with that of the reference image, thereby improving the geometric accuracy of rigid registration.

[0158] Step 4: Loss Function and Model Optimization

[0159] To effectively guide network training, this invention designs a joint loss function. This function consists of a weighted average of the registration network loss and the discrimination network loss.

[0160] 1. Overall Loss Function

[0161] Total loss function The expression is:

[0162] ;

[0163] It is the loss function of the registration network.

[0164] It is the loss function of the discrimination network.

[0165] λ is a hyperparameter used to balance the weights of the two loss functions in the total loss.

[0166] 2. Registering the network loss function

[0167] The registration network's task is to accurately predict six spatial transformation parameters. Therefore, its loss function... Loss due to rotation angle and displacement distance loss Both losses are calculated using the mean squared error loss function. The expression is:

[0168] ;

[0169] in, It is a hyperparameter used to balance rotational and displacement losses.

[0170] Mean squared error loss The formula is:

[0171] ;

[0172] n is the number of samples.

[0173] It is the true label value of the i-th sample (i.e., the true rotation angle or displacement distance).

[0174] This is the network's predicted value for the i-th sample. This function measures the prediction accuracy of the registration network by calculating the sum of squared differences between the predicted and true values.

[0175] 3. Identify the network loss function

[0176] The identification network performs a binary classification task (true or false), therefore the binary cross-entropy loss function is used for calculation. D The formula is:

[0177]

[0178] n is the number of samples.

[0179] It is the label of the i-th image, with a value of 1 (real image) or 0 (generated image).

[0180] This is the output of the discriminator network for the i-th image, representing the probability that the image is predicted to be a real image (labeled 1). This function is used to measure the discriminator's ability to distinguish between real and fake images.

[0181] Step 5: Experimental Verification

[0182] To verify the effectiveness of the method of the present invention, comparative experiments were designed. For example... Figure 14As shown, the registered images of this invention are compared with existing CGAN, SAGAN, WGAN, and reference images. The reference image column contains four different actual 2D TRUS prostate images. Columns 2 to 5 show the corresponding registered images generated from the reference images in the corresponding rows, comparing the registered images of this invention with existing CGAN, SAGAN, and WGAN registered images. The green boxes represent the prostate region, and the white spots within them represent calcified lesions. The comparison results are shown in Table 5.

[0183] Table 5. Comparison of registration performance between the present invention and CGAN, SAGAN, and WGAN networks.

[0184]

[0185] As can be seen from Table 5, compared with CGAN, SAGAN, and WGAN, this invention has advantages in rotation parameters. The accuracy was reduced by 8.04°, 5.85°, and 5.68° respectively, and the RMSE of the displacement parameters was reduced by 0.40mm, 0.34mm, and 0.19mm respectively. In terms of MAE for the six registration parameters, the present invention reduced the accuracy by 3.92, 3.34, and 2.96 compared to GAN, SAGAN, and WGAN respectively. Regarding registration time, the present invention was only slightly longer than SAGAN by 3.06ms, while the difference in registration time compared to CGAN and WGAN was not significant, only 2.77ms and 2.02ms longer respectively. The 30.30ms registration time is sufficient to meet the surgical needs of surgeons. The above qualitative analysis shows that the registration effect of the present invention is superior to the compared methods.

[0186] As can be clearly seen from the comparison images (the area within the box is the main comparison area), the registration image generated by the method of this invention is closest to the real intraoperative image used as a reference in both overall structure and local morphology, which is significantly better than other comparison methods.

[0187] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the technical spirit of the present invention. The technical scope of the present invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

[0188] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A GAN-based method for TRUS prostate biopsy motion compensation, characterized in that, Includes the following steps: Acquire intraoperative 2D TRUS images and preoperative 3D TRUS images; The intraoperative 2D TRUS image is input into a pre-trained registration network, which is trained as a generator in a generative adversarial network. The training process of the registration network includes: The registered images generated by the registration network are used as negative examples, and the intraoperative 2D TRUS images are used as positive examples. Positive and negative samples are input together into the discrimination network in the generative adversarial network for authenticity determination; Based on the discrimination results of the discrimination network, the parameters of the registration network and the discrimination network are optimized through backpropagation; The registration network predicts and outputs the spatial transformation parameters between the intraoperative 2D TRUS image and the preoperative 3D TRUS image based on the features of the intraoperative 2D TRUS image. The spatial transformation parameters are applied to the preoperative 3D TRUS image to achieve motion compensation for the prostate. The registration network includes a multi-scale feature extraction module, which consists of a global feature extraction module, a local detail enhancement module, and a multi-scale feature fusion module. The global feature extraction module integrates a self-attention feature redistribution (SAFR) module. The SAFR module generates query vectors, key vectors, and value vectors through 1×1 convolution, calculates the global correlation matrix between spatial locations, and adaptively controls the intensity of attention mapping using a learnable parameter to fuse the weighted features with the original features. The steps for predicting spatial transformation parameters are to use a physical decoupling regression head to divide the spatial transformation parameters into at least three subgroups: large-angle rotation, small-angle rotation, and translation distance. The parameters of each subgroup are predicted by regression through an independent fully connected layer to decouple the gradients between parameters of different physical dimensions.

2. The method according to claim 1, characterized in that, The formula for calculating the global correlation matrix between spatial locations using the SAFR module is: β = softmax(QK ) , Where Q is the query vector, K is the key vector, and T is the matrix transpose; The intensity of the attention mapping is controlled by a learnable parameter γ1, and the final output feature Y is: Y = γ1·(βV) + X, Where X is the input feature before mapping, and V is the value vector.

3. The method according to claim 1, characterized in that, It also includes data augmentation by performing translation, flipping, and rotation operations on 2DTRUS images before training the registration network to enrich the dataset.

4. The method according to claim 1, characterized in that, The local detail enhancement module adopts a residual dense block structure, which contains multiple densely connected convolutional layers.

5. The method according to claim 1, characterized in that, The training process also includes: the discrimination network determines the geometric consistency between the registered image and the intraoperative 2D TRUS image, and the error signal is passed back to the registration network.

6. The method according to claim 5, characterized in that, The training process is guided by a joint loss function, which is a weighted sum of the registration network loss function and the discriminant network loss function.

7. The method according to claim 6, characterized in that: The registration network loss function consists of the rotation angle loss and displacement distance loss calculated based on the mean square error loss function, and is expressed as: ,in Let the rotation angle loss function be... The displacement distance loss function, For hyperparameters; Both loss functions are calculated using the mean squared error loss function, as shown in the following formula: in, It is the true label value of the i-th sample. This is the predicted value of the i-th sample, expressed as a rotation angle or displacement distance. Indicates the first in the data One sample; The loss function for the discrimination network is calculated using the binary classification cross-entropy loss function, expressed as: in The label value of the i-th image is represented by the set of values. , This represents the probability that the label value of the i-th image is predicted to be 1, where n is the sample size.

8. The method according to claim 1, characterized in that, The discriminant network consists of five convolutional layers and one fully connected layer. The fully connected layer uses patch-level output and scores each position on a 4×4×512 feature. The convolutional layers use 3×3 kernels with a stride of 2. The fully connected layer is followed by a sigmoid activation function.

9. The method according to claim 1, characterized in that, The global feature extraction module consists of four convolutional layers, two max pooling layers, an upsampling layer, a batch normalization (BN) layer, and a self-attention mechanism module.

Images