MRI-TRUS image registration method and device based on weak supervised learning

Abstract

The invention provides an MRI-TRUS image registration method and device based on weak supervised learning, and the method comprises the following steps: firstly, obtaining a to-be-registered MRI image and a to-be-registered TRUS image, inputting the to-be-registered MRI image and the to-be-registered TRUS image into a pre-trained segmentation sub-network, and obtaining a corresponding mask; and inputting the masks and the original image into a pre-trained registration sub-network to obtain a registration result. The registration sub-network comprises a transformation field and a registration unit, the transformation field unit generates a dense transformation field, the registration unit takes the MRI mask and the image as a moving object and takes the TRUS mask and the image as a fixed object, and the moving object is registered into the fixed object according to the transformation field. A to-be-registered MRI mask and a to-be-registered TRUS mask are obtained by segmenting a sub-network, and then accurate registration is carried out in a registration sub-network by taking the to-be-registered MRI mask and the to-be-registered TRUS mask as guidance.

Description

Technical Field

[0001] The present application relates to the field of medical image processing, and in particular to a MRI-TRUS image registration method and device based on weakly supervised learning. Background Art

[0002] Prostate cancer is one of the most common malignant tumors in men worldwide. Its early and accurate diagnosis is crucial to reducing mortality and improving patient prognosis. Currently, biopsy is still the gold standard for confirming prostate cancer, and transrectal ultrasound (TRUS) provides real-time image guidance during the biopsy process. However, due to the limitations of TRUS images in tissue resolution, it is difficult to accurately distinguish between normal and cancerous tissues, resulting in limited diagnostic accuracy. Multi-parameter magnetic resonance imaging (mpMRI) is widely used in the detection and localization of prostate cancer due to its superior soft tissue contrast. However, MRI is mainly used for preoperative image evaluation and cannot be used in real time during surgery, which limits its role in biopsy guidance.

[0003] Therefore, in order to better distinguish normal and cancerous tissues during surgery, MRI and TRUS are often registered. Currently, MRI and TRUS are usually registered manually, which takes a lot of time and effort and easily causes waste of medical resources, especially in high-load clinical environments. If automatic registration is performed through artificial intelligence models, a large amount of labeled data is required, and these labeled data often need to be manually labeled by doctors, which is time-consuming and costly.

[0004] In summary, how to reduce the dependence on labeled data so as to quickly and accurately perform real-time registration of MRI images and TRUS images is a problem that needs to be urgently solved in the existing technology. Summary of the invention

[0005] The embodiment of the present application provides an MRI-TRUS image registration method and device based on weakly supervised learning, which obtains the MRI mask to be registered and the TRUS mask to be registered through a segmentation subnetwork, and then performs precise registration in the registration subnetwork with the MRI mask to be registered and the TRUS mask to be registered as guidance.

[0006] In a first aspect, an embodiment of the present application provides an MRI-TRUS image registration method based on weakly supervised learning, the method comprising: Acquire an MRI image to be registered and a TRUS image to be registered, and input the MRI image to be registered and the TRUS image to be registered into a pre-trained segmentation subnetwork to obtain an MRI mask to be registered and a TRUS mask to be registered; The MRI mask to be registered, the TRUS mask to be registered, the MRI image to be registered and the TRUS image to be registered are input into a pre-trained registration subnetwork to obtain a registration result, wherein the registration subnetwork is composed of a transformation field unit and a registration unit, and the transformation field unit obtains a dense transformation field based on the MRI mask to be registered, the TRUS mask to be registered, the MRI image to be registered and the TRUS image to be registered, and the MRI mask to be registered is used as a mobile label, the MRI image to be registered is used as a mobile image, the TRUS mask to be registered is used as a fixed label, and the TRUS image to be registered is used as a fixed image in the registration unit, and the mobile label and the mobile image are deformed based on the dense transformation field so that the mobile label and the mobile image are registered to the fixed label and the fixed image.

[0007] In a second aspect, an embodiment of the present application provides an MRI-TRUS image registration device based on weakly supervised learning, comprising: An acquisition module is used to acquire the MRI image to be registered and the TRUS image to be registered, and input the MRI image to be registered and the TRUS image to be registered into a pre-trained segmentation subnetwork to obtain the MRI mask to be registered and the TRUS mask to be registered; A registration module is used to input the MRI mask to be registered, the TRUS mask to be registered, the MRI image to be registered and the TRUS image to be registered into a pre-trained registration subnetwork to obtain a registration result, wherein the registration subnetwork consists of a transformation field unit and a registration unit, and the transformation field unit obtains a dense transformation field based on the MRI mask to be registered, the TRUS mask to be registered, the MRI image to be registered and the TRUS image to be registered. In the registration unit, the MRI mask to be registered is used as a mobile label, the MRI image to be registered is used as a mobile image, the TRUS mask to be registered is used as a fixed label, and the TRUS image to be registered is used as a fixed image, and the mobile label and the mobile image are deformed based on the dense transformation field so that the mobile label and the mobile image are registered to the fixed label and the fixed image.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute an MRI-TRUS image registration method based on weakly supervised learning.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process, and the process includes an MRI-TRUS image registration method based on weakly supervised learning.

[0010] The main contributions and innovations of the present invention are as follows: In the embodiment of the present application, a segmentation subnetwork is used to obtain an MRI mask to be registered and a TRUS mask to be registered. In the segmentation subnetwork, the SE module is used to adaptively adjust the key channel features of the feature map, suppress unimportant channels, and enhance the focus on the prostate ROI. The self-attention mechanism is used to capture the long-distance dependency of the image, reduce noise interference, and improve the segmentation accuracy of the subtle area. The feature information of different levels is fused through jump connections to avoid feature loss. The transformation field unit in the registration subnetwork is connected in series through a downsampling module and an upsampling module, and the number of downsampling layers is equal to that of upsampling layers and jump connections are performed. The downsampling adopts residual connection and the upsampling adopts trilinear interpolation, so as to accurately capture the key features of different scales and generate a dense transformation field. The downsampling layer and the upsampling layer can accurately capture the key features at different scales under the guidance of the MRI mask to be registered and the TRUS mask to be registered, thereby improving the registration accuracy, helping the registration subnetwork to focus on the region of interest, and further improving the accuracy and efficiency of the registration.

[0011] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flow chart of an MRI-TRUS image registration method based on weakly supervised learning according to an embodiment of the present application; Figure 2 is a schematic diagram of the structure of a segmented sub-network according to an embodiment of the present application; Figure 3 is a schematic diagram of the performance results of a pre-trained segmentation sub-network on a clinical data set according to an embodiment of the present application; Figure 4 is a structural schematic diagram of a field conversion unit according to an embodiment of the present application; Figure 5 is an overall structural diagram of a registration sub-network according to an embodiment of the present application; Figure 6 is a schematic diagram of the clinical performance results of a registration sub-network according to an embodiment of the present application; Figure 7 is a structural block diagram of an MRI-TRUS image registration device based on weakly supervised learning according to an embodiment of the present application; Figure 8It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0013] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0014] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0015] Embodiment 1 The embodiment of the present application provides an MRI-TRUS image registration method based on weakly supervised learning, which obtains the MRI mask to be registered and the TRUS mask to be registered through a segmentation subnetwork, and then uses the MRI mask to be registered and the TRUS mask to be registered as guidance to perform accurate registration in the registration subnetwork. Specifically, refer to Figure 1 , the method comprising: Acquire an MRI image to be registered and a TRUS image to be registered, and input the MRI image to be registered and the TRUS image to be registered into a pre-trained segmentation subnetwork to obtain an MRI mask to be registered and a TRUS mask to be registered; The MRI mask to be registered, the TRUS mask to be registered, the MRI image to be registered and the TRUS image to be registered are input into a pre-trained registration subnetwork to obtain a registration result, wherein the registration subnetwork is composed of a transformation field unit and a registration unit, and the transformation field unit obtains a dense transformation field based on the MRI mask to be registered, the TRUS mask to be registered, the MRI image to be registered and the TRUS image to be registered, and the MRI mask to be registered is used as a mobile label, the MRI image to be registered is used as a mobile image, the TRUS mask to be registered is used as a fixed label, and the TRUS image to be registered is used as a fixed image in the registration unit, and the mobile label and the mobile image are deformed based on the dense transformation field so that the mobile label and the mobile image are registered to the fixed label and the fixed image.

[0016] In some specific embodiments, multiple MRI images and paired TRUS images as well as a variety of clinical imaging data collected from a hospital are obtained from a public data set as a training data set, the training data set is first preprocessed, and then the preprocessed training data set is used to train the segmentation subnetwork to obtain a pretrained segmentation subnetwork.

[0017] Specifically, this scheme first collected prostate imaging data from public data sets, including 48 MRI images from the MSDProstate data set, 50 MRI images from the Promise12 data set, and 73 pairs of MRI-TRUS paired images from the µ-RegPro data set. In addition, 82 pairs of MRI-TRUS clinical imaging data were collected from a hospital to ensure the reliability of the model in practical applications. When screening the clinical data, 50 cases of data that did not meet the requirements were eliminated. The reasons for these data exclusion included the lack of T2 sequences, artifacts in MRI or TRUS images, and missing or mismatched segmentation labels drawn by doctors. Finally, 32 high-quality clinical paired images were retained for subsequent experiments. In terms of data set division, first, the training data for the automatic prostate region segmentation model included 136 MRI and 65 TRUS images selected from public data sets, and 35 MRI and 8 TRUS images were used for testing; then, in the training and verification of the MRI-TRUS registration model, 65 pairs of paired images were selected from the µ-RegPro data set for training, and 8 pairs of paired images and 32 pairs of clinical images were used for testing; finally, in the experiment to evaluate the effect of MRI-TRUS registration on clinical diagnosis improvement, 32 pairs of registered clinical images were used to compare and analyze the diagnostic accuracy and confidence of doctors through the PI-RADS scoring system.

[0018] Specifically, the training data set is preprocessed by format conversion, size cropping, standardization and data enhancement. In order to reduce redundant information and improve the training efficiency of the segmentation sub-network, the upper and lower 10% slices of the MRI image and the first and last 25% and 12.5% ​​frames of the TRUS image are cropped. In order to improve the generalization ability of the segmentation sub-network, rotation, balancing, scaling and other methods are used for data enhancement.

[0019] In some embodiments, the structure of the segmentation subnetwork is as follows: Figure 2 As shown, the segmentation subnetwork includes a coding region and a decoding region with a symmetrical structure. The coding region is connected in series by a first coding unit and a plurality of second coding units, and the decoding region is connected in series by a plurality of first decoding units and a second decoding unit, and each second coding unit is jump-connected to a first decoding unit of a corresponding scale.

[0020] Specifically, the segmentation subnetwork is trained using training samples in a training data set, where the training samples are MRI images or TRUS images with prostate regions labeled, and a segmentation loss is constructed. When the segmentation loss meets a set condition or reaches an iteration round, the training is completed to obtain a pre-trained segmentation subnetwork.

[0021] Specifically, the segmentation subnetwork in this scheme uses the SE module and the self-attention mechanism to enhance the model's ability to capture long-range dependent features, and fuses feature information at different levels through jump connections, so that shallow detail features are combined with deep semantic features, avoiding the problem of feature loss caused by increased network depth, helping to improve the accuracy of segmentation and making the segmentation results more accurate in distinguishing details such as edges and overall objects, so as to achieve efficient prostate ROI extraction and guide subsequent image registration.

[0022] Furthermore, the first encoding unit is a separate encoder, the second encoding unit is an encoder and an SE module connected in series, the first decoding unit is a self-attention module and a decoder connected in series, and the second decoding unit is a separate decoder.

[0023] Specifically, compared with manual outlining by doctors, obtaining the MRI mask and TRUS mask to be registered through the pre-trained segmentation sub-network has stronger robustness and consistency, and reduces the time and cost of manual labeling, avoids human errors and inconsistent labeling, thereby improving the stability and accuracy of subsequent image registration.

[0024] Specifically, the SE module extracts the global information of each channel through the global average pooling operation and generates adaptive weights for each channel, thereby weighting the important channels, so that the segmentation sub-network pays more attention to the prostate ROI during the learning process and avoids the interference of irrelevant features.

[0025] The SE module first performs global average pooling on each channel of the output feature map through a squeeze operation to obtain a global descriptor for each channel. , Indicates the global information of the corresponding channel, get The formula is as follows:

[0026] in, is the position in the input feature map Channel The characteristic value of and are the height and width of the feature map, respectively. Then, in the Excitation operation, the SE module generates the attention weight of each channel through two fully connected layers. , which controls the contribution of each channel to the final feature, generating The formula is as follows:

[0027] in, and is the weight matrix of the fully connected layer, is the ReLU activation function, Sigmoid activation function is used to ensure the weight of each channel Between 0 and 1. Finally, we get the weight of each channel Finally, the SE module applies these weights to the features of each channel in the original feature map through a recalibration operation to adjust the importance of the channel. The specific operations are as follows:

[0028] in, is the channel when it is not adjusted, For each channel's importance, This adjusts the channel weights after importance.

[0029] This scheme uses the SE module to adaptively adjust the features of key channels in the feature map and suppress unimportant channels, thereby improving the expressiveness of the features. The SE module not only enhances the robustness of the network when processing MRI images and TRUS images, but also effectively improves the segmentation accuracy of complex areas such as the prostate and tumors, providing more accurate segmentation labels for subsequent registration tasks, thereby ensuring the accuracy of the registration process.

[0030] Specifically, the self-attention mechanism in this scheme enables the segmentation sub-network to capture the long-distance dependencies in MRI images and TRUS images. Especially when dealing with areas with blurred boundaries or complex backgrounds, the segmentation sub-network's focus area can be dynamically adjusted. Through the self-attention mechanism, the model can effectively reduce noise interference during the segmentation process and improve the segmentation accuracy of subtle areas such as tumors and prostates. In the self-attention mechanism, first, the input feature map is Calculate the query, key, and value, where N is the number of features and d is the dimension of each feature. The query, key, and value calculation formulas are:

[0031] in, are weight matrices used to generate query Q, key K, and value V, respectively.

[0032] Then the attention weight is obtained by calculating the similarity between the query and the key, that is, the inner product of the query and the key is calculated by the dot product, and the weight distribution is obtained by the Softmax function:

[0033] Finally, use the obtained attention weights to perform weighted averaging on V to obtain the weighted output:

[0034] Through the self-attention mechanism, the segmentation subnetwork can adaptively allocate attention according to the relationship between different input features, focusing on areas with strong correlation, that is, the difference between the prostate and the tumor, while ignoring irrelevant parts. In practical applications, the self-attention mechanism can dynamically adjust the weights of different parts of the features to effectively capture long-distance dependencies and global contextual information. For example, when processing MRI images, the attention can be automatically adjusted according to the lesion area in the image to ensure that the information of key areas such as the prostate and tumors receives more attention. When processing images with blurred boundaries or complex details, the recognition ability of these areas can be improved, thereby better completing the image segmentation.

[0035] For example, the performance results of the pre-trained segmentation subnetwork in this scheme on the clinical dataset are as follows: Figure 3 As shown, Figure 3 In the figure, 1-4 represent four cases of MRI image data, 5-8 represent four cases of TRUS image data, (a) is the original image input into the segmentation sub-network, (b) is the segmentation mask drawn by a professional radiologist, and (c) is the segmentation mask automatically output by the segmentation sub-model.

[0036] The test set is used to test the pre-trained segmentation sub-network. The test results are shown in Table 1: Table 1 Test results of segmentation sub-network

[0037] Table 1 shows the performance evaluation results of the segmentation subnetwork on different datasets, showing the stable performance of the model in each dataset. In the public MRI dataset, the DSC of the model reached 0.9154 and the accuracy (Acc) was 0.9914, indicating that it has high accuracy in the MRI image segmentation of prostate structure. Although the performance on the clinical dataset was slightly reduced, with a DSC of 0.9010 and an Acc of 0.9753, the segmentation performance of the model in actual clinical applications is still relatively excellent. In the segmentation task of TRUS images, the model also performed well. The DSC in the public dataset was 0.9384 and the Acc was 0.9828, which fully demonstrated its segmentation ability in TRUS images. In the clinical dataset, although the accuracy was reduced, with a DSC of 0.9173 and an Acc of 0.9522, the model still showed strong adaptability.

[0038] From the perspective of the HD95 metric for boundary segmentation, no significant statistical difference was found between the public MRI dataset and the clinical MRI dataset (public MRI vs. clinical MRI, p = ns). However, in TRUS images, the performance of the public dataset was significantly better than that of the clinical dataset (public TRUS vs. clinical TRUS, p = 0.007). Similarly, in terms of the HD95 metric, there was no significant difference between MRI images and TRUS images in the public dataset (public MRI vs. public TRUS, p = ns), but in the clinical dataset, the performance of MRI images was significantly better than that of TRUS images (clinicalMRI vs. clinical TRUS, p<0.001). Overall, the model showed good robustness and cross-scenario adaptability in different datasets and imaging modes.

[0039] In some embodiments, the MRI mask and TRUS mask output by the pre-trained segmentation sub-network are used as guiding information to combine the corresponding MRI images and TRUS images in the training data set to perform weak supervision training on the registration sub-network.

[0040] Specifically, since the amount of registration training sample data for MRI images and TRUS images is small and there are individual differences between different patients, this scheme adopts a weakly supervised training method to train the registration subnetwork. Weak supervision can reduce the requirements for labeling accuracy, and only relatively little labeling information is required for registration, saving time and labor costs. At the same time, it can also improve the generalization ability of the registration subnetwork, so that the registration subnetwork will not be overly dependent on the precise labeling mode, and can learn more general registration features and rules. When facing individual differences among different patients and differences in equipment imaging parameters, it is easier to find a suitable registration strategy.

[0041] In some embodiments, the structural diagram of the field conversion unit is as follows: Figure 4 As shown, the transform field unit is connected in series by a down-sampling module and an up-sampling module, the down-sampling module includes a plurality of down-sampling layers, the up-sampling module includes a plurality of up-sampling layers, and the number of the down-sampling layers is equal to the number of the up-sampling layers.

[0042] Specifically, the number of downsampling layers and upsampling layers in this scheme is four. In the registration network, the downsampling layers and upsampling layers can accurately capture the key features at different scales under the guidance of the MRI mask to be registered and the TRUS mask to be registered, thereby improving the registration accuracy and helping the registration subnetwork to focus on the region of interest, further improving the accuracy and efficiency of the registration.

[0043] Furthermore, each of the down-sampling layers is jump-connected to an up-sampling layer of a corresponding scale, and the down-sampling layer is down-sampled by a residual connection, and the up-sampling layer is up-sampled by a trilinear interpolation.

[0044] Specifically, the skip connection can effectively transfer the detail information lost in the downsampling process to the upsampling stage, so that this information can be used to better restore the details of the image during upsampling, which helps to retain the key features in the original image and avoid feature loss or blurring after multiple downsampling and upsampling.

[0045] Specifically, the downsampling layer uses residual connections for downsampling, which can effectively alleviate the gradient vanishing or gradient exploding problems caused by downsampling. By adding the input information and the downsampled information, the network can more easily learn the residual part after downsampling, thereby improving the training stability and convergence speed of the network.

[0046] Specifically, the upsampling layer uses trilinear interpolation to perform upsampling to reasonably estimate the value of the newly inserted pixel based on the values ​​of the surrounding pixels. While ensuring computational efficiency, it can better enlarge the image, thereby combining it with the detail information transmitted by the jump connection to more accurately restore the features and structure of the original image.

[0047] In some specific embodiments, the overall structure of the registration subnetwork is as follows: Figure 5 As shown, in the registration subnetwork, the dense transformation field output by the transformation field unit is a transformation matrix that transforms the moving image and the moving label into the fixed image and the fixed label in the image space.

[0048] Specifically, in the registration subnetwork, the MRI mask to be registered and the TRUS mask to be registered are combined with the corresponding image itself to better capture the key features in the image, thereby improving the registration accuracy. That is to say, in the registration subnetwork, the MRI mask to be registered and the TRUS mask to be registered are used as guiding information to help the registration subnetwork focus on the region of interest, thereby further improving the accuracy and efficiency of the registration.

[0049] Furthermore, the dense transformation field in this scheme is used to optimize multiple surprises, effectively handle anatomical differences in multimodal images, and ensure accurate registration of MRI images and TRUS images.

[0050] Specifically, the mobile labels and the mobile images are deformed by dense transformation fields so as to be aligned with the corresponding fixed labels and the fixed images to complete the registration.

[0051] In some embodiments, when weakly training the registration subnetwork, segmentation consistency loss, boundary error loss and target registration error loss are used as loss functions, wherein the segmentation consistency loss is used to evaluate the degree of overlap between the deformed moving label and the fixed label, the boundary error loss is used to measure the maximum deviation of the boundary points between the deformed moving label and the fixed label, and the target registration error loss is used to measure the spatial alignment error of the key points between the moving image and the fixed image.

[0052] Specifically, the formula of the segmentation consistency loss is expressed as follows:

[0053] Among them, A is the deformed mobile label, and B is the fixed label. is the segmentation consistency loss, which is used to evaluate the degree of overlap between the deformed moving label and the fixed label.

[0054] Specifically, the formula of the boundary error loss is expressed as follows:

[0055] Among them, A is the deformed mobile label, and B is the fixed label. is the boundary error loss, HD95 represents the 95% quantile of the Hausdorff distance, and the boundary error loss is used to measure the maximum deviation of the boundary points between the mobile label and the fixed label after deformation.

[0056] Specifically, the formula of the target registration error loss is expressed as follows:

[0057] Where N is the total number of key points, represents the key points in the moving image, represents the key points in a fixed image, represents the target registration error loss, which is used to measure the spatial alignment error of key points between the moving image and the fixed image.

[0058] Furthermore, in this solution, the segmentation consistency loss, the boundary error loss and the target registration error loss are weightedly summed to obtain a multi-scale loss, and the parameters of the registration subnetwork are adjusted based on the multi-scale loss.

[0059] Specifically, the formula of multi-scale loss is expressed as:

[0060] in, is the multi-scale loss, is the weight of the segmentation consistency loss, is the weight of the boundary error loss, is the weight of the target registration error loss.

[0061] Exemplarily, in this embodiment, is 0.5, is 0.25, is 0.25.

[0062] Specifically, the registration subnetwork in this scheme uses the MRI mask to be registered and the TRUS mask to be registered as segmentation labels to provide more structural information, so that the registration subnetwork can better focus on and align the regions of interest such as the prostate or tumor, thereby improving the registration accuracy. Especially in the registration tasks of complex anatomical structures or lesion areas, the possibility of misregistration can be significantly reduced.

[0063] Specifically, Table 2 shows the evaluation indicators of the registration sub-model in the test dataset.

[0064] Table 2 Evaluation indicators of the registration sub-model in the test dataset

[0065] Table 2 involves six key indicators: DSC (Dice coefficient), HD95 (95% Hausdorff distance), Acc (accuracy), Pre (precision), Rec (recall rate) and TRE (target registration error). On the public dataset, the model's DSC is 0.7481, HD95 is 10.1839, Acc is 0.9313, Pre is 0.7444, Rec is 0.7583, and TRE is 7.8562, showing that the model has a high registration accuracy on this dataset, especially in DSC and Acc indicators, which fully demonstrates its ability to accurately segment and register the prostate region. In contrast, the model's DSC on the clinical dataset is 0.7295, HD95 is 11.1766, Acc is 0.8760, Pre is 0.8813, Rec is 0.6412, and TRE is 8.6307. Although there was no statistically significant difference between the public dataset and the clinical dataset in the three key indicators of DSC, HD95 and TRE, the clinical dataset was significantly better than the public dataset in the Pre indicator (p<0.001), reflecting the robustness of the model in the clinical environment and higher registration accuracy. These results further verify the good generalization ability and adaptability of the model in complex clinical scenarios, indicating that it can continue to provide a high level of segmentation and registration accuracy in actual clinical applications.

[0066] Exemplarily, the clinical performance results of the registration subnetwork are as follows: Figure 6 As shown, in Figure 6 In the figure, (a) is the original MRI image, (b) is the original TRUS image, and (c) is the standardized MRI image. 1 and 2 represent two different cases.

[0067] Embodiment 2 Based on the same idea, refer to Figure 7 , the present application also proposes an MRI-TRUS image registration device based on weakly supervised learning, comprising: An acquisition module is used to acquire the MRI image to be registered and the TRUS image to be registered, and input the MRI image to be registered and the TRUS image to be registered into a pre-trained segmentation subnetwork to obtain the MRI mask to be registered and the TRUS mask to be registered; A registration module is used to input the MRI mask to be registered, the TRUS mask to be registered, the MRI image to be registered and the TRUS image to be registered into a pre-trained registration subnetwork to obtain a registration result, wherein the registration subnetwork is composed of a transformation field unit and a registration unit, and the transformation field unit obtains a dense transformation field based on the MRI mask to be registered, the TRUS mask to be registered, the MRI image to be registered and the TRUS image to be registered, and the MRI mask to be registered is used as a mobile label, the MRI image to be registered is used as a mobile image, the TRUS mask to be registered is used as a fixed label, and the TRUS image to be registered is used as a fixed image in the registration unit, and the mobile label and the mobile image are deformed based on the dense transformation field so that the mobile label and the mobile image are registered to the fixed label and the fixed image.

[0068] Embodiment 3 This embodiment also provides an electronic device, referring to Figure 8 , comprises a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above method embodiments.

[0069] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0070] Among them, the memory 404 may include a large capacity memory 404 for data or instructions. For example, but not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 404 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 404 may be inside or outside the data processing device. In a specific embodiment, the memory 404 is a non-volatile memory. In a specific embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. In appropriate circumstances, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM may be a fast page mode dynamic random access memory 404 (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0071] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .

[0072] The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement any one of the MRI-TRUS image registration methods based on weakly supervised learning in the above embodiments.

[0073] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .

[0074] The transmission device 406 can be used to receive or send data via a network. The specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0075] The input / output device 408 is used to input or output information. In this embodiment, the input information may be an MRI image to be registered and a TRUS image to be registered, etc., and the output information may be a registration result, etc.

[0076] Optionally, in this embodiment, the processor 402 may be configured to perform the following steps through a computer program: Acquire an MRI image to be registered and a TRUS image to be registered, and input the MRI image to be registered and the TRUS image to be registered into a pre-trained segmentation subnetwork to obtain an MRI mask to be registered and a TRUS mask to be registered; The MRI mask to be registered, the TRUS mask to be registered, the MRI image to be registered and the TRUS image to be registered are input into a pre-trained registration subnetwork to obtain a registration result, wherein the registration subnetwork is composed of a transformation field unit and a registration unit, and the transformation field unit obtains a dense transformation field based on the MRI mask to be registered, the TRUS mask to be registered, the MRI image to be registered and the TRUS image to be registered, and the MRI mask to be registered is used as a mobile label, the MRI image to be registered is used as a mobile image, the TRUS mask to be registered is used as a fixed label, and the TRUS image to be registered is used as a fixed image in the registration unit, and the mobile label and the mobile image are deformed based on the dense transformation field so that the mobile label and the mobile image are registered to the fixed label and the fixed image.

[0077] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0078] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0079] Embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, at this point, it should be noted that, for example, Figure 8 Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.

[0080] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0081] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A MRI-TRUS image registration method based on weakly supervised learning, characterized in that: The following steps are involved: Acquire an MRI image to be registered and a TRUS image to be registered, and input the MRI image to be registered and the TRUS image to be registered into a pre-trained segmentation subnetwork to obtain an MRI mask to be registered and a TRUS mask to be registered; The MRI mask to be registered, the TRUS mask to be registered, the MRI image to be registered and the TRUS image to be registered are input into a pre-trained registration subnetwork to obtain a registration result, wherein the registration subnetwork is composed of a transformation field unit and a registration unit, and the transformation field unit obtains a dense transformation field based on the MRI mask to be registered, the TRUS mask to be registered, the MRI image to be registered and the TRUS image to be registered, and the MRI mask to be registered is used as a mobile label, the MRI image to be registered is used as a mobile image, the TRUS mask to be registered is used as a fixed label, and the TRUS image to be registered is used as a fixed image in the registration unit, and the mobile label and the mobile image are deformed based on the dense transformation field so that the mobile label and the mobile image are registered to the fixed label and the fixed image.

2. The MRI-TRUS image registration method based on weakly supervised learning according to claim 1, characterized in that: The MRI mask and TRUS mask output by the pre-trained segmentation subnetwork are used as guiding information to combine the corresponding MRI images and TRUS images in the training data set to perform weak supervision training on the registration subnetwork.

3. The MRI-TRUS image registration method based on weakly supervised learning according to claim 2, characterized in that: When weakly training the registration subnetwork, segmentation consistency loss, boundary error loss and target registration error loss are used as loss functions, wherein the segmentation consistency loss is used to evaluate the overlap between the deformed moving label and the fixed label, the boundary error loss is used to measure the maximum deviation of the boundary points between the deformed moving label and the fixed label, and the target registration error loss is used to measure the spatial alignment error of the key points between the moving image and the fixed image.

4. The MRI-TRUS image registration method based on weakly supervised learning according to claim 1, characterized in that: The segmentation subnetwork is trained using training samples in the training data set, where the training samples are MRI images or TRUS images with prostate regions labeled, and segmentation loss is constructed. When the segmentation loss meets the set conditions or reaches the iteration round, the training is completed to obtain a pre-trained segmentation subnetwork.

5. The MRI-TRUS image registration method based on weakly supervised learning according to claim 1, characterized in that: The segmentation subnetwork includes an encoding region and a decoding region of a symmetrical structure, wherein the encoding region is composed of a first encoding unit and a plurality of second encoding units connected in series, and the decoding region is composed of a plurality of first decoding units and a second decoding unit connected in series, and each second encoding unit is jump-connected to a first decoding unit of a corresponding scale, wherein the first encoding unit is a separate encoder, the second encoding unit is a series-connected encoder and SE module, the first decoding unit is a series-connected self-attention module and decoder, and the second decoding unit is a separate decoder.

6. The MRI-TRUS image registration method based on weakly supervised learning according to claim 1, characterized in that: The transform field unit is connected in series by a down-sampling module and an up-sampling module, wherein the down-sampling module includes a plurality of down-sampling layers, and the up-sampling module includes a plurality of up-sampling layers, and the number of the down-sampling layers is equal to the number of the up-sampling layers.

7. The MRI-TRUS image registration method based on weakly supervised learning according to claim 6, characterized in that: Each of the down-sampling layers is jump-connected to an up-sampling layer of a corresponding scale, and the down-sampling layer is down-sampled by a residual connection, and the up-sampling layer is up-sampled by a trilinear interpolation.

8. An MRI-TRUS image registration device based on weakly supervised learning, characterized in that: include: An acquisition module is used to acquire the MRI image to be registered and the TRUS image to be registered, and input the MRI image to be registered and the TRUS image to be registered into a pre-trained segmentation subnetwork to obtain the MRI mask to be registered and the TRUS mask to be registered; A registration module is used to input the MRI mask to be registered, the TRUS mask to be registered, the MRI image to be registered and the TRUS image to be registered into a pre-trained registration subnetwork to obtain a registration result, wherein the registration subnetwork is composed of a transformation field unit and a registration unit, and the transformation field unit obtains a dense transformation field based on the MRI mask to be registered, the TRUS mask to be registered, the MRI image to be registered and the TRUS image to be registered, and the MRI mask to be registered is used as a mobile label, the MRI image to be registered is used as a mobile image, the TRUS mask to be registered is used as a fixed label, and the TRUS image to be registered is used as a fixed image in the registration unit, and the mobile label and the mobile image are deformed based on the dense transformation field so that the mobile label and the mobile image are registered to the fixed label and the fixed image.

9. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the MRI-TRUS image registration method based on weakly supervised learning according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, wherein the computer program includes a program code for controlling a process to execute a process, wherein the process includes an MRI-TRUS image registration method based on weakly supervised learning according to any one of claims 1-7.

Images