Multi-modal prostate image registration method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a multi-modal prostate image registration method, system and device and a medium. The method comprises the following steps: constructing a multi-modal prostate image registration model; the method comprises the following steps: collecting prostate TRUS data and MRI / CT data to be registered; and processing the to-be-registered prostate TRUS data and the to-be-registered MRI / CT data according to the multi-mode prostate image registration model to complete prostate image registration. According to the prostate image registration method, the defects existing in an existing prostate image registration scheme are overcome, the INPUT and the OUTPUT are expanded to be two, so that a multi-modal prostate image registration model can process multi-modal data at the same time, and a registration deformation field is output; the method combines TRUS prostate envelope segmentation and registration, not only greatly reduces calculation time, but also improves the accuracy of prostate image registration, and is suitable for popularization and application.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of prostate image registration, and in particular to a multimodal prostate image registration method, system, electronic device and storage medium. Background Art

[0002] As we all know, Transrectal Ultrasound (TRUS) is a medical imaging technique used to obtain images of the prostate and surrounding structures. It produces real-time 2D or 3D images of the prostate by inserting a small ultrasound probe into the rectum. Magnetic Resonance Imaging (MRI) uses strong magnetic fields, radiofrequency waves, and gradient magnetic fields to produce detailed images of structures inside the body. Computed Tomography (CT) uses X-rays and computer processing to produce detailed cross-sectional images of the internal structures of the human body.

[0003] In the field of prostate biopsy or treatment, since the location of the lesion is well visualized under MRI or CT, but difficult to distinguish under TRUS, it is often necessary to fused the TRUS and MRI / CT data after registration to help doctors distinguish the location of the lesion under TRUS. In the clinical operation process, the existing registration and fusion scheme is to first independently segment the prostate capsule of MRI / CT data and TRUS ultrasound data, perform rigid or non-rigid registration on the two segmented models, and then perform data fusion display based on the registration results.

[0004] However, due to the characteristics of ultrasound imaging, and the fact that the prostate of patients is often accompanied by pathological hypertrophy and invasion of surrounding tissues (such as seminal vesicles), TRUS data will make it difficult to segment the prostate capsule in ultrasound images, especially at the tip and bottom of the prostate. In addition, the segmentation and registration of the prostate are carried out in two steps, which takes a long time to calculate and is inefficient.

[0005] The above problems need to be solved urgently. Summary of the invention

[0006] In order to solve the related technical problems, the present invention provides a multimodal prostate image registration method, system, electronic device and storage medium to solve the problems mentioned in the above background technology part.

[0007] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0008] In a first aspect, an embodiment of the present invention provides a multimodal prostate image registration method, the method comprising:

[0009] Constructing a multimodal prostate image registration model; wherein the multimodal prostate image registration model processes prostate TRUS data and MRI / CT data simultaneously, and outputs TRUS segmentation results and registration deformation fields;

[0010] Collect the prostate TRUS data and MRI / CT data to be registered;

[0011] The prostate TRUS data and MRI / CT data to be registered are processed according to the multimodal prostate image registration model to complete the prostate image registration.

[0012] As an optional implementation, in the multimodal prostate image registration model, both INPUT and OUTPUT are expanded to two.

[0013] As an optional implementation, the content of the INPUT is set, which includes but is not limited to TRUS data and MRI / CT segmentation results.

[0014] As an optional implementation, the content of the OUTPUT is set, which includes but is not limited to the segmentation result of TRUS and the registration deformation field.

[0015] As an optional implementation, the multimodal prostate image registration model adopts UNET3D-MultiModal-Fusion, the INPUT is TRUS data and MRI / CT segmentation results; the OUTPUT content is the TRUS segmentation results and the registration deformation field.

[0016] In a second aspect, an embodiment of the present invention provides a multimodal prostate image registration system, which adopts the multimodal prostate image registration method described in any one of the first aspects above, including:

[0017] A model building module, used to build a multimodal prostate image registration model; wherein the multimodal prostate image registration model processes prostate TRUS data and MRI / CT data simultaneously, and outputs TRUS segmentation results and registration deformation fields;

[0018] A registration data acquisition module, used to acquire TRUS data and MRI / CT data of the prostate to be registered;

[0019] The prostate image registration module is used to process the prostate TRUS data and MRI / CT data to be registered according to the multimodal prostate image registration model to complete the prostate image registration.

[0020] As an optional implementation, the model building module expands both the INPUT and OUTPUT in the multimodal prostate image registration model into two.

[0021] As an optional implementation, the model building module is further used to set the TRUS data and the MRI / CT segmentation result as the content of INPUT, and set the TRUS segmentation result and the registration deformation field as the content of OUTPUT.

[0022] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory connected to the processor, wherein program data is stored in the memory, and the processor retrieves the program data stored in the memory to execute the multimodal prostate image registration method provided in the embodiment of the first aspect above.

[0023] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the multimodal prostate image registration method provided in the embodiment of the first aspect above.

[0024] The technical solution proposed in the embodiment of the present invention solves the drawbacks of the existing prostate image registration scheme. The INPUT and OUTPUT are expanded into two respectively, so that the multimodal prostate image registration model can simultaneously process multimodal data and output the registration deformation field; the prostate capsule segmentation and registration of TRUS are combined, which not only greatly reduces the calculation time, but also improves the accuracy of prostate image registration, and is suitable for promotion and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate and understand the technical solutions in the embodiments of the present invention, the following briefly introduces the background technology of the present invention and the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the contents of the embodiments of the present invention and these drawings without paying creative work.

[0026] Figure 1 A flow chart of a multimodal prostate image registration method provided by an embodiment of the present invention;

[0027] Figure 2 This is a structural block diagram of a multimodal prostate image registration system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the technical problems solved by the present invention, the technical solutions adopted and the technical effects achieved clearer, the technical solutions of the embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0029] Embodiment 1

[0030] Please refer to Figure 1 As stated, Figure 1 A flow chart of a multimodal prostate image registration method provided in an embodiment of the present invention. As shown in the figure, a multimodal prostate image registration method 100 in this embodiment includes:

[0031] S101. Constructing a multimodal prostate image registration model; wherein the multimodal prostate image registration model simultaneously processes prostate TRUS data and MRI / CT data, and outputs TRUS segmentation results and registration deformation fields;

[0032] S102. Acquire prostate TRUS data and MRI / CT data to be registered;

[0033] S103. Process the prostate TRUS data and MRI / CT data to be registered according to the multimodal prostate image registration model to complete prostate image registration.

[0034] Exemplarily, in the multimodal prostate image registration model, both INPUT and OUTPUT are expanded to two.

[0035] Exemplarily, the content of the INPUT is set, which includes but is not limited to TRUS data and MRI / CT segmentation results.

[0036] Exemplarily, the content of the OUTPUT is set, which includes but is not limited to the segmentation result of TRUS and the registration deformation field.

[0037] In this embodiment, the multimodal prostate image registration model adopts UNET3D-MultiModal-Fusion, the INPUT is TRUS data and MRI / CT segmentation results; the OUTPUT content is the TRUS segmentation results and the registration deformation field.

[0038] The multimodal prostate image registration method proposed in this embodiment solves the drawbacks of existing prostate image registration schemes. It expands the INPUT and OUTPUT into two respectively, so that the multimodal prostate image registration model can simultaneously process multimodal data and output a registration deformation field; it combines the prostate capsule segmentation and registration of TRUS, which not only greatly reduces the calculation time, but also improves the accuracy of prostate image registration, and is suitable for promotion and application.

[0039] Embodiment 2

[0040] The multimodal prostate image registration method in this embodiment includes:

[0041] S201. Construct a multimodal prostate image registration model; wherein the multimodal prostate image registration model simultaneously processes prostate TRUS data and MRI / CT data, and outputs TRUS segmentation results and registration deformation fields;

[0042] S202. Acquire prostate TRUS data and MRI / CT data to be registered;

[0043] S203. Process the prostate TRUS data and MRI / CT data to be registered according to the multimodal prostate image registration model to complete prostate image registration.

[0044] Exemplarily, in the multimodal prostate image registration model, both INPUT and OUTPUT are expanded to two. The content of the INPUT is set, including but not limited to TRUS data and MRI / CT segmentation results. The content of the OUTPUT is set, including but not limited to TRUS segmentation results and registration deformation field.

[0045] In this embodiment, the multimodal prostate image registration model adopts UNET3D-MultiModal-Fusion, the INPUT is TRUS data and MRI / CT segmentation results; the OUTPUT content is the TRUS segmentation results and the registration deformation field.

[0046] Specifically, the multimodal prostate image registration model proposed in this embodiment is designed based on UNET3D, wherein the UNet3D is an extended version of the UNet architecture for three-dimensional data processing, which adopts an encoder-decoder structure and introduces skip connections to retain more spatial information and reduce information loss. In this embodiment, UNET3D-MultiModal-Fusion is specifically introduced as follows:

[0047] For the input layer (INPUT), it includes:

[0048] 1. TRUS volume data (T): the size is `(batch_size, 1, H, W, D)`; wherein the TRUS volume data is the same as the TRUS data above;

[0049] 2. Segmentation results from MRI / CT (M): size is `(batch_size,1,H,W,D)`

[0050] For the encoder:

[0051] 1. Shared encoder:

[0052] Convolutional layer 1: input `T` and `M`, the number of output channels is 64, and the activation function is ReLU

[0053] Convolutional layer 2: Input the output of the previous layer, the number of output channels is 128, and the activation function is ReLU

[0054] Convolutional layer 3: Input the output of the previous layer, the number of output channels is 256, and the activation function is ReLU

[0055] 2.TRUS feature extraction encoder:

[0056] Convolutional layer 4: input `T`, output channel number is 64, activation function is ReLU

[0057] Convolutional layer 5: Input the output of the previous layer, the number of output channels is 128, and the activation function is ReLU

[0058] 3. MRI / CT feature extraction encoder:

[0059] Convolutional layer 6: input `M`, output channels are 64, activation function is ReLU

[0060] Convolutional layer 7: Input the output of the previous layer, the number of output channels is 128, and the activation function is ReLU

[0061] For the decoder:

[0062] 1. Fusion decoder:

[0063] Upsampling layer 1: Input shared encoder output, output size is `(batch_size,256,H / 2,W / 2,D / 2)`

[0064] Convolutional layer 8: Input the output of the previous layer, the number of output channels is 128, and the activation function is ReLU

[0065] Upsampling layer 2: Input the output of the previous layer, the output size is `(batch_size,128,H,W,D)`

[0066] Convolutional layer 9: input the output of the previous layer, the number of output channels is 64, and the activation function is ReLU

[0067] 2. For the TRUS segmentation result decoder:

[0068] Upsampling layer 3: Input the output of the TRUS feature extraction encoder, the output size is `(batch_size,128,H / 2,W / 2,D / 2)`

[0069] Convolutional layer 10: input the output of the previous layer, the number of output channels is 64, and the activation function is ReLU

[0070] For MRI / CT segmentation result decoder:

[0071] Upsampling layer 4: Input the output of the MRI / CT feature extraction encoder, the output size is `(batch_size,128,H / 2,W / 2,D / 2)`

[0072] Convolutional layer 11: Input the output of the previous layer, the number of output channels is 64, and the activation function is ReLU

[0073] For the output layer (OUTPUT), it includes:

[0074] TRUS segmentation result (T_seg): size is `(batch_size,1,H,W,D)`

[0075] Registration deformation field (M_reg): size is `(batch_size,3,H,W,D)`

[0076] The multimodal prostate image registration method proposed in this embodiment solves the drawbacks of existing prostate image registration schemes. It expands the INPUT and OUTPUT into two respectively, so that the multimodal prostate image registration model can simultaneously process multimodal data and output a registration deformation field; it combines the prostate capsule segmentation and registration of TRUS, which not only greatly reduces the calculation time, but also improves the accuracy of prostate image registration, and is suitable for promotion and application.

[0077] Embodiment 3

[0078] like Figure 2 As shown, Figure 2 The structure block diagram of the multimodal prostate image registration system provided by the embodiment of the present invention. As shown in the figure, the multimodal prostate image registration system 200 in this embodiment adopts the multimodal prostate image registration method proposed in any one of the first embodiment or the second embodiment, and the system includes:

[0079] A model building module 201 is used to build a multimodal prostate image registration model; wherein the multimodal prostate image registration model processes prostate TRUS data and MRI / CT data simultaneously, and outputs TRUS segmentation results and registration deformation fields;

[0080] A registration data acquisition module 202, used to acquire TRUS data and MRI / CT data of the prostate to be registered;

[0081] The prostate image registration module 203 is used to process the prostate TRUS data and MRI / CT data to be registered according to the multimodal prostate image registration model to complete the prostate image registration.

[0082] In this embodiment, the model building module 201 expands both the INPUT and OUTPUT in the multimodal prostate image registration model to two. In this embodiment, the model building module 201 is also used to set the TRUS data and the MRI / CT segmentation results as the content of the INPUT, and set the TRUS segmentation results and the registration deformation field as the content of the OUTPUT.

[0083] Specifically, the multimodal prostate image registration model adopts UNET3D-MultiModal-Fusion, which is designed based on UNET3D:

[0084] For the input layer (INPUT), it includes:

[0085] 1. TRUS volume data (T): the size is `(batch_size, 1, H, W, D)`; wherein the TRUS volume data is the same as the TRUS data above;

[0086] 2. Segmentation results from MRI / CT (M): size is `(batch_size,1,H,W,D)`

[0087] For the encoder:

[0088] 1. Shared encoder:

[0089] Convolutional layer 1: input `T` and `M`, the number of output channels is 64, and the activation function is ReLU

[0090] Convolutional layer 2: Input the output of the previous layer, the number of output channels is 128, and the activation function is ReLU

[0091] Convolutional layer 3: Input the output of the previous layer, the number of output channels is 256, and the activation function is ReLU

[0092] 2.TRUS feature extraction encoder:

[0093] Convolutional layer 4: input `T`, output channel number is 64, activation function is ReLU

[0094] Convolutional layer 5: Input the output of the previous layer, the number of output channels is 128, and the activation function is ReLU

[0095] 3. MRI / CT feature extraction encoder:

[0096] Convolutional layer 6: input `M`, output channels are 64, activation function is ReLU

[0097] Convolutional layer 7: Input the output of the previous layer, the number of output channels is 128, and the activation function is ReLU

[0098] For the decoder:

[0099] 1. Fusion decoder:

[0100] Upsampling layer 1: Input shared encoder output, output size is `(batch_size,256,H / 2,W / 2,D / 2)`

[0101] Convolutional layer 8: Input the output of the previous layer, the number of output channels is 128, and the activation function is ReLU

[0102] Upsampling layer 2: Input the output of the previous layer, the output size is `(batch_size,128,H,W,D)`

[0103] Convolutional layer 9: input the output of the previous layer, the number of output channels is 64, and the activation function is ReLU

[0104] 2. For the TRUS segmentation result decoder:

[0105] Upsampling layer 3: Input the output of the TRUS feature extraction encoder, the output size is `(batch_size,128,H / 2,W / 2,D / 2)`

[0106] Convolutional layer 10: input the output of the previous layer, the number of output channels is 64, and the activation function is ReLU

[0107] For MRI / CT segmentation result decoder:

[0108] Upsampling layer 4: Input the output of the MRI / CT feature extraction encoder, the output size is `(batch_size,128,H / 2,W / 2,D / 2)`

[0109] Convolutional layer 11: Input the output of the previous layer, the number of output channels is 64, and the activation function is ReLU

[0110] For the output layer (OUTPUT), it includes:

[0111] TRUS segmentation result (T_seg): size is `(batch_size,1,H,W,D)`

[0112] Registration deformation field (M_reg): size is `(batch_size,3,H,W,D)`

[0113] The multimodal prostate image registration system 200 proposed in this embodiment expands the INPUT and OUTPUT into two respectively, so that the multimodal prostate image registration model can simultaneously process multimodal data and output the registration deformation field; the TRUS prostate capsule segmentation and registration are combined, which not only greatly reduces the calculation time, but also improves the accuracy of prostate image registration.

[0114] Embodiment 4

[0115] This embodiment provides an electronic device, which includes a processor and a memory connected to the processor, wherein program data is stored in the memory, and the processor calls the program data stored in the memory to execute the multimodal prostate image registration method proposed in any of the above-mentioned embodiments 1 or 2.

[0116] Embodiment 5

[0117] This embodiment provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the multi-modal prostate image registration method proposed in any one of the above-mentioned embodiments 1 or 2.

[0118] It should be noted that the above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0119] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A multimodal prostate image registration method, characterized in that: include: Constructing a multimodal prostate image registration model; wherein the multimodal prostate image registration model processes prostate TRUS data and MRI / CT data simultaneously, and outputs TRUS segmentation results and registration deformation fields; Collect the prostate TRUS data and MRI / CT data to be registered; The prostate TRUS data and MRI / CT data to be registered are processed according to the multimodal prostate image registration model to complete the prostate image registration.

2. The multimodal prostate image registration method according to claim 1, characterized in that: In the multimodal prostate image registration model, both INPUT and OUTPUT are expanded to two.

3. The multimodal prostate image registration method according to claim 2, characterized in that: The content of the INPUT is set, which includes but is not limited to TRUS data and MRI / CT segmentation results.

4. The multimodal prostate image registration method according to claim 3, characterized in that: Set the content of the OUTPUT, which includes but is not limited to the segmentation result of TRUS and the registered deformation field.

5. The multimodal prostate image registration method according to claim 4, characterized in that: The multimodal prostate image registration model adopts UNET3D-MultiModal-Fusion, the INPUT is TRUS data and MRI / CT segmentation results; the OUTPUT content is the TRUS segmentation results and the registration deformation field.

6. A multimodal prostate image registration system, characterized in that: The system adopts the multimodal prostate image registration method of claim 1, comprising: A model building module, used to build a multimodal prostate image registration model; wherein the multimodal prostate image registration model processes prostate TRUS data and MRI / CT data simultaneously, and outputs TRUS segmentation results and registration deformation fields; A registration data acquisition module, used to acquire TRUS data and MRI / CT data of the prostate to be registered; The prostate image registration module is used to process the prostate TRUS data and MRI / CT data to be registered according to the multimodal prostate image registration model to complete the prostate image registration.

7. The multimodal prostate image registration system according to claim 6, characterized in that: The model building module expands both the INPUT and OUTPUT in the multimodal prostate image registration model into two.

8. The multimodal prostate image registration system according to claim 7, characterized in that: The model building module is also used to set the TRUS data and the MRI / CT segmentation results as the content of INPUT, and set the TRUS segmentation results and the registration deformation field as the content of OUTPUT.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory connected to the processor, wherein the memory stores program data, and the processor retrieves the program data stored in the memory to execute the multimodal prostate image registration method as described in claim 1.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the multi-modality prostate image registration method as claimed in claim 1.

Images