Segmentation network model of heart 3D image, training method, segmentation method and equipment

Abstract

The invention discloses a heart 3D image segmentation network model, a training method, a segmentation method and equipment, and aims to greatly reduce the annotation cost in a traditional method through a weak supervised learning method and more accurately perform heart 3D image segmentation. According to the training method of the segmentation network model, MIP projection is carried out on a heart 3D image, depth information of the heart 3D image is obtained, then the heart is calibrated for 2D projection, and then the depth information is combined to restore to a 3D volume as a pseudo label. And then combining the difference between the output of the 3D network and the pseudo label and the difference between the projection output by the 3D network and the 2D label as a loss function to guide the network to train, and finally obtaining a required segmentation network model.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and more specifically, to a segmentation network model, training method, segmentation method and device for a cardiac 3D image. Background Art

[0002] The main pathophysiological change of mitral regurgitation is that mitral regurgitation increases the left atrial load and left ventricular diastolic load. The current surgery for treating this disease is called mitral valve ablation. This surgery requires doctors to puncture the guidewire from the blood vessel to the heart and send the clip to the mitral valve to complete the repair. It is often not enough to rely solely on the guidance of DSA (subtraction angiography), and it often depends on the doctor's experience to a greater extent. However, the non-rigidity of the guidewire brings great difficulties to the navigation of the guidewire. Doctors can only see the position of the guidewire in the blood vessel through CT data, but lack information about the relative position of its 3D posture, especially when the placement of the clip is completed by puncture, which often brings great danger to the patient. Therefore, more visual information is very necessary. 3D segmentation of the heart based on CT data can help it accurately distinguish the 3D position of the guidewire in the heart and relieve the patient's pain. However, previous segmentation is often based on supervised 3D segmentation, which often requires a large amount of real annotation. Therefore, the cost of 3D annotation is often very high, which brings great difficulties to this work. So how to achieve more accurate segmentation at a lower cost is a problem that needs to be solved. Summary of the invention

[0003] In view of the problems existing in the prior art, the present invention proposes a segmentation network model, training method, segmentation method and device for cardiac 3D images based on weakly supervised learning, which greatly reduces the annotation cost while providing very accurate cardiac 3D segmentation, providing more visual navigation for cardiac surgery.

[0004] To achieve the above objectives, in a first aspect, the present invention provides a training method for a segmentation network model of a cardiac 3D image, characterized in that it comprises:

[0005] Step S1, collecting and performing MIP (maximal intensity projection) processing on a 3D image of the heart to obtain depth information of the image;

[0006] Step S2, performing 2D projection on the acquired 3D heart image, and annotating the heart in the 2D projection;

[0007] Step S3, combining the depth information of the image, restoring the annotated 2D projection to a 3D volume as a pseudo label;

[0008] Step S4: The difference between the 3D output of the segmentation network model and the pseudo label and the difference between the 2D projection output by the segmentation network model and the 2D projection annotation are combined as a loss function through weight parameters to guide the training of the segmentation network model.

[0009] The present invention performs MIP projection on 3D data, then restores the 2D image to 3D after annotation, and obtains joint loss to guide the training of the segmentation network through cross-comparison, which greatly reduces the cost of annotation and can also provide very accurate 3D segmentation of the heart.

[0010] In some embodiments of the first aspect of the present application, in step S1, the heart 3D image is a heart CT, CTA (coronary CT angiography) image or MR (magnetic resonance imaging) image.

[0011] In some embodiments of the first aspect of the present application, in step S3, in order to incorporate depth information in the MIP, the first and last z coordinates with the same intensity as the MIP are represented as the forward depth zfw = arg maxz I (x, y, z) and the backward depth zbw = arg minz I (x, y, z), respectively. This information can be used for the following purposes: (1) to enhance MIP visualization, or (2) to provide a way to map pixels from a 2D MIP back to a 3D space (depth map).

[0012] Furthermore, step S3 maps the foreground pixels from the 2D annotation to the 3D volume by combining the 2D annotation with the forward and backward depths, specifically comprising the following steps: first, create an empty 3D volume D ∈ RNx×Ny×Nz, where RNi represents an n-dimensional vector, which is the value stored at the position with coordinate i, i is x, y or z; for each foreground pixel at position (x, y) in the annotation A, mark (x, y, zfw) and (x, y, zbw) as foreground pixels in D; if the intensity fluctuation along the ray rxy between zfw and zbw in the source image I is lower than a certain threshold, the intermediate pixel is also marked as foreground in D.

[0013] In some embodiments of the first aspect of the present application, in step S4, the training set Dtr(I, A, D) consists of a 3D volume I paired with a 2D annotation A and its corresponding 3D depth map D; given a 3D U-net network output Y = θ(I), the following loss function is minimized during training:

[0014]

[0015] where weight α ∈ [0, 1], is the cross entropy of the network output and 2D annotations projected to 2D, is the cross entropy between the network output and the depth map.

[0016] In a second aspect, the present invention provides a segmentation network model for cardiac 3D images, characterized in that the segmentation network model is one of 3D U-net, VT-Unet, MedNext, and SAT network models, and is pre-trained by executing the training method described above.

[0017] In a third aspect, the present invention provides a method for segmenting a 3D cardiac image, characterized in that the 3D cardiac image segmentation is performed using a pre-trained segmentation network model as described above.

[0018] In a fourth aspect, the present invention provides an electronic device, characterized by a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the training method as described above and / or the cardiac 3D image segmentation method as described above.

[0019] Compared with the prior art, the present invention has the following technical effects:

[0020] The present invention has developed a new method for training segmentation network models, that is, by performing MIP projection on 3D data, restoring the annotations of 2D images to 3D space, and cross-matching to obtain a joint loss function to guide network training, which can greatly reduce the cost of annotation in traditional methods. At the same time, the segmentation network model trained by the present invention can also provide very accurate 3D segmentation of the heart, providing more visual navigation for cardiac surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Flow chart of pre-training of a segmentation network model in one embodiment of the present invention.

[0022] Figure 2 In one embodiment of the present invention, the segmentation network model is pre-trained Figure 1 Schematic diagram of the heart 3D image processing process.

[0023] Figure 3 This is an example diagram of the process and effect of 3D heart segmentation in one embodiment of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0025] In the following detailed description, many specific details are set forth to provide a more thorough understanding of the present invention. However, it is obvious to those skilled in the art that well-known algorithms are not shown in detail to avoid obscuring the subject matter of the present invention.

[0026] In addition, the order of execution of actions, steps, etc. in the devices and methods shown in the claims, specifications and drawings can be implemented in any order as long as there is no special explicit limitation on the order and the output of the previous processing is not used in the subsequent processing.

[0027] Example 1

[0028] This example uses the 3D U-Net segmentation network model as an example (which can be transplanted to one of the VT-Unet, MedNext, and SAT network models). Figure 1 , providing a training method for a segmentation network model of a heart 3D image, comprising:

[0029] Step S1, collect and MIP process the heart 3D image to obtain the depth information of the image. Preferably, the heart 3D image is a heart CT, CTA or MR image. The image usually collected is a DICOM image, which consists of many 2D slices to form a 3D scan result or a specific part of the body.

[0030] MIP projection, or Maximum Intensity Projection, is a commonly used technique in medical imaging and image processing. MIP projection is generated by calculating the maximum density pixel encountered along each ray of the scanned object. When the light beam passes through the original image of a section of tissue, the pixel with the highest density in the image is retained and projected onto a two-dimensional plane to form a MIP reconstructed image.

[0031] MIP technology can also be understood as superimposing several layers of scanned images, projecting the high-density parts and deleting the low-density parts, forming a two-dimensional projection of the three-dimensional structure of these high-density parts. It can be projected from any angle, or multiple images of continuous angles can be continuously broadcast on the monitor to give the viewer a three-dimensional sense.

[0032] MIP is mostly used for vascular imaging, such as cerebrovascular and renal vascular imaging, and is one of the most accurate CT post-processing techniques for measuring blood vessels. MIP can reflect the X-ray attenuation value of the corresponding pixel, and even small density changes can be displayed on the MIP image. It can well display vascular stenosis, dilation, and filling defects, and distinguish calcification on the vascular wall from contrast agents in the vascular cavity.

[0033] Compared with VIP (Volume Intensity Projection, stereoscopic imaging), MIP mode specifies the highest density or maximum brightness value of the voxels closest to the viewpoint, while the farther voxels will gradually weaken according to their distance from the viewpoint. MIP can more clearly show the entire course and opening of the coronary artery in the near field of view.

[0034] Step S2, perform 2D projection on the acquired 3D cardiac image and annotate the heart in the 2D projection. Preferably, the acquired 3D cardiac image is forward projected and backward projected. For simplicity, we describe the MIP only along the z-axis, but they can be performed on any image axis. Using the fact that arteries are high signals in arterial phase CT, we annotate the MIP of the input volume for binary segmentation. See Figure 2 , Figure 2 (a) shows Figure 1 Forward projection of 3D images of the heart, Figure 2 (b) shows its back projection, Figure 2 (c) shows the corresponding binary annotations. For manual calibration annotation of projection images, it is usually sufficient to calibrate a single 2D image. The high signal of the artery ensures its visibility in the MIP, while additional processing removes most of the occluded nearby tissue. Given the binary 2D annotation of MIP A, we map the foreground pixels in A to the original 3D image space. See Figure 2 , Figure 2 (d) shows the restoration of 2D annotated images to 3D by incorporating depth information.

[0035] Step S3, combined with the depth information of the image, restore the annotated 2D projection to the 3D volume as a pseudo-label. In order to incorporate depth information in the MIP projection, we represent the first and last z coordinates with the same intensity as the MIP as the forward depth zfw = arg maxz I(x, y, z) and the backward depth zbw = arg minz I(x, y, z). This information can be used for the following purposes: (1) to enhance MIP visualization, or (2) to provide a way to map pixels from the 2D MIP back to 3D space (depth map). The reason why the maximum intensity is reached multiple times along the ray is because the image is cropped, which eliminates a lot of intensity fluctuations. See Figure 2 , Figure 2 (e) shows the corresponding forward depth map, Figure 2 (f) shows the corresponding backward depth, Figure 2 (g) shows the corresponding depth map. A depth map is a 2D image in which each pixel value reflects the depth information of the location.

[0036] We encode depth information into the MIPs by combining them with the forward and backward depths, respectively, to enable better depth perception during annotation: pfw = pmip zfw defines the forward projection, while pbw = pmip zbw defines the backward projection. This is achieved by using the first and last z-coordinates where the maximum intensity is observed along any projection ray. This step results in a fairly complete surface of the vessel tree due to the fact that vessels in the abdominal cavity are relatively sparse in 2D projections and most of the occluded tissue is removed in post-processing.

[0037] By combining 2D annotations with forward and backward depth, the foreground pixels from 2D annotations are mapped to a 3D volume, specifically including the following steps: first, create an empty 3D volume D ∈ RNx×Ny×Nz, where RNi represents an n-dimensional vector, which is the value stored at the position with coordinate i, i is x, y or z; for each foreground pixel at position (x, y) in the annotation A, (x, y, zfw) and (x, y, zbw) are marked as foreground pixels in D; if the intensity fluctuation along the ray rxy between zfw and zbw in the source image I is lower than a certain threshold, the intermediate pixel is also marked as foreground in D. Figure 2 (h) shows the 3D ground truth.

[0038] Step S4, the difference between the 3D output of the segmentation network model and the pseudo label and the difference between the 2D projection output of the segmentation network model and the 2D projection annotation are combined as a loss function through weight parameters to guide the segmentation network model to train. Preferably, we train a 3D U-Net segmentation network to predict 3D heart segmentation. The U-Net architecture is a powerful medical image segmentation model. 3D U-Net extends the classic U-Net model to 3D segmentation. It consists of an encoding (downsampling) path and a decoding (upsampling) path.

[0039] The encoding path captures the context in the input image, while the decoding path allows for precise localization. 3D U-Net is very effective in processing the 3D characteristics of volumetric images.

[0040] Given a 3D input volume with 2D annotations, the training set Dtr(I, A, D) consists of the 3D volume I paired with the 2D annotation A and its corresponding 3D depth map D. Given the 3D network output Y = θ(I), we minimize the following loss during training:

[0041]

[0042] where weights α ∈ [0, 1]. Our final loss is a convex combination of: (a) the cross entropy (CE) between the network output and the 2D annotations projected to 2D, and (b) the cross entropy between the network output and the depth map, but only applied to positive pixels in the depth map. Notably, the 2D loss constrains the shape of the vessel, while the depth loss promotes segmentation of the vessel interior.

[0043] From the above description, it can be seen that the segmentation network model training method mentioned above mainly utilizes 2D projection and depth information. It is a weakly supervised method, which can not only greatly reduce the cost of annotation in traditional methods, but also the trained model takes into account the shape of blood vessels and the internal structure of blood vessels at the same time, so it can perform more accurate heart 3D image segmentation.

[0044] Example 2

[0045] This embodiment provides a segmentation network model for a heart 3D image. The segmentation network model is a 3D U-net network model, and is pre-trained using the training method described in Embodiment 1.

[0046] This embodiment also provides a method for segmenting a 3D cardiac image, using the pre-trained segmentation network model to perform 3D cardiac image segmentation, and the segmentation effect is as follows: Figure 3 . Figure 3 (a) is the projected 2D image, Figure 3 (b) is 2D annotation, Figure 3 (c) is the 2D projection of the segmentation effect. Figure 3 (d) is the 3D ground truth, Figure 3 (e) is the 3D segmentation effect. From the example, we can see that such a weakly supervised learning segmentation network model makes it possible to reconstruct and register the subsequent images, providing good visual assistance for many cardiac surgeries such as mitral valve ablation, and laying a foundation for the cardiac CT navigation system.

[0047] Example 3

[0048] This embodiment provides an electronic device, comprising a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the training method described in Example 1 and / or the cardiac 3D image segmentation method described in Example 2.

[0049] The above-mentioned segmentation network model and its training method can be embodied in the form of a computer program product or a software functional unit. If the above-mentioned segmentation network model and its training method are implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium. Therefore, the essence of the technical solution or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for an electronic system (which can be a personal computer, a server, or a network system, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0050] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0051] In summary, the present invention provides a segmentation network model, training method, segmentation method and device for a 3D cardiac image, which aims to greatly reduce the cost of annotation in traditional methods through a weakly supervised learning method, and can more accurately segment 3D cardiac images. The training method of the segmentation network model performs MIP projection on the 3D cardiac image and obtains its depth information, then calibrates the heart on the 2D projection, and then restores the 3D volume as a pseudo-label in combination with the depth information, and then combines the difference between the output of the 3D network and the pseudo-label and the difference between the projection of the 3D network output and the 2D annotation as a loss function to guide the network to train, and finally obtain the required segmentation network model.

[0052] Those skilled in the art should understand that those skilled in the art can implement variations by combining the prior art and the above embodiments, which will not be described in detail here. Such variations do not affect the essential content of the present invention, and will not be described in detail here.

[0053] The above describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and the systems and structures that are not described in detail should be understood to be implemented in a common manner in the art; any technician familiar with the art can use the above-disclosed methods and technical contents to make many possible changes and modifications to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, or modify them into equivalent embodiments of equivalent changes, which does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention are still within the scope of protection of the technical solutions of the present invention.

Claims

1. A training method for a segmentation network model of a heart 3D image, characterized in that: include: Step S1, collecting and performing MIP processing on a 3D cardiac image to obtain depth information of the image; Step S2, performing 2D projection on the acquired 3D heart image, and annotating the heart in the 2D projection; Step S3, combining the depth information of the image, restoring the annotated 2D projection to a 3D volume as a pseudo label; Step S4: The difference between the 3D output of the segmentation network model and the pseudo label and the difference between the 2D projection output by the segmentation network model and the 2D projection annotation are combined as a loss function through weight parameters to guide the training of the segmentation network model.

2. The training method of the cardiac 3D image segmentation network model according to claim 1, characterized in that: In the step S1, the cardiac 3D image is a cardiac CT, CTA or MR image.

3. The training method of the segmentation network model of the heart 3D image according to claim 1, characterized in that: In step S3, in order to incorporate depth information in the MIP, the first and last z coordinates having the same intensity as the MIP are represented as the forward depth zfw = arg maxz I (x, y, z) and the backward depth zbw = arg minz I (x, y, z), respectively.

4. The training method of the cardiac 3D image segmentation network model according to claim 3, characterized in that: The step S3 maps the foreground pixels from the 2D annotation to the 3D volume by combining the 2D annotation with the forward and backward depths, specifically comprising the following steps: first, create an empty 3D volume D ∈ RNx×Ny×Nz, where RNi represents an n-dimensional vector, which is the value stored at the position with coordinate i, i is x, y or z; for each foreground pixel at position (x, y) in the annotation A, mark (x, y, zfw) and (x, y, zbw) as foreground pixels in D; if the intensity fluctuation along the ray rxy between zfw and zbw in the source image I is lower than a certain threshold, the intermediate pixel is also marked as foreground in D.

5. The training method of the cardiac 3D image segmentation network model according to claim 1, characterized in that: In step S4, the training set Dtr(I, A, D) consists of a 3D volume I paired with a 2D annotation A and its corresponding 3D depth map D; given the 3DU-net network output Y = θ(I), the following loss function is minimized during training: where weight α ∈ [0, 1], is the cross entropy of the network output and 2D annotations projected to 2D, is the cross entropy between the network output and the depth map.

6. A network model for segmentation of cardiac 3D images, characterized in that: The segmentation network model is one of 3D U-net, VT-Unet, MedNext, and SAT network models, and is pre-trained by executing the training method described in any one of claims 1 to 5.

7. A method for segmenting a 3D cardiac image, characterized in that: The pre-trained segmentation network model according to claim 6 is used to perform cardiac 3D image segmentation.

8. An electronic device, characterized in that: A memory storing executable program code; a processor coupled to the memory; the processor calling the executable program code stored in the memory to execute the training method according to any one of claims 1 to 5 and / or the cardiac 3D image segmentation method according to claim 7.

Images