Whole heart CT segmentation method and device based on multi-label and multi-decoder

Abstract

The present invention proposes a whole-heart CT segmentation method and device based on multi-label and multi-decoder, which includes the following steps: obtaining a training CT image, the training CT image including multiple target objects to be segmented and first label information corresponding to the multiple target objects to be segmented; fusing the first label information of the multiple target objects to be segmented according to a preset rule to obtain multiple second label information, wherein the granularity of the first label information is smaller than the granularity of the second label information; preprocessing the CT image, and cropping the preprocessed CT image into multiple image blocks of preset sizes; constructing a U-net network, the U-net network including a first decoder and a second decoder, the first decoder being used to segment fine-grained labels, and the second decoder being used to segment coarse-grained labels; training the U-net network according to the image blocks, the first label information and the second label information to generate a segmentation model; thereby achieving semantic segmentation of the whole heart at different granularities.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing and computer vision, and in particular relates to a whole heart CT segmentation method based on multiple labels and multiple decoders. Background Art

[0002] Cardiovascular disease has become the leading cause of death worldwide. According to the "China Cardiovascular Health and Disease Report 2019," cardiovascular disease currently leads the list of all causes of death in both urban and rural areas, accounting for 45.91% in rural areas and 43.56% in urban areas. Cardiovascular disease poses a significant threat to human health and places an increasing burden on the healthcare system. With the advancement of computer technology and medical imaging equipment, medical imaging has become widely used in the diagnosis of cardiovascular disease. Clinically, there is a significant demand for computer-assisted diagnosis and segmentation of cardiac anatomical structures from chest CT scans, which plays a crucial role in computer-assisted diagnosis and interventional therapy.

[0003] The goal of whole heart segmentation is to segment important cardiac anatomical structures in medical images, including the left and right ventricles, left and right atria, myocardium, and major blood vessels (such as the ascending aorta and pulmonary artery). In recent years, deep learning technology has been increasingly applied to the field of image segmentation and has demonstrated considerable segmentation results, gradually becoming a research hotspot in the field of image segmentation. However, in the field of medical image segmentation, there are two main problems with heart segmentation methods:

[0004] First, existing heart segmentation methods mainly focus on segmenting one or several anatomical structures of the heart, such as the left atrium and left ventricle, and rarely involve segmentation of the entire heart. The whole heart segmentation task usually has more categories and usually covers all the main anatomical structures of the heart, such as the left atrium, left ventricle, right atrium, right ventricle, aorta, pulmonary artery, etc.

[0005] Secondly, existing heart segmentation methods are mainly single-label segmentation methods, which ignore the relationship between different anatomical structures. Clinically, CT angiography can enhance the distinction between various anatomical structures of the heart and is an important means of examining cardiac lesions. Affected by the structural characteristics of the heart, the contrast agent concentrations of connected blood chambers (such as the left atrium and ventricle) are similar, and their grayscale differences are small in CT images; however, the left and right blood chambers of the heart (such as the left atrium and right atrium) are not directly connected, and are affected by the human body's blood circulation. The difference in contrast agent concentration is relatively large, which is reflected in the large grayscale difference in CT images. This grayscale difference causes inconsistency in the inter-class distances of different anatomical structures. For example, the left atrium and left ventricle have a higher similarity in grayscale, while the left atrium and right atrium have a lower similarity. Summary of the Invention

[0006] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0007] To this end, the first object of the present invention is to propose a whole heart CT segmentation method based on multi-label and multi-decoder to achieve semantic segmentation of the whole heart at different granularities.

[0008] The second object of the present invention is to propose a whole heart CT segmentation device based on multiple labels and multiple decoders.

[0009] To achieve the above-mentioned purpose, an embodiment of the first aspect of the present invention proposes a whole-heart CT segmentation method based on multiple labels and multiple decoders, comprising: obtaining a training CT image, the training CT image including multiple target objects to be segmented and first label information corresponding to the multiple target objects to be segmented; fusing the first label information of the multiple target objects to be segmented according to a preset rule to obtain multiple second label information, wherein the granularity of the first label information is smaller than the granularity of the second label information; preprocessing the CT image, and cropping the preprocessed CT image into multiple image blocks of preset sizes; constructing a U-net network, the U-net network including a first decoder and a second decoder, the first decoder being used to segment fine-grained labels, and the second decoder being used to segment coarse-grained labels; training the U-net network according to the image blocks, and the first label information and the second label information to generate a segmentation model.

[0010] The embodiment of the present invention proposes a whole-heart CT segmentation method based on multiple labels and multiple decoders. The whole-heart CT segmentation method based on multiple labels and multiple decoders rationally utilizes the inconsistency information of inter-class distances to guide the segmentation model to better learn the characteristics of the cardiac anatomical structure, thereby improving the segmentation effect. The structurally connected categories are fused with labels as coarse-grained segmentation labels. The whole heart is segmented at different granularities on fine-grained and coarse-grained labels, that is, the whole heart segmentation is regarded as a multi-label segmentation task, in which each voxel has a coarse-grained category and a fine-grained category. This multi-semantic label segmentation method aims to make full use of the inter-class inconsistency information of each category, making the model easier to learn and optimize. Finally, a 3D U-net network model using multiple decoders is proposed, and each decoder predicts labels of different granularities. A multi-task learning mechanism is adopted to jointly optimize each decoder to achieve semantic segmentation of the whole heart at different granularities, thereby realizing the learning of the segmentation model assisted by the inconsistency information of inter-class distances.

[0011] In addition, the whole heart CT segmentation method based on multi-label and multi-decoder according to the above embodiment of the present invention may also have the following additional technical features:

[0012] Furthermore, in one embodiment of the present invention, after generating the segmentation model, the method further includes:

[0013] Acquire a CT image to be segmented;

[0014] Preprocessing the CT image to be segmented and dividing it into a plurality of image blocks, respectively inputting the plurality of image blocks into the segmentation model to obtain a segmentation result of each image block;

[0015] The segmentation results are spliced to generate a target segmentation result of the CT image to be segmented.

[0016] Furthermore, in one embodiment of the present invention, preprocessing the CT image includes:

[0017] Isotropic sampling, linear window grayscale truncation, and unified CT image coordinate system.

[0018] Furthermore, in one embodiment of the present invention, a training CT image is collected, which includes the entire heart structure, and the image is manually segmented and labeled to obtain first label information, wherein the segmented categories include left atrium, left ventricle, aorta, right atrium, right ventricle, pulmonary artery, and left myocardium.

[0019] Furthermore, in one embodiment of the present invention, the preset rules include: fusing the first label information to obtain the second label information based on the prior anatomical structure of the heart and the similarity differences between classes of different anatomical structures, wherein the categories include: left blood cavity, right blood cavity, and left myocardium.

[0020] Furthermore, in one embodiment of the present invention, isotropic sampling includes:

[0021] The spatial resolution is unified to m millimeters, where m is selected according to the characteristics of the data and task requirements. After sampling, the spatial interval between each voxel and its upper, lower, left, and right adjacent voxels is m millimeters.

[0022] Furthermore, in one embodiment of the present invention, the linear window grayscale truncation includes:

[0023] According to the grayscale range of the heart, a linear window is added to the CT images in the dataset, and the grayscale is clipped to between -1000 and 800. Then, each CT image is normalized to the maximum and minimum values.

[0024] Furthermore, in one embodiment of the present invention, the unified CT image coordinate system includes:

[0025] The coordinate systems of all CTs are unified into the LAS coordinate system, and the Left, Anterior, and Superior directions are selected as the positive directions of the three dimensions of CT.

[0026] Furthermore, in one embodiment of the present invention, a 3D U-net network architecture includes an encoder and two decoders with non-shared parameters, and there is a skip-layer connection between the encoder and the two decoders; wherein, the first decoder segments the first label information, and the second decoder segments the second label information. The two decoders use weighted cross entropy as the loss function for coarse-grained and fine-grained segmentation tasks, and optimize the segmentation model by joint training.

[0027] To achieve the above-mentioned purpose, the second embodiment of the present invention proposes a whole-heart CT segmentation device based on multiple labels and multiple decoders, including: an acquisition module for acquiring a training CT image, wherein the training CT image includes multiple target objects to be segmented and first label information corresponding to the multiple target objects to be segmented; a fusion module for fusing the first label information of the multiple target objects to be segmented according to a preset rule to obtain multiple second label information, wherein the granularity of the first label information is smaller than the granularity of the second label information; a preprocessing module for preprocessing the CT image and cropping the preprocessed CT image into multiple image blocks of preset sizes; a construction module for constructing a U-net network, wherein the U-net network includes a first decoder and a second decoder, the first decoder is used to segment fine-grained labels, and the second decoder is used to segment coarse-grained labels; a training module for training the U-net network according to the image blocks, the first label information and the second label information by minimizing the loss function to generate a segmentation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0029] Figure 1 This is a flow chart of a whole heart CT segmentation method based on multiple labels and multiple decoders provided by an embodiment of the present invention.

[0030] Figure 2 This is a flow chart of a whole heart CT segmentation device based on multiple labels and multiple decoders provided by an embodiment of the present invention.

[0031] Figure 3 This is a schematic diagram of the overall flow of the whole heart CT segmentation method based on multiple labels and multiple decoders provided by an embodiment of the present invention.

[0032] Figure 4 This is a schematic diagram of the anatomical structure prior of the heart and the similarity differences between classes of different anatomical structures provided by an embodiment of the present invention.

[0033] Figure 5A schematic diagram of fine-grained category labels provided by an embodiment of the present invention.

[0034] Figure 6 A schematic diagram of coarse-grained category labels provided by an embodiment of the present invention.

[0035] Figure 7 A schematic diagram of the 3D U-net architecture provided by an embodiment of the present invention.

[0036] Figure 8 、 9 10 is a schematic diagram of the segmentation result provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0038] The following describes a whole heart CT segmentation method and apparatus based on multiple labels and multiple decoders according to an embodiment of the present invention with reference to the accompanying drawings.

[0039] Figure 1 This is a flow chart of a whole heart CT segmentation method based on multiple labels and multiple decoders provided by an embodiment of the present invention.

[0040] like Figure 1 As shown, the whole heart CT segmentation method based on multi-label and multi-decoder includes the following steps:

[0041] S1: Acquire a training CT image, where the training CT image includes a plurality of target objects to be segmented and first label information corresponding to the plurality of target objects to be segmented;

[0042] S2: fusing the first label information of the plurality of target objects to be segmented according to a preset rule to obtain a plurality of second label information, wherein the granularity of the first label information is smaller than the granularity of the second label information;

[0043] S3: preprocessing the CT image and cutting the preprocessed CT image into multiple image blocks of preset sizes;

[0044] S4: constructing a U-net network, wherein the U-net network includes a first decoder and a second decoder, the first decoder is used to segment fine-grained labels, and the second decoder is used to segment coarse-grained labels;

[0045] S5: Training a U-net network according to the image block, the first label information, and the second label information to generate a segmentation model.

[0046] Furthermore, in one embodiment of the present invention, after generating the segmentation model, the method further includes:

[0047] Acquire a CT image to be segmented;

[0048] Preprocessing the CT image to be segmented and dividing it into a plurality of image blocks, respectively inputting the plurality of image blocks into the segmentation model to obtain a segmentation result of each image block;

[0049] The segmentation results are spliced to generate a target segmentation result of the CT image to be segmented.

[0050] Furthermore, in one embodiment of the present invention, a training CT image is collected, which includes the entire heart structure, and the image is manually segmented and labeled to obtain first label information, wherein the segmented categories include left atrium, left ventricle, aorta, right atrium, right ventricle, pulmonary artery, and left myocardium.

[0051] Furthermore, in one embodiment of the present invention, the preset rules include: fusing the first label information to obtain the second label information based on the prior anatomical structure of the heart and the similarity differences between classes of different anatomical structures, wherein the categories include: left blood cavity, right blood cavity, and left myocardium.

[0052] Preferably, in step S2, based on the anatomical structure prior of the heart and the similarity differences between classes of different anatomical structures, the fine-grained labels marked in step S1, i.e., left atrium, left ventricle, aorta, right atrium, right ventricle, pulmonary artery, and left myocardium, are fused to generate coarse-grained category labels, i.e., left blood cavity, right blood cavity, and left myocardium, wherein the left blood cavity class in the coarse-grained category is composed of the fine-grained categories of left atrium, left ventricle, and aorta, and the right blood cavity is composed of the right atrium, right ventricle, and pulmonary artery.

[0053] Through this label fusion approach, we can fully represent coarse and fine-grained information, and treat whole-heart segmentation as a multi-label segmentation task, where each voxel has coarse-grained and fine-grained categories. This multi-semantic label segmentation approach aims to fully utilize the inter-class inconsistency information of each category, making the model easier to learn and optimize.

[0054] Furthermore, in one embodiment of the present invention, preprocessing the CT image includes:

[0055] Isotropic sampling, linear window grayscale truncation, and unified CT image coordinate system.

[0056] Furthermore, in one embodiment of the present invention, isotropic sampling includes:

[0057] The spatial resolution is unified to m millimeters, where m is selected according to the characteristics of the data and task requirements. After sampling, the spatial interval between each voxel and its upper, lower, left, and right adjacent voxels is m millimeters.

[0058] Furthermore, in one embodiment of the present invention, the linear window grayscale truncation includes:

[0059] According to the grayscale range of the heart, a linear window is added to the CT images in the dataset, and the grayscale is clipped to between -1000 and 800. Then, each CT image is normalized to the maximum and minimum values.

[0060] Furthermore, in one embodiment of the present invention, the unified CT image coordinate system includes:

[0061] The coordinate systems of all CTs are unified into the LAS coordinate system, and the Left, Anterior, and Superior directions are selected as the positive directions of the three dimensions of CT.

[0062] Preferably, in step S3, the preprocessing operations include isotropic sampling, linear window grayscale truncation, and unified CT image coordinate system.

[0063] Preferably, the isotropic sampling operation is as follows: due to different examination purposes and acquisition equipment, different CT images usually have different spatial resolutions, and the same CT image usually has different spatial resolutions along different directions. For example, the spatial resolution along the z-axis is usually smaller than the resolution along the x- and y-axis directions, that is, the anisotropy problem of spatial resolution. When a model trained on data with the same resolution segments images with different resolutions, it often causes performance degradation. Therefore, in response to the anisotropy problem of the spatial resolution of CT images, a grid sampling method is adopted to unify the spatial resolution to m millimeters (m can be selected according to the characteristics of the data and task requirements). After sampling, the spatial interval between each voxel and its upper, lower, left and right adjacent voxels is m millimeters. By achieving isotropy of spatial resolution, the area covered by the convolution kernels of the same size in the subsequent CNN segmentation model can be the same volume, solving the degradation of segmentation accuracy caused by the anisotropy of the spatial resolution of CT images.

[0064] Preferably, the linear window grayscale truncation operation is as follows: the grayscale of each voxel in the CT image reflects the absorption coefficient of the corresponding tissue structure to X-rays, which is usually expressed in HU values. The range of HU values is usually between -1000 and 1000, with the HU value of water being equal to 0 as the benchmark. -1000 is usually the HU value of air, and the HU value of bone tissue with a dense structure is usually greater than 400. Since there are many organs and tissues in the chest cavity, the density difference between different tissues is large, such as the lungs have a lower density and the spine and ribs have a higher density, which makes the grayscale of the chest CT have a higher dynamic range, increasing the difficulty of heart segmentation. Therefore, according to the grayscale range of the heart area during examination, a linear window is added to the CT image, and the grayscale is cropped to between -1000 and 800. Finally, each CT image is normalized to the maximum and minimum values.

[0065] Preferably, the unified CT image coordinate system is operated as follows: due to the differences in different CT image acquisition devices and the different shooting angles of patients, the acquired images have large angle differences. This angle difference seriously affects the segmentation accuracy. We have achieved the unification of all CT coordinate systems into the LAS coordinate system, that is, selecting the Left, Anterior, and Superior directions as the positive directions of the three dimensions of CT. Subsequent training and reasoning all use the LAS coordinate system, which solves the problem of poor generalization ability of the model for CT image segmentation in different coordinate systems.

[0066] Furthermore, in one embodiment of the present invention, a 3D U-net network architecture includes an encoder and two decoders with non-shared parameters, and there is a skip-layer connection between the encoder and the two decoders; wherein, the first decoder segments the first label information, and the second decoder segments the second label information. The two decoders use weighted cross entropy as the loss function for coarse-grained and fine-grained segmentation tasks, and optimize the segmentation model by joint training.

[0067] Preferably, in steps S4 and S5, the 3D U-net architecture of the dual decoder constructed is a fully convolutional network, and the basic convolution operation adopts 3D CNN. The 3D U-net architecture of the dual decoder has an encoder with shared parameters and two decoders with non-shared parameters. There are skip-layer connections between the encoder and the two decoders, wherein the first decoder segments the fine-grained labels and the second decoder segments the coarse-grained labels. Both decoders use weighted cross entropy as the loss function of the coarse-grained and fine-grained segmentation task, and adopt a joint training method to optimize the model to achieve coarse-grained and fine-grained multi-label segmentation. The basic units of the network include 3D convolution layer, pooling layer, upsampling layer, and batch norm layer. The convolution kernel size of the 3D convolution layer is 3*3*3, which is used to extract CT image features; the step size of the pooling layer is 2, which is used to downsample the feature map to enhance the representation of high-level semantics while reducing the amount of computation; the function of the upsampling layer is to upsample the feature map and restore the image resolution; the function of the batch norm layer is to normalize the data and accelerate the convergence speed during the training phase.

[0068] The whole heart CT segmentation method based on multi-label and multi-decoder proposed in the embodiment of the present invention has the following beneficial effects:

[0069] First, the present invention rationally utilizes the inconsistency information of inter-class distances to guide the segmentation model to better learn the characteristics of the cardiac anatomical structure, thereby improving the segmentation effect. For fine-grained labels, the structurally connected categories are fused to generate coarse-grained segmentation labels. On fine-grained and coarse-grained labels, the whole heart is segmented at different granularities, that is, the whole heart segmentation is regarded as a multi-label segmentation task, in which each voxel has a coarse-grained category and a fine-grained category. This multi-semantic label segmentation method aims to make full use of the inter-class inconsistency information of each category, making the model easier to learn and optimize. Finally, a 3D U-net network model with multiple decoders is proposed, and each decoder predicts labels of different granularities. A multi-task learning mechanism is adopted to jointly optimize each decoder to achieve semantic segmentation of different granularities for the whole heart, thereby realizing the learning of the segmentation model assisted by the inconsistency information of inter-class distances.

[0070] Second, the present invention adopts a grid sampling method to unify the spatial resolution to m millimeters (m can be selected according to the characteristics of the data and task requirements). After sampling, the spatial interval between each voxel and its upper, lower, left and right adjacent voxels is m millimeters. By achieving isotropy of spatial resolution, the areas covered by convolution kernels of the same size in the subsequent CNN segmentation model can be the same volume, solving the problem of segmentation accuracy degradation caused by the anisotropy of spatial resolution of CT images.

[0071] Third, the present invention solves the problem of poor generalization ability of the model for CT image segmentation in different coordinate systems by unifying the CT coordinate system to the LAS coordinate system, that is, selecting the Left, Anterior, and Superior directions as the positive directions of the three dimensions of CT.

[0072] This example uses CT images from the public whole heart segmentation dataset MM-WHS. The overall process is as follows: Figure 3 As shown, the detailed operation steps of the present invention are demonstrated, including the training phase and the inference phase.

[0073] Training phase:

[0074] Step 1.1: Read CT training images and corresponding labels from the MM-WHS dataset;

[0075] Step 1.2: The seven categories in the label are fused according to the connectivity of each heart structure to obtain a coarse-grained label with three categories. Specifically, the labels of the left atrium, left ventricle, and aorta in the original label are fused into a single category, recorded as the left blood cavity category. Similarly, the labels of the right atrium, right ventricle, and pulmonary artery are fused into a single category, recorded as the right blood cavity category. The resulting coarse-grained label has three coarse-grained categories, namely the left blood cavity, the right blood cavity, and the left myocardium.

[0076] Step 1.3: Preprocess the input chest CT image, including isotropic sampling, linear window grayscale truncation, and normalization of the CT image coordinate system. Then, randomly crop the image block with a length, width, and height of 96*96*96, and also crop the corresponding fine-grained and coarse-grained labels.

[0077] Step 1.4: Build a U-net network with two decoders. The first decoder is used to segment fine-grained labels, and the second decoder is used to segment coarse-grained labels. The loss function uses weighted cross-entropy loss.

[0078] Step 1.5: Train the multi-decoding U-net using the sampled image patches and coarse-grained and fine-grained labels;

[0079] Step 1.6: After training is complete, save the model weights;

[0080] Furthermore, in step 1.2, if Figure 4 As shown, according to the anatomical structure prior of the heart and the similarity difference between classes of different anatomical structures, the fine-grained labels marked in step 1.1 (such as Figure 5 As shown), namely left atrium, left ventricle, aorta, right atrium, right ventricle, pulmonary artery, left myocardium, perform label fusion to generate coarse-grained category labels (such as Figure 6The coarse-grained left blood cavity is composed of the fine-grained left atrium, left ventricle, and aorta, while the right blood cavity is composed of the right atrium, right ventricle, and pulmonary artery. This label fusion approach fully characterizes both coarse and fine-grained information, while whole-heart segmentation is treated as a multi-label segmentation task, where each voxel has both coarse-grained and fine-grained categories. This multi-semantic label segmentation approach aims to fully utilize the inconsistency between categories, making the model easier to learn and optimize.

[0081] Furthermore, in steps 1.3 and 2.3, the preprocessing operations include isotropic sampling, linear window grayscale truncation, and unified CT image coordinate system. The detailed operations are as follows:

[0082] 1) Isotropic sampling

[0083] To address the anisotropy of the spatial resolution of CT images, a grid sampling method is used to unify the spatial resolution to m millimeters. In this embodiment, m is set to 1, which means that the spatial resolution is unified to 1 millimeter. After sampling, the spatial interval between each voxel and its upper, lower, left, and right adjacent voxels (if any) is 1 millimeter. By achieving isotropy of spatial resolution, the area covered by the same-sized convolution kernel in the subsequent CNN segmentation model can be the same volume, solving the degradation of segmentation accuracy caused by the anisotropy of the spatial resolution of CT images.

[0084] 2) Linear window grayscale truncation

[0085] The grayscale of each voxel in a CT image reflects the absorption coefficient of the corresponding tissue structure to X-rays, and is usually expressed in HU values. The HU value usually ranges from -1000 to 1000, with the HU value of water being equal to 0 as the benchmark. -1000 is usually the HU value of air, and the HU value of bone tissue with a dense structure is usually greater than 400. By statistically analyzing the grayscale of the CT images in the MM-WHS dataset, it was found that the grayscale values of most CT images are distributed in the range of approximately -3000 to 2000. Therefore, based on the grayscale range of the heart, a linear window is added to the CT images in the dataset, and the grayscale is cropped to between -1000 and 800. Finally, each CT image is normalized to the maximum and minimum values:

[0086] 3) Unified CT image coordinate system

[0087] Different equipment or patient examination postures may cause the positive direction of the CT image coordinate system to be different. Experiments have found that this difference in coordinate system seriously affects the segmentation accuracy. We have unified the coordinate systems of all CTs into the LAS coordinate system, that is, selecting the Left, Anterior, and Superior directions as the positive directions of the three dimensions of the CT. Subsequent training and reasoning all use the LAS coordinate system, which solves the problem of poor generalization ability of the model for CT image segmentation in different coordinate systems. In step 1.4, if Figure 7 As shown in the figure, the dual-decoder 3D U-net architecture is a fully convolutional network, using a 3D CNN for the basic convolution operations. The dual-decoder 3D U-net architecture features an encoder with shared parameters and two decoders with non-shared parameters, with skip-layer connections between the encoder and decoder. One decoder performs fine-grained label segmentation, while the other performs coarse-grained label segmentation. The specific network parameters for the encoder and decoder are shown in Tables 1 and 2. Both decoders use weighted cross-entropy as the loss function for the coarse-grained and fine-grained segmentation tasks, and a joint training approach is used to optimize the model, achieving coarse-grained and fine-grained multi-label segmentation. The basic units of the network include 3D convolutional layers, pooling layers, upsampling layers, and batch norm layers. The convolution kernel size of the 3D convolutional layer is 3*3*3, which is used to extract CT image features; the pooling layer has a stride of 2 and is used to downsample the feature map, enhancing the representation of high-level semantics while reducing the amount of computation; the upsampling layer upsamples the feature map to restore the image resolution; the batch norm layer normalizes the data to accelerate convergence during the training phase. The weighted cross entropy loss formula is as follows: Among them, p ic The probability of predicting the i-th pixel as category c; y ic is the true label of the pixel, that is, when pixel i belongs to the cth class, y ic is 1, otherwise it is 0; C is the total number of categories, and N is the number of pixels in an image; N C is the number of pixels of class c in the image.

[0088] In Table 2, class_num is the number of categories to be segmented, corresponding to Figure 7 In the fine-grained segmentation decoder, class_num is 8, indicating that there are 8 classes, namely background, left atrium, left ventricle, aorta, right atrium, right ventricle, pulmonary artery, and left myocardium; corresponding to Figure 7 In the coarse-grained segmentation decoder, class_num is 4, indicating that there are 4 classes, namely background, left blood cavity, right blood cavity, and left myocardium.

[0089] Layer Type Convolution kernel Input size Output size Convolutional layer 3×3×3,16 96×96×96×1 96×96×96×16 BN+ReLU+pooling layer - 96×96×96×16 48×48×48×16 Convolutional layer 3×3×3,32 48×48×48×16 48×48×48×32 BN+ReLu+pooling layer - 48×48×48×32 24×24×24×32 Convolutional layer 3×3×3,64 24×24×24×32 24×24×24×64 BN+ReLU - 24×24×24×64 24×24×24×64 Convolutional layer 3×3×3,64 24×24×24×64 24×24×24×64 BN+ReLu+pooling layer - 24×24×24×64 12×12×12×64 Convolutional layer 3×3×3,128 12×12×12×64 12×12×12×128 BN+ReLU - 12×12×12×128 12×12×12×128 Convolutional layer 3×3×3,128 12×12×12×128 12×12×12×128 BN+ReLu+pooling layer - 12×12×12×128 6×6×6×128 Convolutional layer 3×3×3,128 6×6×6×128 6×6×6×128 BN+ReLU - 6×6×6×128 6×6×6×128 Convolutional layer 3×3×3,128 6×6×6×128 6×6×6×128 BN+ReLU - 6×6×6×128 6×6×6×128

[0090] Table 1 Encoder parameters

[0091]

[0092] Table 2 Decoder parameters

[0093] Finally, the CT image is segmented according to the following steps of the reasoning stage. The segmentation results are as follows: Figure 8 、 9 , as shown in 10.

[0094] Reasoning stage:

[0095] Step 2.1: Load the model saved in step 1.6;

[0096] Step 2.2: Input the CT image to be segmented;

[0097] Step 2.3: Preprocess the CT image, including isotropic sampling, linear window grayscale truncation, and normalization of the CT image coordinate system. Next, crop the image into k overlapping image blocks, each with a size of 96*96*96.

[0098] Step 2.4: Use the model to segment the k image blocks separately and obtain k segmentation results;

[0099] Step 2.5: Concatenate the k segmentation results to obtain the final segmentation result.

[0100] In order to implement the above embodiment, the present invention also proposes a whole heart CT segmentation device based on multiple labels and multiple decoders.

[0101] Figure 2 This is a structural diagram of a whole heart CT segmentation device based on multiple labels and multiple decoders provided by an embodiment of the present invention.

[0102] like Figure 2As shown, the whole heart CT segmentation device based on multi-label and multi-decoder includes: an acquisition module 10, a fusion module 20, a preprocessing module 30, a construction module 40, and a training module 50; wherein the acquisition module is used to acquire a training CT image, wherein the training CT image includes multiple target objects to be segmented and first label information corresponding to the multiple target objects to be segmented; the fusion module is used to fuse the first label information of the multiple target objects to be segmented according to a preset rule to obtain multiple second label information, wherein the granularity of the first label information is smaller than the granularity of the second label information; the preprocessing module is used to preprocess the CT image and crop the preprocessed CT image into multiple image blocks of preset sizes; the construction module is used to construct a U-net network, wherein the U-net network includes a first decoder and a second decoder, the first decoder is used to segment fine-grained labels, and the second decoder is used to segment coarse-grained labels; the training module is used to train the U-net network according to the image blocks, the first label information and the second label information to generate a segmentation model.

[0103] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0104] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0105] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limiting the present invention. A person skilled in the art may change, modify, replace, and modify the above embodiments within the scope of the present invention.

Claims

1. A whole heart CT segmentation method based on multi-label and multi-decoder, characterized in that: include: Acquire a training CT image, where the training CT image includes a plurality of target objects to be segmented and first label information corresponding to the plurality of target objects to be segmented; Fusing the first label information of the plurality of target objects to be segmented according to a preset rule to obtain a plurality of second label information, wherein the granularity of the first label information is smaller than the granularity of the second label information; Preprocessing the CT image, and cutting the preprocessed CT image into a plurality of image blocks of preset sizes; Constructing a U-net network, the U-net network including a first decoder and a second decoder, the first decoder is used to segment fine-grained labels, and the second decoder is used to segment coarse-grained labels; The U-net network is trained according to the image block, the first label information, and the second label information to generate a segmentation model.

2. The method according to claim 1, characterized in that After generating the segmentation model, it also includes: Acquire a CT image to be segmented; Preprocessing the CT image to be segmented and dividing it into a plurality of image blocks, respectively inputting the plurality of image blocks into the segmentation model to obtain a segmentation result of each image block; The segmentation results are spliced to generate a target segmentation result of the CT image to be segmented.

3. The method according to claim 1, wherein the CT image is preprocessed, include: Isotropic sampling, linear window grayscale truncation, and unified CT image coordinate system.

4. The method according to claim 1, wherein Also includes: Training CT images, including the entire heart structure, were collected and manually segmented and labeled to obtain the first label information. The segmented categories included left atrium, left ventricle, aorta, right atrium, right ventricle, pulmonary artery, and left myocardium.

5. The method according to claim 4, characterized in that The preset rules include: fusing the first label information to obtain second label information based on the prior knowledge of the anatomical structure of the heart and the similarity differences between classes of different anatomical structures, wherein the categories include: left blood cavity, right blood cavity, and left myocardium.

6. The method according to claim 3, characterized in that The isotropic sampling includes: The spatial resolution is unified to m millimeters, where m is selected according to the characteristics of the data and task requirements. After sampling, the spatial interval between each voxel and its upper, lower, left, and right adjacent voxels is m millimeters.

7. The method according to claim 3, characterized in that The linear window grayscale truncation includes: According to the grayscale range of the heart, a linear window is added to the CT images in the dataset, and the grayscale is clipped to between -1000 and 800. Then, each CT image is normalized to the maximum and minimum values.

8. The method according to claim 3, characterized in that The unified CT image coordinate system includes: The coordinate systems of all CTs are unified into the LAS coordinate system, and the Left, Anterior, and Superior directions are selected as the positive directions of the three dimensions of CT.

9. The method according to claim 1, characterized in that Also includes: The U-net network architecture includes an encoder and two decoders with non-shared parameters, and there is a skip-layer connection between the encoder and the two decoders; wherein, the first decoder segments the first label information, and the second decoder segments the second label information. The two decoders use weighted cross entropy as the loss function for coarse-grained and fine-grained segmentation tasks, and adopt a joint training method to optimize the segmentation model.

10. A whole heart CT segmentation device based on multi-label and multi-decoder, characterized in that: include: An acquisition module acquires a training CT image, wherein the training CT image includes a plurality of target objects to be segmented and first label information corresponding to the plurality of target objects to be segmented; a fusion module, fusing the first label information of the plurality of target objects to be segmented according to a preset rule to obtain a plurality of second label information, wherein the granularity of the first label information is smaller than the granularity of the second label information; a preprocessing module, for preprocessing the CT image and cutting the preprocessed CT image into a plurality of image blocks of preset sizes; A construction module is used to construct a U-net network, where the U-net network includes a first decoder and a second decoder, where the first decoder is used to segment fine-grained labels and the second decoder is used to segment coarse-grained labels; A training module trains the U-net network according to the image block, the first label information, and the second label information to generate a segmentation model.

Images