Coronary artery and pericoronal adipose tissue segmentation system based on self-supervised learning

Abstract

The invention discloses a coronary artery and pericoronal adipose tissue segmentation system based on self-supervised learning, and belongs to the technical field of medical image analysis. The problem that an existing supervised learning model needs to depend on a large-scale annotation data set is solved. According to the method, the CCTA image is obtained through the coronary artery CT radiography technology, and the coronary artery segmentation model is pre-trained on the unannotated CCTA image in a self-supervised learning mode, so that the model learns rich feature representation; and then coronary artery segmentation model fine tuning is carried out on annotation data, so that the dependence on a large-scale annotation data set is reduced. The pericoronal fat segmentation algorithm adopts a traditional image processing algorithm, does not need annotation, and saves the network training cost. According to the method, an efficient segmentation framework is constructed, and the problem of high cost required in existing coronary artery and peripheral adipose tissue segmentation work is solved. The method can be applied to coronary artery and pericoronal adipose tissue segmentation.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image analysis, and in particular relates to a coronary artery and pericoronary fat tissue segmentation system based on self-supervised learning. Background Art

[0002] Coronary CT Angiography (CCTA) has been widely used as a non-invasive and information-rich diagnostic tool for the assessment of coronary artery lesions. Pericoronary Adipose Tissue (PCAT) attenuation is considered a novel biomarker associated with coronary inflammation and has been shown to be associated with the severity of coronary artery lesions. PCAT attenuation assessed by CCTA is becoming a predictor of acute coronary syndrome. Therefore, reliable automated coronary artery segmentation not only facilitates anatomical assessment but also supports subsequent PCAT attenuation quantification.

[0003] The development of deep learning has introduced a new paradigm for medical image segmentation. However, the reliance of supervised learning models on large-scale annotated datasets has become a major bottleneck, especially when professional annotations of CCTA images are scarce and costly. Self-supervised learning, as an emerging branch of unsupervised learning, uses unlabeled data to extract feature representations and then fine-tunes the task using a small amount of annotated data. Therefore, how to use self-supervised learning to perform data-efficient segmentation of coronary arteries and pericoronary fat tissue is an urgent problem to be solved. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the existing supervised learning model needs to rely on large-scale labeled data sets, and propose a coronary artery and pericoronary fat tissue segmentation system based on self-supervised learning.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a coronary artery and pericoronary fat tissue segmentation system based on self-supervised learning, the system comprising a CCTA image acquisition module, a CCTA image preprocessing module, a CCTA image segmentation mask extraction module, a Transformer encoder, an image reconstruction module, a feature comparison module, a coronary artery segmentation module and a pericoronary fat tissue segmentation module; wherein:

[0006] The CCTA image acquisition module is used to acquire a CCTA image training set, and the training set includes labeled CCTA images and unlabeled CCTA images;

[0007] The CCTA image preprocessing module is used to preprocess each CCTA image in the acquired training set to obtain each preprocessed CCTA image;

[0008] The CCTA image segmentation mask extraction module is used to extract the segmentation mask of the acquired labeled CCTA image, and obtain the segmentation mask of each labeled CCTA image respectively;

[0009] The Transformer encoder, image reconstruction module and feature comparison module are jointly pre-trained based on the pre-processed CCTA images;

[0010] The coronary segmentation module and the pre-trained Transformer encoder are jointly trained using the segmentation masks of each labeled CCTA image;

[0011] After the CCTA image to be segmented is preprocessed by the CCTA image preprocessing module, the coronary artery segmentation is performed on the preprocessed CCTA image to be segmented by using the trained Transformer encoder and the coronary artery segmentation module to obtain the coronary artery segmentation result of the CCTA image to be segmented;

[0012] The pericoronary fat tissue segmentation module is used to process the coronary artery segmentation result of the CCTA image to be segmented, and obtain the pericoronary fat tissue segmentation result of the CCTA image to be segmented.

[0013] Furthermore, the CCTA image preprocessing module crops and standardizes each CCTA image in the training set to unify each CCTA image into the same size.

[0014] Furthermore, the Transformer encoder includes 12 encoding units connected in series;

[0015] For any preprocessed CCTA image, the preprocessed CCTA image is evenly divided into non-overlapping patches, and the size of each patch is P×P×P, where P is the predefined patch resolution;

[0016] Each patch obtained by division is flattened and passed through the linear projection layer, and then the output of the linear projection layer is passed through the position embedding layer, that is, the output of the linear projection layer is positionally embedded to obtain the position embedding results corresponding to each patch;

[0017] The position embedding results of all patches are then used as the input of the Transformer encoder. In the Transformer encoder, the position embedding results of all patches are sequentially passed through 12 serially connected encoding units, and the feature maps output by the 3rd encoding unit, the 6th encoding unit, the 9th encoding unit, and the 12th encoding unit are stored. That is, for each patch, a set of feature maps is obtained.

[0018] Furthermore, each encoding unit of the Transformer encoder includes a first normalization layer, a multi-head attention layer, a second normalization layer and a multi-layer perceptron;

[0019] Take the first coding unit as an example

[0020] In the first encoding unit, the input features first pass through the first normalization layer, and then the output of the first normalization layer is used as the input of the multi-head attention layer;

[0021] Then add the output of the multi-head attention layer and the input features of the first encoding unit to obtain the addition result A;

[0022] The addition result A is then passed through the second normalization layer, and the output of the second normalization layer is used as the input of the multi-layer perceptron;

[0023] Then add the output of the multilayer perceptron and A to obtain the addition result B, and use the addition result B as the output of the first encoding unit.

[0024] Furthermore, the Transformer encoder, image reconstruction module and feature comparison module are jointly pre-trained based on the pre-processed CCTA images. The specific process of pre-training is as follows:

[0025] Step 1, initialize the pre-training round t=1;

[0026] Step 2: Flatten the patches corresponding to each preprocessed CCTA image, and then pass each patch through the linear projection layer, position embedding layer and Transformer encoder, and output a set of feature maps for each patch through the Transformer encoder;

[0027] Step 3: For any patch, record the patch as z i , the stored z i The corresponding set of feature maps and z i As input to the image reconstruction module;

[0028] The working process in the image reconstruction module is:

[0029] (1) z i It passes through the first convolution layer, the first BN layer, the first ReLU activation function layer, the second convolution layer, the second BN layer, and the second ReLU activation function layer in sequence; that is

[0030] z i First, it passes through the first convolutional layer, and the output of the first convolutional layer is used as the input of the first BN layer;

[0031] The output of the first BN layer is used as the input of the first ReLU activation function layer;

[0032] Use the output of the first ReLU activation function layer as the input of the second convolutional layer;

[0033] Use the output of the second convolutional layer as the input of the second BN layer;

[0034] The output of the second BN layer is used as the input of the second ReLU activation function layer, and the output of the second ReLU activation function layer is recorded as a;

[0035] (2) The feature map output by the third encoding unit passes through the first deconvolution subunit, the second deconvolution subunit and the third deconvolution subunit in sequence; that is,

[0036] The feature map output by the third encoding unit first passes through the first deconvolution subunit, and the output of the first deconvolution subunit is used as the input of the second deconvolution subunit;

[0037] Using the output of the second deconvolution subunit as the input of the third deconvolution subunit;

[0038] The output of the third deconvolution subunit is denoted as b;

[0039] (3) The feature map output by the sixth encoding unit passes through the fourth deconvolution subunit and the fifth deconvolution subunit in sequence; that is,

[0040] The feature map output by the sixth encoding unit first passes through the fourth deconvolution subunit, and the output of the fourth deconvolution subunit is used as the input of the fifth deconvolution subunit;

[0041] The output of the fifth deconvolution subunit is denoted as c;

[0042] (4) The feature map output by the ninth encoding unit passes through the sixth deconvolution subunit, and the output of the sixth deconvolution subunit is recorded as d;

[0043] (5) passing the feature map output by the twelfth encoding unit through the first upsampling unit, and recording the output of the first upsampling unit as e;

[0044] (6) After the output e and the output d are concatenated, the concatenated result is sequentially passed through the third convolutional layer, the third BN layer, the third ReLU activation function layer, the fourth convolutional layer, the fourth BN layer, and the fourth ReLU activation function layer, that is,

[0045] The concatenated result first passes through the third convolutional layer, and the output of the third convolutional layer is used as the input of the third BN layer;

[0046] The output of the third BN layer is used as the input of the third ReLU activation function layer;

[0047] The output of the third ReLU activation function layer is used as the input of the fourth convolutional layer;

[0048] The output of the fourth convolutional layer is used as the input of the fourth BN layer;

[0049] The output of the fourth BN layer is used as the input of the fourth ReLU activation function layer;

[0050] Then the output of the fourth ReLU activation function layer passes through the second upsampling unit, and the output of the second upsampling unit is recorded as i';

[0051] (7) After the output c and the output i' are concatenated, the concatenated result is sequentially passed through the fifth convolutional layer, the fifth BN layer, the fifth ReLU activation function layer, the sixth convolutional layer, the sixth BN layer, and the sixth ReLU activation function layer, that is,

[0052] The concatenated result first passes through the fifth convolutional layer, and the output of the fifth convolutional layer is used as the input of the fifth BN layer;

[0053] The output of the fifth BN layer is used as the input of the fifth ReLU activation function layer;

[0054] The output of the fifth ReLU activation function layer is used as the input of the sixth convolutional layer;

[0055] Use the output of the sixth convolutional layer as the input of the sixth BN layer;

[0056] The output of the sixth BN layer is used as the input of the sixth ReLU activation function layer;

[0057] Then the output of the sixth ReLU activation function layer passes through the third upsampling unit, and the output of the third upsampling unit is recorded as j';

[0058] (8) After the output b and the output j' are concatenated, the concatenated result is sequentially passed through the seventh convolutional layer, the seventh BN layer, the seventh ReLU activation function layer, the eighth convolutional layer, the eighth BN layer and the eighth ReLU activation function layer, that is,

[0059] The concatenated result first passes through the seventh convolutional layer, and the output of the seventh convolutional layer is used as the input of the seventh BN layer;

[0060] The output of the seventh BN layer is used as the input of the seventh ReLU activation function layer;

[0061] The output of the seventh ReLU activation function layer is used as the input of the eighth convolutional layer;

[0062] Use the output of the eighth convolutional layer as the input of the eighth BN layer;

[0063] The output of the eighth BN layer is used as the input of the eighth ReLU activation function layer;

[0064] Then the output of the eighth ReLU activation function layer passes through the fourth upsampling unit, and the output of the fourth upsampling unit is recorded as k;

[0065] (9) After concatenating the output k and the output a, the concatenated result is sequentially passed through the ninth convolutional layer, the ninth BN layer, the ninth ReLU activation function layer, the tenth convolutional layer, the tenth BN layer, and the tenth ReLU activation function layer, that is,

[0066] The concatenated result first passes through the ninth convolutional layer, and the output of the ninth convolutional layer is used as the input of the ninth BN layer;

[0067] The output of the ninth BN layer is used as the input of the ninth ReLU activation function layer;

[0068] The output of the ninth ReLU activation function layer is used as the input of the tenth convolutional layer;

[0069] Use the output of the tenth convolutional layer as the input of the tenth BN layer;

[0070] The output of the tenth BN layer is used as the input of the tenth ReLU activation function layer;

[0071] Then use the output of the tenth ReLU activation function layer as the input of the eleventh convolutional layer, and use the output of the eleventh convolutional layer as z i The reconstruction result

[0072] Step 4: Store the z i The corresponding set of feature maps is input into the feature comparison module. In the feature comparison module, z i The corresponding feature map passes through the global average pooling layer and the linear projection layer in turn to obtain z i The feature embedding z i ′;

[0073] Step 5: Based on the reconstruction results and feature embedding z′ i Calculate the total loss function L, perform back propagation based on the total loss function L, and adjust the network parameters in the Transformer encoder, image reconstruction module, and feature comparison module;

[0074] Stop the tth round of pre-training until the total loss function L converges, and then continue to step 6;

[0075] Step 6: Determine whether the maximum number of pre-training rounds has been reached;

[0076] If achieved, the final pre-training result is obtained;

[0077] If it is not reached, each pre-processed CCTA image is re-divided into patches, and t=t+1 is set, and the process returns to step 2.

[0078] Furthermore, the first deconvolution subunit includes a deconvolution layer, a convolution layer, a BN layer and a ReLU activation function layer in sequence, that is, in the first deconvolution subunit, the input feature map first passes through the deconvolution layer, and then the output of the deconvolution layer is used as the input of the convolution layer;

[0079] Use the output of the convolutional layer as the input of the BN layer;

[0080] Use the output of the BN layer as the input of the ReLU activation function layer;

[0081] And the output of the ReLU activation function layer is used as the output of the first deconvolution subunit.

[0082] Furthermore, the specific process of step 3 is as follows:

[0083] z i ′=h(g(f))

[0084] Where f is z i The feature map of , g(f) is the feature vector output by the global average pooling layer, h represents the linear projection layer, z i ′ is the feature embedding output by the linear projection layer.

[0085] Furthermore, the total loss function L is:

[0086] L=L recon +L contrast

[0087] Among them, L recon is the reconstruction loss, L contrast It is the patch contrast loss;

[0088]

[0089] Where M is the set of all patches, ||·||2 is the L2 norm;

[0090]

[0091] Among them, z′ i Yes i The feature embedding of z + is with z i Feature embedding of another patch from the same CCTA image, sim(z i ,z + ) represents z i With z +The similarity between them, N(i) represents the difference between z i The set of all patches except i , z′ j ) represents z′ i and z′ j The similarity between j Yes j feature embedding.

[0092] Furthermore, the coronary segmentation module and the pre-trained Transformer encoder are jointly trained using the segmentation masks of each labeled CCTA image. The specific training process is as follows:

[0093] Step 1: Initialize the training round p = 1;

[0094] Step 2: Flatten the patches corresponding to each preprocessed CCTA image, and then pass each patch through the linear projection layer, position embedding layer and Transformer encoder, and output a set of feature maps for each patch through the Transformer encoder;

[0095] Step 3: For any patch, record the patch as z i , in z i In the corresponding set of feature maps, the feature maps output by the third coding unit, the sixth coding unit, the ninth coding unit, and the twelfth coding unit, as well as z i As the input of the coronary segmentation module;

[0096] Step 4: Calculate the segmentation loss L′ based on the coronary segmentation result predicted by the coronary segmentation module and the segmentation mask, perform back propagation based on the segmentation loss L′, adjust the network parameters in the Transformer encoder and the coronary segmentation module, and stop the pth round of training when the segmentation loss L′ converges;

[0097] The segmentation loss L′ is:

[0098] L′=L CE +L Dice

[0099] Among them, L CE is the cross entropy loss, L Dice is the Dice loss;

[0100]

[0101] Among them, N is the set of all patches corresponding to the labeled CCTA image, K is the number of pixels in a patch, Yesi The true label corresponding to the k-th pixel in the segmentation mask, Yes i The prediction result corresponding to the k-th pixel in ;

[0102]

[0103] Step 5: Determine whether the maximum number of training rounds has been reached;

[0104] If achieved, the final training result is obtained;

[0105] If it is not reached, each pre-processed CCTA image is re-divided into patches, and p=p+1 is set, and the process returns to step 2.

[0106] Furthermore, the working process of the pericoronal fat tissue segmentation module is as follows:

[0107] Step ①, using skeletonization algorithm to extract the coronary centerline of the coronary foreground in the segmentation result;

[0108] Step ②, according to the extracted coronary artery centerline, make a normal vector of the coronary artery centerline through each point on the coronary artery centerline, the normal vector obtained is the expansion direction of the coronary artery lumen at the corresponding point on the coronary artery centerline, and then calculate the distance from each point along the normal vector to the coronary artery lumen wall, that is, the coronary artery radius at each point on the coronary artery centerline;

[0109] Step ③, dilate the coronary artery so that the radius of the coronary artery at each point on the coronary artery centerline after dilation is doubled;

[0110] Step ④: Use the Gaussian filtering algorithm to smooth the dilated coronary artery, and then perform threshold segmentation on the smoothing result, that is, pixels with grayscale values ​​in the range of [-190HU, -30HU] are taken as pixels corresponding to the pericoronary fat tissue to obtain the pericoronary fat tissue segmentation result.

[0111] The beneficial effects of the present invention are:

[0112] The present invention obtains CCTA images through coronary CT angiography technology, and uses self-supervised learning to pre-train the coronary segmentation model on unannotated CCTA images, so that the model learns rich feature representations; then the coronary segmentation model is fine-tuned on the annotated data, thereby reducing the dependence on large-scale labeled data sets. The pericoronary fat segmentation algorithm uses traditional image processing algorithms, does not require annotations and saves network training costs. The present invention constructs an efficient segmentation framework, which solves the high cost problem required for existing coronary artery and peripheral fat tissue segmentation work, and can be widely used in clinical practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] Figure 1 It is a framework diagram of the training process of the coronary artery and pericoronary fat tissue segmentation system based on self-supervised learning of the present invention;

[0114] Figure 2 It is a structural schematic diagram of the Transformer encoder and image reconstruction module of the present invention;

[0115] Figure 3 is a flow chart of the pericoronal fat tissue segmentation algorithm. DETAILED DESCRIPTION

[0116] Specific implementation method 1: Combination Figure 1 This embodiment is described. The coronary artery and pericoronary fat tissue segmentation system based on self-supervised learning described in this embodiment includes a CCTA image acquisition module, a CCTA image preprocessing module, a CCTA image segmentation mask extraction module, a Transformer encoder, an image reconstruction module, a feature comparison module, a coronary artery segmentation module and a pericoronary fat tissue segmentation module; wherein:

[0117] The CCTA image acquisition module is used to acquire a CCTA image training set, and the training set includes labeled CCTA images and unlabeled CCTA images;

[0118] The CCTA image preprocessing module is used to preprocess each CCTA image in the acquired training set to obtain each preprocessed CCTA image;

[0119] The CCTA image segmentation mask extraction module is used to extract the segmentation mask of the acquired labeled CCTA image, and obtain the segmentation mask of each labeled CCTA image respectively;

[0120] For any labeled CCTA image, the segmentation mask of the labeled CCTA image is denoted as y gt ∈R H×w×C , C is the number of foreground categories to be segmented contained in the CCTA image, which is taken as 1 in the coronary artery segmentation task. For each pixel in the image, if the pixel belongs to the coronary artery, it is marked as 1 on the corresponding mask, otherwise it is marked as 0;

[0121] The Transformer encoder, image reconstruction module and feature comparison module are jointly pre-trained based on the pre-processed CCTA images;

[0122] The coronary segmentation module and the pre-trained Transformer encoder are jointly trained using the segmentation masks of each labeled CCTA image;

[0123] After preprocessing the CCTA image to be segmented (which can be obtained by the CCTA image acquisition module) using the CCTA image preprocessing module, the trained Transformer encoder and the coronary artery segmentation module are used to perform coronary artery segmentation on the preprocessed CCTA image to be segmented, and the coronary artery segmentation result of the CCTA image to be segmented is obtained;

[0124] The pericoronary fat tissue segmentation module is used to process the coronary artery segmentation result of the CCTA image to be segmented, and obtain the pericoronary fat tissue segmentation result of the CCTA image to be segmented.

[0125] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that the CCTA image preprocessing module crops and standardizes each CCTA image in the training set and unifies each CCTA image into the same size.

[0126] The other steps and parameters are the same as those in the first embodiment.

[0127] CCTA images are obtained using computed tomography technology. By cropping and standardizing CCTA images in sequence, a set of CCTA images with a size of 512×512×256 can be obtained. And according to actual needs, each CCTA image can be unified into other sizes other than 512×512×256.

[0128] Specific implementation method three: This implementation method is different from specific implementation method one or two in that the Transformer encoder includes 12 encoding units connected in series;

[0129] For any preprocessed CCTA image, the preprocessed CCTA image is evenly divided into non-overlapping patches (slices), and the size of each patch is P×P×P, where P is the predefined patch resolution;

[0130] Flatten each patch obtained by the division and pass it through a linear projection layer (i.e. project the flattened result into an embedding space to create a sequence X∈R N×d , where N is the number of patches and d is the embedding dimension), and then the output of the linear projection layer is passed through the position embedding layer (which is crucial for learning the relative positions between input patches), that is, the output of the linear projection layer is positionally embedded to obtain the position embedding results corresponding to each patch;

[0131] The position embedding results of all patches are then used as the input of the Transformer encoder. In the Transformer encoder, the position embedding results of all patches are sequentially passed through 12 serially connected encoding units, and the feature maps output by the 3rd encoding unit, the 6th encoding unit, the 9th encoding unit, and the 12th encoding unit are stored. That is, for each patch, a set of feature maps is obtained.

[0132] The other steps and parameters are the same as those in the first or second embodiment.

[0133] Specific implementation method 4: This implementation method is different from any one of specific implementation methods 1 to 3 in that each encoding unit of the Transformer encoder includes a first normalization layer, a multi-head attention layer, a second normalization layer and a multi-layer perceptron;

[0134] Take the first coding unit as an example

[0135] In the first encoding unit, the input features first pass through the first normalization layer, and then the output of the first normalization layer is used as the input of the multi-head attention layer;

[0136] Then add the output of the multi-head attention layer and the input features of the first encoding unit to obtain the addition result A;

[0137] The addition result A is then passed through the second normalization layer, and the output of the second normalization layer is used as the input of the multi-layer perceptron;

[0138] Then add the output of the multilayer perceptron and A to obtain the addition result B, and use the addition result B as the output of the first encoding unit.

[0139] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0140] The present invention uses the output of the first encoding unit as the input of the second encoding unit. The working process in the second encoding unit is the same as the working process in the first encoding unit. The output of the second encoding unit is used as the input of the third encoding unit, and so on. The working process of each encoding unit is the same, and the feature map output by each encoding unit is stored, that is, a set of feature maps is obtained for each patch.

[0141] Specific implementation method five: Combination Figure 2 This embodiment is different from the first to fourth embodiments in that the Transformer encoder, image reconstruction module and feature comparison module are jointly pre-trained based on the pre-processed CCTA images, and the specific process of pre-training is as follows:

[0142] Step 1, initialize the pre-training round t=1;

[0143] Step 2: After flattening the patches corresponding to each preprocessed CCTA image, each patch is passed through a linear projection layer, a position embedding layer, and a Transformer encoder, and a set of feature maps for each patch is output through the Transformer encoder (i.e., for each patch, there is a corresponding set of feature maps);

[0144] Step 3: For any patch, record the patch as z i , the stored z i The corresponding set of feature maps and z i As input to the image reconstruction module;

[0145] The working process in the image reconstruction module is:

[0146] (1) z i It passes through the first convolution layer, the first BN layer, the first ReLU activation function layer, the second convolution layer, the second BN layer, and the second ReLU activation function layer in sequence; that is

[0147] z i First, it passes through the first convolutional layer, and the output of the first convolutional layer is used as the input of the first BN layer;

[0148] The output of the first BN layer is used as the input of the first ReLU activation function layer;

[0149] Use the output of the first ReLU activation function layer as the input of the second convolutional layer;

[0150] Use the output of the second convolutional layer as the input of the second BN layer;

[0151] The output of the second BN layer is used as the input of the second ReLU activation function layer, and the output of the second ReLU activation function layer is recorded as a;

[0152] (2) The feature map output by the third encoding unit passes through the first deconvolution subunit, the second deconvolution subunit and the third deconvolution subunit in sequence; that is,

[0153] The feature map output by the third encoding unit first passes through the first deconvolution subunit, and the output of the first deconvolution subunit is used as the input of the second deconvolution subunit;

[0154] Using the output of the second deconvolution subunit as the input of the third deconvolution subunit;

[0155] The output of the third deconvolution subunit is denoted as b;

[0156] (3) The feature map output by the sixth encoding unit passes through the fourth deconvolution subunit and the fifth deconvolution subunit in sequence; that is,

[0157] The feature map output by the sixth encoding unit first passes through the fourth deconvolution subunit, and the output of the fourth deconvolution subunit is used as the input of the fifth deconvolution subunit;

[0158] The output of the fifth deconvolution subunit is denoted as c;

[0159] (4) The feature map output by the ninth encoding unit passes through the sixth deconvolution subunit, and the output of the sixth deconvolution subunit is recorded as d;

[0160] (5) The feature map output by the twelfth encoding unit passes through the first upsampling unit (i.e., the deconvolution layer), and the output of the first upsampling unit is recorded as e;

[0161] (6) After the output e and the output d are concatenated, the concatenated result is sequentially passed through the third convolutional layer, the third BN layer, the third ReLU activation function layer, the fourth convolutional layer, the fourth BN layer, and the fourth ReLU activation function layer, that is,

[0162] The concatenated result first passes through the third convolutional layer, and the output of the third convolutional layer is used as the input of the third BN layer;

[0163] The output of the third BN layer is used as the input of the third ReLU activation function layer;

[0164] The output of the third ReLU activation function layer is used as the input of the fourth convolutional layer;

[0165] The output of the fourth convolutional layer is used as the input of the fourth BN layer;

[0166] The output of the fourth BN layer is used as the input of the fourth ReLU activation function layer;

[0167] Then the output of the fourth ReLU activation function layer passes through the second upsampling unit, and the output of the second upsampling unit is recorded as i';

[0168] (7) After the output c and the output i' are concatenated, the concatenated result is sequentially passed through the fifth convolutional layer, the fifth BN layer, the fifth ReLU activation function layer, the sixth convolutional layer, the sixth BN layer, and the sixth ReLU activation function layer, that is,

[0169] The concatenated result first passes through the fifth convolutional layer, and the output of the fifth convolutional layer is used as the input of the fifth BN layer;

[0170] The output of the fifth BN layer is used as the input of the fifth ReLU activation function layer;

[0171] The output of the fifth ReLU activation function layer is used as the input of the sixth convolutional layer;

[0172] Use the output of the sixth convolutional layer as the input of the sixth BN layer;

[0173] The output of the sixth BN layer is used as the input of the sixth ReLU activation function layer;

[0174] Then the output of the sixth ReLU activation function layer passes through the third upsampling unit, and the output of the third upsampling unit is recorded as j';

[0175] (8) After the output b and the output j' are concatenated, the concatenated result is sequentially passed through the seventh convolutional layer, the seventh BN layer, the seventh ReLU activation function layer, the eighth convolutional layer, the eighth BN layer and the eighth ReLU activation function layer, that is,

[0176] The concatenated result first passes through the seventh convolutional layer, and the output of the seventh convolutional layer is used as the input of the seventh BN layer;

[0177] The output of the seventh BN layer is used as the input of the seventh ReLU activation function layer;

[0178] The output of the seventh ReLU activation function layer is used as the input of the eighth convolutional layer;

[0179] Use the output of the eighth convolutional layer as the input of the eighth BN layer;

[0180] The output of the eighth BN layer is used as the input of the eighth ReLU activation function layer;

[0181] Then the output of the eighth ReLU activation function layer passes through the fourth upsampling unit, and the output of the fourth upsampling unit is recorded as k;

[0182] (9) After concatenating the output k and the output a, the concatenated result is sequentially passed through the ninth convolutional layer, the ninth BN layer, the ninth ReLU activation function layer, the tenth convolutional layer, the tenth BN layer, and the tenth ReLU activation function layer, that is,

[0183] The concatenated result first passes through the ninth convolutional layer, and the output of the ninth convolutional layer is used as the input of the ninth BN layer;

[0184] The output of the ninth BN layer is used as the input of the ninth ReLU activation function layer;

[0185] The output of the ninth ReLU activation function layer is used as the input of the tenth convolutional layer;

[0186] The output of the tenth convolutional layer is used as the input of the tenth BN layer;

[0187] The output of the tenth BN layer is used as the input of the tenth ReLU activation function layer;

[0188] Then use the output of the tenth ReLU activation function layer as the input of the eleventh convolutional layer, and use the output of the eleventh convolutional layer as z i The reconstruction result

[0189] Step 4: Store the z i The corresponding set of feature maps is input into the feature comparison module. In the feature comparison module, z i The corresponding feature map passes through the global average pooling layer and the linear projection layer in turn to obtain z i The feature embedding z i ′;

[0190] Step 5: Based on the reconstruction results and feature embedding z′ i Calculate the total loss function L, perform back propagation based on the total loss function L, and adjust the network parameters in the Transformer encoder, image reconstruction module, and feature comparison module;

[0191] Stop the tth round of pre-training until the total loss function L converges, and then continue to step 6;

[0192] Step 6: Determine whether the maximum number of pre-training rounds has been reached;

[0193] If achieved, the final pre-training result is obtained;

[0194] If it is not reached, each pre-processed CCTA image is re-divided into patches, and t=t+1 is set, and the process returns to step 2.

[0195] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0196] Specific implementation method 6: This implementation method is different from any one of specific implementation methods 1 to 5 in that the first deconvolution subunit includes a deconvolution layer, a convolution layer, a BN layer and a ReLU activation function layer in sequence, that is, in the first deconvolution subunit, the input feature map first passes through the deconvolution layer, and then the output of the deconvolution layer is used as the input of the convolution layer;

[0197] Use the output of the convolutional layer as the input of the BN layer;

[0198] Use the output of the BN layer as the input of the ReLU activation function layer;

[0199] And the output of the ReLU activation function layer is used as the output of the first deconvolution subunit.

[0200] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0201] The structures and working processes of the second deconvolution subunit, the third deconvolution subunit, the fourth deconvolution subunit, the fifth deconvolution subunit and the sixth deconvolution subunit are the same as those of the first deconvolution subunit.

[0202] Specific implementation method 7: This implementation method is different from specific implementation methods 1 to 6 in that the specific process of step 3 is as follows:

[0203] z i ′=h(g(f))

[0204] Where f is z i The feature map of , g(f) is the feature vector output by the global average pooling layer, h represents the linear projection layer, z i ′ is the feature embedding output by the linear projection layer.

[0205] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0206] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that the total loss function L is:

[0207] L=L trcon +L contrast

[0208] Among them, L recon is the reconstruction loss, L contrast It is the patch contrast loss;

[0209]

[0210] Where M is the set of all patches, ||·||2 is the L2 norm;

[0211]

[0212] Among them, z′ i Yes i The feature embedding of z + is with z i Feature embedding of another patch from the same CCTA image, sim(z i ,z + ) represents z i With z + The similarity between them, N(i) represents the difference between z i The set of all patches except i , z′ j ) represents z′ i and z′ j The similarity between j Yes jfeature embedding.

[0213] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0214] Specific embodiment 9: This embodiment is different from any one of specific embodiments 1 to 8 in that the coronary segmentation module and the pre-trained Transformer encoder are jointly trained using the segmentation masks of each labeled CCTA image. The specific training process is as follows:

[0215] Step 1: Initialize the training round p = 1;

[0216] Step 2: Flatten the patches corresponding to each preprocessed CCTA image, and then pass each patch through the linear projection layer, position embedding layer and Transformer encoder, and output a set of feature maps for each patch through the Transformer encoder;

[0217] Step 3: For any patch, record the patch as z i , in z i In the corresponding set of feature maps, the feature maps output by the third coding unit, the sixth coding unit, the ninth coding unit, and the twelfth coding unit, as well as z i As the input of the coronary segmentation module;

[0218] Step 4: Calculate the segmentation loss L′ based on the coronary segmentation result predicted by the coronary segmentation module and the segmentation mask, perform back propagation based on the segmentation loss L′, adjust the network parameters in the Transformer encoder and the coronary segmentation module, and stop the pth round of training when the segmentation loss L′ converges;

[0219] The segmentation loss L′ is:

[0220] L′=L CE +L Dice

[0221] Among them, L CE is the cross entropy loss, L Dice is the Dice loss;

[0222]

[0223] Among them, N is the set of all patches corresponding to the labeled CCTA image, K is the number of pixels in a patch, Yes i The true label corresponding to the k-th pixel in the segmentation mask, Yes i The prediction result corresponding to the k-th pixel in ;

[0224]

[0225] Step 5: Determine whether the maximum number of training rounds has been reached;

[0226] If achieved, the final training result is obtained;

[0227] If it is not reached, each pre-processed CCTA image is re-divided into patches, and p=p+1 is set, and the process returns to step 2.

[0228] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0229] It should be noted that the structure of the coronary artery segmentation module is roughly the same as that of the image reconstruction module. Specifically, for the part before the tenth ReLU activation function layer, the structure of the coronary artery segmentation module is exactly the same as that of the image reconstruction module. The only difference is that, in the coronary artery segmentation module, the output of the tenth ReLU activation function layer passes through the upsampling unit, the ReLU activation function layer and the softmax activation function layer in sequence, and the coronary artery segmentation result is output through the softmax activation function layer.

[0230] Dice loss is used to evaluate the degree of overlap in the segmentation task. Combining cross entropy loss with Dice loss can help the segmentation model converge better during training and improve the accuracy of segmentation.

[0231] Specific implementation method ten: Combination Figure 3 The present embodiment is different from the first to ninth embodiments in that the working process of the pericoronal fat tissue segmentation module is as follows:

[0232] Step ①, using skeletonization algorithm to extract the coronary centerline of the coronary foreground in the segmentation result;

[0233] Step ②, according to the extracted coronary artery centerline, make a normal vector of the coronary artery centerline through each point on the coronary artery centerline, the normal vector obtained is the expansion direction of the coronary artery lumen at the corresponding point on the coronary artery centerline, and then calculate the distance from each point along the normal vector to the coronary artery lumen wall, that is, the coronary artery radius at each point on the coronary artery centerline;

[0234] Step ③, dilate the coronary artery so that the radius of the coronary artery at each point on the coronary artery centerline after dilation is doubled;

[0235] Step ④: Use the Gaussian filtering algorithm to smooth the dilated coronary artery, and then perform threshold segmentation on the smoothing result, that is, pixels with grayscale values ​​in the range of [-190HU, -30HU] are taken as pixels corresponding to the pericoronary fat tissue to obtain the pericoronary fat tissue segmentation result.

[0236] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0237] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A coronary artery and pericoronary fat tissue segmentation system based on self-supervised learning, characterized in that: The system includes a CCTA image acquisition module, a CCTA image preprocessing module, a CCTA image segmentation mask extraction module, a Transformer encoder, an image reconstruction module, a feature contrast module, a coronary artery segmentation module and a pericoronary fat tissue segmentation module; wherein: The CCTA image acquisition module is used to acquire a CCTA image training set, and the training set includes labeled CCTA images and unlabeled CCTA images; The CCTA image preprocessing module is used to preprocess each CCTA image in the acquired training set to obtain each preprocessed CCTA image; The CCTA image segmentation mask extraction module is used to extract the segmentation mask of the acquired labeled CCTA image, and obtain the segmentation mask of each labeled CCTA image respectively; The Transformer encoder, image reconstruction module and feature comparison module are jointly pre-trained based on the pre-processed CCTA images; The coronary segmentation module and the pre-trained Transformer encoder are jointly trained using the segmentation masks of each labeled CCTA image; After the CCTA image to be segmented is preprocessed by the CCTA image preprocessing module, the coronary artery segmentation is performed on the preprocessed CCTA image to be segmented by using the trained Transformer encoder and the coronary artery segmentation module to obtain the coronary artery segmentation result of the CCTA image to be segmented; The pericoronary fat tissue segmentation module is used to process the coronary segmentation result of the CCTA image to be segmented, and obtain the pericoronary fat tissue segmentation result of the CCTA image to be segmented.

2. The coronary artery and pericoronary fat tissue segmentation system based on self-supervised learning according to claim 1, characterized in that: The CCTA image preprocessing module crops and standardizes each CCTA image in the training set, and unifies each CCTA image into the same size.

3. The coronary artery and pericoronary fat tissue segmentation system based on self-supervised learning according to claim 2, characterized in that: The Transformer encoder includes 12 encoding units connected in series; For any preprocessed CCTA image, the preprocessed CCTA image is evenly divided into non-overlapping patches, and the size of each patch is P×P×P, where P is the predefined patch resolution; Each patch obtained by division is flattened and passed through the linear projection layer, and then the output of the linear projection layer is passed through the position embedding layer, that is, the output of the linear projection layer is positionally embedded to obtain the position embedding results corresponding to each patch; The position embedding results of all patches are then used as the input of the Transformer encoder. In the Transformer encoder, the position embedding results of all patches are sequentially passed through 12 serially connected encoding units, and the feature maps output by the 3rd encoding unit, the 6th encoding unit, the 9th encoding unit, and the 12th encoding unit are stored. That is, for each patch, a set of feature maps is obtained.

4. The coronary artery and pericoronary fat tissue segmentation system based on self-supervised learning according to claim 3, characterized in that: Each encoding unit of the Transformer encoder includes a first normalization layer, a multi-head attention layer, a second normalization layer and a multi-layer perceptron; Take the first coding unit as an example In the first encoding unit, the input features first pass through the first normalization layer, and then the output of the first normalization layer is used as the input of the multi-head attention layer; Then add the output of the multi-head attention layer and the input features of the first encoding unit to obtain the addition result A; The addition result A is then passed through the second normalization layer, and the output of the second normalization layer is used as the input of the multi-layer perceptron; Then add the output of the multilayer perceptron and A to obtain the addition result B, and use the addition result B as the output of the first encoding unit.

5. The coronary artery and pericoronary fat tissue segmentation system based on self-supervised learning according to claim 4, characterized in that: The Transformer encoder, image reconstruction module and feature comparison module are jointly pre-trained based on the pre-processed CCTA images. The specific process of pre-training is as follows: Step 1, initialize the pre-training round t = 1; Step 2: Flatten the patches corresponding to each preprocessed CCTA image, and then pass each patch through the linear projection layer, position embedding layer and Transformer encoder, and output a set of feature maps for each patch through the Transformer encoder; Step 3: For any patch, record the patch as z i , the stored z i The corresponding set of feature maps and z i As input to the image reconstruction module; The working process in the image reconstruction module is: (1) z i It passes through the first convolution layer, the first BN layer, the first ReLU activation function layer, the second convolution layer, the second BN layer, and the second ReLU activation function layer in sequence; that is z i First, it passes through the first convolutional layer, and the output of the first convolutional layer is used as the input of the first BN layer; The output of the first BN layer is used as the input of the first ReLU activation function layer; Use the output of the first ReLU activation function layer as the input of the second convolutional layer; Use the output of the second convolutional layer as the input of the second BN layer; The output of the second BN layer is used as the input of the second ReLU activation function layer, and the output of the second ReLU activation function layer is recorded as a; (2) The feature map output by the third encoding unit passes through the first deconvolution subunit, the second deconvolution subunit and the third deconvolution subunit in sequence; that is, The feature map output by the third encoding unit first passes through the first deconvolution subunit, and the output of the first deconvolution subunit is used as the input of the second deconvolution subunit; Using the output of the second deconvolution subunit as the input of the third deconvolution subunit; The output of the third deconvolution subunit is denoted as b; (3) The feature map output by the sixth encoding unit passes through the fourth deconvolution subunit and the fifth deconvolution subunit in sequence; that is, The feature map output by the sixth encoding unit first passes through the fourth deconvolution subunit, and the output of the fourth deconvolution subunit is used as the input of the fifth deconvolution subunit; The output of the fifth deconvolution subunit is denoted as c; (4) The feature map output by the ninth encoding unit passes through the sixth deconvolution subunit, and the output of the sixth deconvolution subunit is recorded as d; (5) passing the feature map output by the twelfth encoding unit through the first upsampling unit, and recording the output of the first upsampling unit as e; (6) After the output e and the output d are concatenated, the concatenated result is sequentially passed through the third convolutional layer, the third BN layer, the third ReLU activation function layer, the fourth convolutional layer, the fourth BN layer, and the fourth ReLU activation function layer, that is, The concatenated result first passes through the third convolutional layer, and the output of the third convolutional layer is used as the input of the third BN layer; The output of the third BN layer is used as the input of the third ReLU activation function layer; The output of the third ReLU activation function layer is used as the input of the fourth convolutional layer; The output of the fourth convolutional layer is used as the input of the fourth BN layer; The output of the fourth BN layer is used as the input of the fourth ReLU activation function layer; Then the output of the fourth ReLU activation function layer passes through the second upsampling unit, and the output of the second upsampling unit is recorded as i'; (7) After the output c and the output i' are concatenated, the concatenated result is sequentially passed through the fifth convolutional layer, the fifth BN layer, the fifth ReLU activation function layer, the sixth convolutional layer, the sixth BN layer, and the sixth ReLU activation function layer, that is, The concatenated result first passes through the fifth convolutional layer, and the output of the fifth convolutional layer is used as the input of the fifth BN layer; The output of the fifth BN layer is used as the input of the fifth ReLU activation function layer; The output of the fifth ReLU activation function layer is used as the input of the sixth convolutional layer; Use the output of the sixth convolutional layer as the input of the sixth BN layer; The output of the sixth BN layer is used as the input of the sixth ReLU activation function layer; Then the output of the sixth ReLU activation function layer passes through the third upsampling unit, and the output of the third upsampling unit is recorded as j'; (8) After the output b and the output j' are concatenated, the concatenated result is sequentially passed through the seventh convolutional layer, the seventh BN layer, the seventh ReLU activation function layer, the eighth convolutional layer, the eighth BN layer and the eighth ReLU activation function layer, that is, The concatenated result first passes through the seventh convolutional layer, and the output of the seventh convolutional layer is used as the input of the seventh BN layer; The output of the seventh BN layer is used as the input of the seventh ReLU activation function layer; The output of the seventh ReLU activation function layer is used as the input of the eighth convolutional layer; Use the output of the eighth convolutional layer as the input of the eighth BN layer; The output of the eighth BN layer is used as the input of the eighth ReLU activation function layer; Then the output of the eighth ReLU activation function layer passes through the fourth upsampling unit, and the output of the fourth upsampling unit is recorded as k; (9) After concatenating the output k and the output a, the concatenated result is sequentially passed through the ninth convolutional layer, the ninth BN layer, the ninth ReLU activation function layer, the tenth convolutional layer, the tenth BN layer, and the tenth ReLU activation function layer, that is, The concatenated result first passes through the ninth convolutional layer, and the output of the ninth convolutional layer is used as the input of the ninth BN layer; The output of the ninth BN layer is used as the input of the ninth ReLU activation function layer; The output of the ninth ReLU activation function layer is used as the input of the tenth convolutional layer; The output of the tenth convolutional layer is used as the input of the tenth BN layer; The output of the tenth BN layer is used as the input of the tenth ReLU activation function layer; Then use the output of the tenth ReLU activation function layer as the input of the eleventh convolutional layer, and use the output of the eleventh convolutional layer as z i The reconstruction result Step 4: Store the z i The corresponding set of feature maps is input into the feature comparison module. In the feature comparison module, z i The corresponding feature map passes through the global average pooling layer and the linear projection layer in turn to obtain z i The feature embedding z i ′; Step 5: Based on the reconstruction results and feature embedding z′ i Calculate the total loss function L, perform back propagation based on the total loss function L, and adjust the network parameters in the Transformer encoder, image reconstruction module, and feature comparison module; Stop the tth round of pre-training until the total loss function L converges, and then continue to step 6; Step 6: Determine whether the maximum number of pre-training rounds has been reached; If achieved, the final pre-training result is obtained; If it is not reached, each pre-processed CCTA image is re-divided into patches, and t=t+1 is set, and the process returns to step 2.

6. The coronary artery and pericoronary fat tissue segmentation system based on self-supervised learning according to claim 5, characterized in that: The first deconvolution subunit includes a deconvolution layer, a convolution layer, a BN layer and a ReLU activation function layer in sequence, that is, in the first deconvolution subunit, the input feature map first passes through the deconvolution layer, and then the output of the deconvolution layer is used as the input of the convolution layer; Use the output of the convolutional layer as the input of the BN layer; Use the output of the BN layer as the input of the ReLU activation function layer; And the output of the ReLU activation function layer is used as the output of the first deconvolution subunit.

7. The coronary artery and pericoronary fat tissue segmentation system based on self-supervised learning according to claim 6, characterized in that: The specific process of step 3 is as follows: z i ′=h(g(f)) Where f is z i The feature map of , g(f) is the feature vector output by the global average pooling layer, h represents the linear projection layer, z i ′ is the feature embedding output by the linear projection layer.

8. The coronary artery and pericoronary fat tissue segmentation system based on self-supervised learning according to claim 7, characterized in that: The total loss function L is: L=L recon +L contrast Among them, L recon is the reconstruction loss, L contrast It is the patch contrast loss; Where M is the set of all patches, ||·||2 is the L2 norm; Among them, z′ i Yes i The feature embedding of z + is with z i Feature embedding of another patch from the same CCTA image, sim(z i ,z + ) represents z i With z + The similarity between them, N(i) represents the difference between z i The set of all patches except i , z′ j ) represents z′ i and z′ j The similarity between j Yes j feature embedding.

9. The coronary artery and pericoronary fat tissue segmentation system based on self-supervised learning according to claim 8, characterized in that: The coronary segmentation module and the pre-trained Transformer encoder are jointly trained using the segmentation masks of each labeled CCTA image. The specific training process is as follows: Step 1: Initialize the training round p = 1; Step 2: Flatten the patches corresponding to each preprocessed CCTA image, and then pass each patch through the linear projection layer, position embedding layer and Transformer encoder, and output a set of feature maps for each patch through the Transformer encoder; Step 3: For any patch, record the patch as z i , in z i In the corresponding set of feature maps, the feature maps output by the third coding unit, the sixth coding unit, the ninth coding unit, and the twelfth coding unit, as well as z i As the input of the coronary segmentation module; Step 4: Calculate the segmentation loss L′ based on the coronary segmentation result predicted by the coronary segmentation module and the segmentation mask, perform back propagation based on the segmentation loss L′, adjust the network parameters in the Transformer encoder and the coronary segmentation module, and stop the pth round of training when the segmentation loss L′ converges; The segmentation loss L′ is: L′=L CE +L Dice Among them, L CE is the cross entropy loss, L Dice is the Dice loss; Where N is the set of all patches corresponding to the labeled CCTA image, J is the number of pixels in a patch, Yes i The true label corresponding to the k-th pixel in the segmentation mask, Yes i The prediction result corresponding to the k-th pixel in ; Step 5: Determine whether the maximum number of training rounds has been reached; If achieved, the final training result is obtained; If it is not reached, each pre-processed CCTA image is re-divided into patches, and p=p+1 is set, and the process returns to step 2.

10. The coronary artery and pericoronary fat tissue segmentation system based on self-supervised learning according to claim 9, characterized in that: The working process of the pericoronal fat tissue segmentation module is as follows: Step ①, using skeletonization algorithm to extract the coronary centerline of the coronary foreground in the segmentation result; Step ②, according to the extracted coronary artery centerline, make a normal vector of the coronary artery centerline through each point on the coronary artery centerline, the normal vector obtained is the expansion direction of the coronary artery lumen at the corresponding point on the coronary artery centerline, and then calculate the distance from each point along the normal vector to the coronary artery lumen wall, that is, the coronary artery radius at each point on the coronary artery centerline; Step ③, dilate the coronary artery so that the radius of the coronary artery at each point on the coronary artery centerline after dilation is doubled; Step ④: Use the Gaussian filtering algorithm to smooth the dilated coronary artery, and then perform threshold segmentation on the smoothing result, that is, pixels with grayscale values ​​in the range of [-190HU, -30HU] are taken as pixels corresponding to the pericoronary fat tissue to obtain the pericoronary fat tissue segmentation result.

Images