Deep learning segmentation model training method and blood vessel image segmentation method and device

Abstract

The invention discloses a training method of a deep learning segmentation model. The training method comprises the following steps: acquiring a three-dimensional blood vessel image containing an interested blood vessel segment; processing the three-dimensional blood vessel image through a to-be-trained deep learning segmentation model to obtain a middle membrane segmentation result image and an inner membrane segmentation result image; determining a loss function according to the middle membrane segmentation result label, the inner membrane segmentation result label, the middle membrane segmentation result image and the inner membrane segmentation result image, and performing parameter updating on a to-be-trained deep learning segmentation model to obtain a trained deep learning segmentation model; wherein a middle membrane segmentation device and an inner membrane segmentation device of the deep learning segmentation model adopt a cross attention mechanism to segment the three-dimensional blood vessel image, and a middle membrane segmentation result image and an inner membrane segmentation result image are obtained. According to the method, the intima and the intima of the blood vessel can be accurately segmented. The invention further discloses a training device of the deep learning segmentation model, a blood vessel image segmentation method and device, electronic equipment and a readable storage medium.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a deep learning segmentation model training method, a deep learning segmentation model training device, a vascular image segmentation method, a vascular image segmentation device, an electronic device and a computer-readable storage medium. Background Art

[0002] Cardiovascular disease is the disease with the highest morbidity and mortality in my country. Coronary computed tomography (CTA) is an important technology for detecting cardiovascular diseases. Currently, there are many methods that automatically perform vessel segmentation and centerline extraction to further calculate quantitative parameters to assist doctors in diagnosis.

[0003] In the prior art, coronary CTA image processing technology usually only focuses on single-layer vascular intima segmentation, aiming to segment the vascular intima and vascular lumen, and then directly observe the vascular condition based on the segmentation results of the intima, and then make a corresponding diagnosis. However, in areas with high plaque accumulation or severe stenosis, the intima structure is complex and varies greatly. The accuracy of the existing single intima segmentation method that only segments the intima and the vascular lumen is significantly reduced, ignoring the potential value of the vascular media, and not considering the relationship between the media and the intima during segmentation. The segmentation results are not accurate enough, resulting in inaccurate extraction of the vascular centerline, which will have an adverse impact on subsequent clinical diagnosis and treatment decisions. Summary of the invention

[0004] The present invention proposes a training method for a deep learning segmentation model to solve the above problems.

[0005] In a first aspect, an embodiment of the present invention discloses a training method for a deep learning segmentation model, wherein the deep learning segmentation model includes a media segmentor and an inner membrane segmentor, wherein the media segmentor is used to segment a media segmentation result image, and the inner membrane segmentor is used to segment an inner membrane segmentation result image, and the training method includes:

[0006] Acquire a three-dimensional vascular image including a vascular segment of interest; acquire a media segmentation result label and an intima segmentation result label of the annotated vascular segment of interest;

[0007] Processing the three-dimensional vascular image including the vascular segment of interest by the deep learning segmentation model to be trained to obtain a media segmentation result image and an intima segmentation result image;

[0008] Determine a loss function according to the media segmentation result label, the intima segmentation result label, the media segmentation result image and the intima segmentation result image of the blood vessel segment of interest;

[0009] According to the three-dimensional vascular image containing the vascular segment of interest, the media segmentation result image, the intima segmentation result image, the media segmentation result label, the intima segmentation result label and the loss function, the parameters of the deep learning segmentation model to be trained are updated until the deep learning segmentation model to be trained meets the preset training conditions, thereby obtaining a trained deep learning segmentation model;

[0010] Among them, the media segmenter and the endothelium segmenter use a cross-attention mechanism to segment the three-dimensional vascular image containing the vascular segment of interest, and obtain the media segmentation result image and the endothelium segmentation result image.

[0011] By adopting the above technical scheme, the training method of the deep learning segmentation model of the present invention is combined with the cross-attention mechanism, through which the middle membrane features and the inner membrane features can interact, so that the trained deep learning segmentation model can more accurately and simultaneously segment the middle membrane segmentation result image and the inner membrane segmentation result image of the blood vessel segment of interest, thereby improving the segmentation accuracy of the middle membrane and the inner membrane.

[0012] According to another specific embodiment of the present invention, the process of segmenting a three-dimensional blood vessel image containing a blood vessel segment of interest using a cross-attention mechanism specifically includes:

[0013] Extracting the media features of the three-dimensional vascular image containing the vascular segment of interest by a media segmenter to obtain media feature data;

[0014] Extracting intimal features of a three-dimensional vascular image including a vascular segment of interest by an intimal segmentator to obtain intimal feature data;

[0015] The middle membrane segmenter fuses the feature data of the middle membrane and the feature data of the inner membrane through the cross attention mechanism to obtain the result image of the middle membrane segmentation;

[0016] The endometrium segmenter fuses the middle membrane feature data and the endometrium feature data through a cross-attention mechanism to obtain the endometrium segmentation result image.

[0017] According to another specific embodiment of the present invention, the loss function includes a cross entropy loss function, a Dice loss function and a consistency loss function.

[0018] According to another specific embodiment of the present invention, the consistency loss function is obtained in the following way:

[0019] According to the middle membrane segmentation result image, a soft skeleton extraction method is used to obtain the middle membrane skeletonization result data of the middle membrane segmentation result image, and according to the inner membrane segmentation result image, a soft skeleton extraction method is used to obtain the inner membrane skeletonization result data of the inner membrane segmentation result image;

[0020] The consistency loss function is determined according to the middle membrane skeletonization result data and the inner membrane skeletonization result data.

[0021] According to another specific embodiment of the present invention, the loss function is specifically:

[0022] Total Loss = Dice Lumen +Dice Wall +CE Lumen +CE Wall +Consistency lumen-wall

[0023]

[0024] Among them, Total Loss represents the loss function, Dice Lumen +Dice Wall represents the Dice loss function, CE Lumen +CE Wall represents the cross entropy loss function, Consistency lumen-wall represents the consistency loss function, G L Indicates the result label of the inner membrane segmentation, P L represents the image of the intima segmentation result, S L Indicates the inner membrane skeletonization result data, G W Indicates the result label of the middle membrane segmentation, P W represents the result image of the middle membrane segmentation, S W Represents the medial membrane skeletonization result data.

[0025] According to another specific embodiment of the present invention, the consistency loss function is specifically:

[0026] S l =SoftSkeleton(P l )

[0027] S w =SoftSkeleton(P w )

[0028]

[0029] Among them, P I represents the image of the intima segmentation result, S I represents the inner membrane skeletonization result data, P W represents the result image of the middle membrane segmentation, S W Indicates the result data of the middle membrane skeletonization, SoftSkeleton indicates the soft skeleton extraction method, and Consistency lumen-wall represents the consistency loss function.

[0030] According to another specific embodiment of the present invention, acquiring a three-dimensional blood vessel image including a blood vessel segment of interest specifically includes:

[0031] acquiring an unprocessed three-dimensional vascular image of a vascular segment of interest;

[0032] The unprocessed three-dimensional vascular image is normalized and data enhanced to obtain a three-dimensional vascular image.

[0033] According to another specific embodiment of the present invention, the deep learning segmentation model further includes a feature extractor, and the loss function is also used to train the feature extractor. The three-dimensional vascular image including the vascular segment of interest is processed by the deep learning segmentation model to be trained, specifically including:

[0034] An attention mechanism is used to extract vascular features from a three-dimensional vascular image containing a vascular segment of interest through a feature extractor to obtain feature data of the three-dimensional vascular image of the vascular segment of interest;

[0035] The attention mechanism includes a spatial attention mechanism and / or a channel attention mechanism;

[0036] The spatial attention mechanism is used to extract the spatial feature data of the vessel of interest in the three-dimensional vessel image containing the vessel of interest;

[0037] The channel attention mechanism is used to extract the channel feature data of the vessel of interest of the vessel segment of interest in the three-dimensional vessel image containing the vessel segment of interest.

[0038] In a second aspect, an embodiment of the present invention discloses a training device for a deep learning segmentation model, including: the deep learning segmentation model includes a middle membrane segmentor and an inner membrane segmentor, the middle membrane segmentor is used to segment the middle membrane segmentation result image, the inner membrane segmentor is used to segment the inner membrane segmentation result image, and the training device includes:

[0039] An image acquisition module is used to acquire a three-dimensional blood vessel image including a blood vessel segment of interest, and to acquire a media segmentation result label and an intima segmentation result label of the annotated blood vessel segment of interest;

[0040] A pre-segmentation module is used to process the three-dimensional vascular image containing the vascular segment of interest through the deep learning segmentation model to be trained to obtain a media segmentation result image and an intima segmentation result image;

[0041] A loss function determination module is used to determine a loss function according to the media segmentation result label, the intima segmentation result label, the media segmentation result image and the intima segmentation result image of the blood vessel segment of interest;

[0042] A training module is used to update the parameters of the deep learning segmentation model to be trained according to the three-dimensional vascular image containing the vascular segment of interest, the media segmentation result image, the intima segmentation result image, the media segmentation result label, the intima segmentation result label and the loss function, until the deep learning segmentation model to be trained meets the preset training conditions, thereby obtaining a trained deep learning segmentation model;

[0043] Among them, the media segmenter and the endothelium segmenter use a cross-attention mechanism to segment the three-dimensional vascular image containing the vascular segment of interest, and obtain the media segmentation result image and the endothelium segmentation result image.

[0044] By adopting the above technical scheme, the training device of the deep learning segmentation model of the present invention is combined with the cross-attention mechanism, through which the middle membrane features and the inner membrane features can interact, so that the trained deep learning segmentation model can more accurately and simultaneously segment the middle membrane segmentation result image and the inner membrane segmentation result image of the blood vessel segment of interest, thereby improving the segmentation accuracy of the middle membrane and the inner membrane.

[0045] According to another specific embodiment of the present invention, in the process of processing the three-dimensional vascular image containing the vascular segment of interest by the deep learning segmentation model to be trained, the pre-segmentation module segments the three-dimensional vascular image containing the vascular segment of interest by using the media segmenter and the intima segmenter using a cross-attention mechanism to obtain a media segmentation result image and an intima segmentation result image;

[0046] The process of segmenting a 3D vascular image containing a vascular segment of interest using a cross-attention mechanism specifically includes:

[0047] Extracting the media features of the three-dimensional vascular image containing the vascular segment of interest by a media segmenter to obtain media feature data;

[0048] Extracting intimal features of a three-dimensional vascular image including a vascular segment of interest by an intimal segmentator to obtain intimal feature data;

[0049] The middle membrane segmenter fuses the feature data of the middle membrane and the feature data of the inner membrane through the cross attention mechanism to obtain the result image of the middle membrane segmentation;

[0050] The endometrium segmenter fuses the middle membrane feature data and the endometrium feature data through a cross-attention mechanism to obtain the endometrium segmentation result image.

[0051] In a third aspect, an embodiment of the present invention discloses a vascular image segmentation method, wherein the vascular image is segmented by a deep learning segmentation model, wherein the deep learning segmentation model includes a media segmentor and an intima segmentor, wherein the media segmentor is used to segment a media segmentation result image, and the intima segmentor is used to segment an intima segmentation result image, wherein the vascular image segmentation method includes:

[0052] Acquire three-dimensional vascular image feature data of a vascular segment of interest;

[0053] The deep learning segmentation model uses a cross-attention mechanism to segment the three-dimensional vascular image feature data of the vascular segment of interest, and obtains the media segmentation result image and the intima segmentation result image of the vascular segment of interest.

[0054] By adopting the above technical solution, the blood vessel image segmentation method of the present invention can improve the segmentation accuracy of the middle membrane and the inner membrane by adopting a cross-attention mechanism and making full use of the interactive relationship between the middle membrane and the inner membrane.

[0055] According to another specific embodiment of the present invention, the process of segmenting the three-dimensional vascular image feature data using the cross attention mechanism specifically includes:

[0056] Extracting the middle membrane features of the three-dimensional blood vessel image feature data by a middle membrane segmenter to obtain middle membrane feature data;

[0057] Extracting the intima features of the three-dimensional blood vessel image feature data by an intima segmenter to obtain intima feature data;

[0058] The middle membrane segmenter fuses the feature data of the middle membrane and the feature data of the inner membrane through the cross attention mechanism to obtain the result image of the middle membrane segmentation;

[0059] The endometrium segmenter fuses the middle membrane feature data and the endometrium feature data through a cross-attention mechanism to obtain the endometrium segmentation result image.

[0060] According to another specific embodiment of the present invention, the deep learning segmentation model further includes a feature extractor for acquiring three-dimensional vascular image feature data of a vascular segment of interest, specifically including:

[0061] acquiring a three-dimensional vascular image including a vascular segment of interest;

[0062] Acquiring three-dimensional vascular image feature data of the vascular segment of interest according to the three-dimensional vascular image containing the vascular segment of interest, including: extracting vascular features from the three-dimensional vascular image of the vascular segment of interest through a feature extractor using an attention mechanism to obtain the three-dimensional vascular image feature data of the vascular segment of interest;

[0063] The attention mechanism includes a spatial attention mechanism and / or a channel attention mechanism;

[0064] The spatial attention mechanism is used to extract the spatial feature data of the vessel of interest in the 3D vessel image.

[0065] The channel attention mechanism is used to extract the channel feature data of the vessel of interest of the vessel segment of interest in the three-dimensional vessel image.

[0066] According to another specific embodiment of the present invention, acquiring a three-dimensional blood vessel image including a blood vessel segment of interest specifically includes:

[0067] acquiring an unprocessed three-dimensional vascular image of a vascular segment of interest;

[0068] The unprocessed three-dimensional vascular image is normalized to obtain a three-dimensional vascular image of the vascular segment of interest.

[0069] According to another specific embodiment of the present invention, an image enhancement optimization operation is performed on the media segmentation result image and the intima segmentation result image to obtain an enhanced final media segmentation result image and a final intima segmentation result image;

[0070] Among them, the image enhancement optimization operation includes morphological operation and / or connectivity analysis, the morphological operation is selected from one or more of the following: corrosion operation, expansion operation, opening operation, closing operation, morphological gradient, top hat and black hat, and the connectivity analysis is selected from one or more of the following: connected component labeling, region growing, boundary tracing, hole filling, skeleton extraction and region segmentation.

[0071] In a fourth aspect, an embodiment of the present invention discloses a vascular image segmentation device, which segments a vascular image by a deep learning segmentation model, wherein the deep learning segmentation model includes a media segmentor and an intima segmentor, wherein the media segmentor is used to segment a media segmentation result image, and the intima segmentor is used to segment an intima segmentation result image, and the vascular image segmentation device includes:

[0072] A feature acquisition module, used to acquire three-dimensional vascular image feature data of a vascular segment of interest;

[0073] An image segmentation module is used to segment the three-dimensional vascular image feature data of the vascular segment of interest by using a deep learning segmentation model and a cross-attention mechanism to obtain a media segmentation result image and an intima segmentation result image of the vascular segment of interest;

[0074] By adopting the above technical solution, the blood vessel image segmentation device of the present invention can improve the segmentation accuracy of the middle membrane and the inner membrane by adopting a cross-attention mechanism and making full use of the interactive relationship between the middle membrane and the inner membrane.

[0075] According to another specific embodiment of the present invention, the image segmentation module uses the cross attention mechanism to segment the three-dimensional blood vessel image feature data, specifically including:

[0076] Extracting the middle membrane features of the three-dimensional blood vessel image feature data by a middle membrane segmenter to obtain middle membrane feature data;

[0077] Extracting the intima features of the three-dimensional blood vessel image feature data by an intima segmenter to obtain intima feature data;

[0078] The middle membrane segmenter fuses the feature data of the middle membrane and the feature data of the inner membrane through the cross attention mechanism to obtain the result image of the middle membrane segmentation;

[0079] The endometrium segmenter fuses the middle membrane feature data and the endometrium feature data through a cross-attention mechanism to obtain the endometrium segmentation result image.

[0080] In a fifth aspect, an embodiment of the present invention discloses an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, and when the at least one instruction is executed by the processor, implements the training method of the deep learning segmentation model in any of the aforementioned embodiments, and / or the vascular image segmentation method in any of the aforementioned embodiments.

[0081] By adopting the above technical solution, the electronic device can combine the cross-attention mechanism and, through the interaction of the middle membrane feature data and the inner membrane feature data, enable the trained deep learning segmentation model to more accurately and simultaneously segment the middle membrane segmentation result image and the inner membrane segmentation result image of the blood vessel segment of interest, thereby improving the segmentation accuracy of the middle membrane and the inner membrane.

[0082] In a sixth aspect, an embodiment of the present invention discloses a computer-readable storage medium, in which at least one instruction is stored, and when the at least one instruction is executed, the training method of the deep learning segmentation model in any of the aforementioned embodiments and / or the vascular image segmentation method in any of the aforementioned embodiments is implemented.

[0083] By adopting the above technical solution, the computer-readable storage medium can be combined with the cross-attention mechanism, and through the interaction of the middle membrane feature data and the inner membrane feature data, the trained deep learning segmentation model can more accurately and simultaneously segment the middle membrane segmentation result image and the inner membrane segmentation result image of the blood vessel segment of interest, thereby improving the segmentation accuracy of the middle membrane and the inner membrane.

[0084] It should be noted that the training method of a deep learning segmentation model and the vascular image segmentation method of the present invention do not involve surgical methods, but are based on the processing of medical images, that is, their direct objects are medical images, such as three-dimensional coronary CTA images. The training method of a deep learning segmentation model and the vascular image segmentation method of the present invention have non-therapeutic purposes.

[0085] The training device of the deep learning segmentation model, the vascular image segmentation device, the electronic device and the computer-readable storage medium of the present invention are also not devices related to surgical methods, but are related devices based on the processing of medical images, that is, their direct objects are medical images, such as three-dimensional coronary CTA images. The training device of the deep learning segmentation model, the vascular image segmentation device, the electronic device and the computer-readable storage medium of the present invention all have non-therapeutic purposes.

[0086] The beneficial effects of the present invention are as follows:

[0087] 1. The training method and the training device of the deep learning segmentation model provided by the present invention introduce a cross-attention mechanism into the deep learning segmentation model, and promote the middle membrane segmentation and the inner membrane segmentation through the interaction between the middle membrane feature data and the inner membrane feature data. The trained deep learning segmentation model can simultaneously segment the middle membrane segmentation results and the inner membrane segmentation results of the blood vessel segment of interest, thereby improving the accuracy of the middle membrane and the inner membrane segmentation.

[0088] During the training process of the deep learning segmentation model, the present invention also designs a consistency loss function, through which the relationship between the middle membrane and the inner membrane is established to ensure the coordination between the subsequent segmentation results of the inner membrane and the middle membrane.

[0089] In addition, the present invention also performs normalization and data enhancement processing on the unprocessed three-dimensional vascular images containing the vascular segments of interest used in the training process of the deep learning segmentation model, which can expand the amount of data available during training, improve data diversity, and enable the trained deep learning segmentation model to have better generalization ability.

[0090] 2. The vascular image segmentation method and vascular image segmentation device provided by the present invention perform feature fusion of the middle membrane feature data and the endothelium feature data through a cross-attention mechanism, so that the middle membrane feature data and the endothelium feature data interact with each other, promote the segmentation of the middle membrane and the endothelium, and improve the segmentation accuracy of the middle membrane and the endothelium.

[0091] The present invention also introduces a spatial attention mechanism and / or a channel attention mechanism when extracting the three-dimensional vascular image feature data of the vascular segment of interest, which can more accurately locate the spatial position of the vascular segment of interest in the image, and can also increase the weight of the feature channel corresponding to the important vascular features, highlight the vascular segment of interest, and enhance the deep learning segmentation model's ability to characterize the key features of the target blood vessel, which helps to improve the accuracy of the segmentation of the tunica media and tunica intima.

[0092] In addition, after obtaining the endothelial segmentation result image and the media segmentation result image, the present invention also performs image enhancement optimization operations, and uses morphological operations and / or connectivity analysis to fill possible voids inside the segmented blood vessels, remove noise generated during segmentation, and further improve the accuracy and robustness of the media and endothelial segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 Schematic diagram showing the process of training the deep learning segmentation model in an embodiment of the present invention Figure 1 ;

[0094] Figure 2 Schematic diagram showing the process of training the deep learning segmentation model in an embodiment of the present invention Figure 2 ;

[0095] Figure 3 Schematic diagram showing the process of training the deep learning segmentation model in an embodiment of the present invention Figure 3 ;

[0096] Figure 4 Schematic diagram showing the process of training the deep learning segmentation model in an embodiment of the present invention Figure 4 ;

[0097] Figure 5 The flow chart of the blood vessel image segmentation method in the embodiment of the present invention is shown as follows: Figure 1 ;

[0098] Figure 6 The flow chart of the blood vessel image segmentation method in the embodiment of the present invention is shown as follows: Figure 2 ;

[0099] Figure 7 The flow chart of the blood vessel image segmentation method in the embodiment of the present invention is shown as follows: Figure 3 ;

[0100] Figure 8 The flow chart of the blood vessel image segmentation method in the embodiment of the present invention is shown as follows: Figure 4 ;

[0101] Fig. 9 A schematic diagram showing the structure of a training device for a deep learning segmentation model in an embodiment of the present invention is shown;

[0102] Fig.10 A schematic diagram showing the structure of a blood vessel image segmentation device according to an embodiment of the present invention is shown;

[0103] Fig.11 A schematic diagram showing the structure of an electronic device in an embodiment of the present invention is shown;

[0104] Fig.12 A schematic diagram showing the structure of a deep learning segmentation model in an embodiment of the present invention is shown;

[0105] Fig.13 The original three-dimensional coronary CTA section image in the embodiment of the present invention is shown;

[0106] Fig.14 The three-dimensional coronary CTA section image obtained after normalization processing of the original three-dimensional coronary CTA section image in an embodiment of the present invention is shown;

[0107] Fig.15 An image obtained after enhancing the normalized three-dimensional coronary CTA section image data in an embodiment of the present invention is shown;

[0108] Fig.16 The result image of the media segmentation result image and the intima segmentation result image superimposed on the original three-dimensional blood vessel image in an embodiment of the present invention is shown;

[0109] Fig.17 A schematic diagram showing a three-dimensional visualization result of the segmented middle membrane and the segmented inner membrane in an embodiment of the present invention;

[0110] Fig.18 A plan view cut along the center line of a blood vessel in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0111] The following specific embodiments illustrate the implementation of the present invention, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may extend based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many specific details. The present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present invention, some specific details will be omitted in the description. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0112] It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0113] The terms “first”, “second”, etc. are only used for distinguishing descriptions and should not be understood as indicating or implying relative importance.

[0114] In the description of this embodiment, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this embodiment can be understood according to specific circumstances.

[0115] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0116] The inventors have carefully studied the prior art and found that the prior art only focuses on the segmentation between the vascular intima and the vascular lumen but not the segmentation between the vascular intima and the vascular media. The prior art segmentation is not accurate enough, which is not conducive to the subsequent accurate extraction of the centerline and will affect the subsequent clinical diagnosis of cardiovascular diseases. In addition, since the media and the intima are closely connected in tissue structure, the boundary between the media and the intima is not easy to distinguish in the image. Different areas of the vascular intima and the vascular media may also have different thicknesses. The vascular morphology may also make the media and the intima more difficult to distinguish. The above situations will increase the complexity of image segmentation. It is difficult for the existing vascular image segmentation algorithm to accurately segment the vascular intima and the vascular media. The inventors hope to solve the above problems.

[0117] In a first aspect, an embodiment of the present invention discloses a training method for a deep learning segmentation model, wherein the deep learning segmentation model includes a middle membrane segmentor and an inner membrane segmentor, wherein the middle membrane segmentor is used to segment a middle membrane segmentation result image, and the inner membrane segmentor is used to segment an inner membrane segmentation result image.

[0118] refer to Figure 1 Combined with Fig.12 , the training method of the deep learning segmentation model specifically includes the following steps:

[0119] S1: Acquire a three-dimensional vascular image including a vascular segment of interest, and obtain a media segmentation result label and an intima segmentation result label of the annotated vascular segment of interest.

[0120] Specifically, the three-dimensional vascular image is a three-dimensional coronary CTA image, which can clearly observe the condition of the heart vessels. The media segmentation result label of the vascular segment of interest is, for example, the media segmentation result manually annotated in advance, and the intima segmentation result label is, for example, the intima segmentation result manually annotated in advance.

[0121] S2: Processing the three-dimensional vascular image including the vascular segment of interest by the deep learning segmentation model to be trained to obtain a media segmentation result image and an intima segmentation result image.

[0122] Among them, the media segmenter and the endothelium segmenter use a cross-attention mechanism to segment the three-dimensional vascular image containing the vascular segment of interest, and obtain the media segmentation result image and the endothelium segmentation result image.

[0123] More specifically, in the process of segmenting a three-dimensional vascular image containing a vascular segment of interest using a cross-attention mechanism, the middle membrane features of the three-dimensional vascular image containing the vascular segment of interest are extracted by a middle membrane segmentor to obtain middle membrane feature data; the endothelium features of the three-dimensional vascular image containing the vascular segment of interest are extracted by an endothelium segmentor to obtain endothelium feature data; the middle membrane segmentor performs feature fusion of the middle membrane feature data and the endothelium feature data through a cross-attention mechanism to obtain a middle membrane segmentation result image; the endothelium segmentor performs feature fusion of the middle membrane feature data and the endothelium feature data through a cross-attention mechanism to obtain an endothelium segmentation result image.

[0124] The Cross-Attention Mechanism is a mechanism used in deep learning neural networks to process sequence-to-sequence (Seq2Seq) tasks, especially in the interaction between the encoder and decoder of deep learning neural networks. Its core idea is that the Cross-Attention Mechanism can be used in both the encoder and the decoder to interact, fuse and update features from different objects (specifically, the middle membrane features and the inner membrane features in this invention).

[0125] In the present invention, the application of cross attention is further improved. Specifically, the media segmenter and the endothelium segmenter are both decoders. The present invention segments the three-dimensional vascular image through the cross attention mechanism. The cross attention mechanism can make the media segmenter and the endothelium segmenter promote each other's learning when segmenting the media and the endothelium, respectively, and establish a mutual dependence between the endothelium and the media, so that the media segmenter can combine the vascular endothelium features during the endothelium segmentation to perform feature learning and decoding on the vascular media features, and the endothelium segmenter can combine the vascular media features during the media segmentation to perform feature learning and decoding on the vascular endothelium features, and then the media segmenter and the endothelium segmenter can simultaneously segment and obtain the media segmentation results and the endothelium segmentation results respectively. In this way, the mutual relationship between the vascular media and the vascular endothelium can be fully utilized, and the efficient feature interaction between the vascular media and the vascular endothelium can be realized, which can improve the expression ability of the deep learning segmentation model and improve the segmentation accuracy of the media segmenter and the endothelium segmenter for the vascular media and the vascular endothelium.

[0126] S3: determining a loss function according to the media segmentation result label, the intima segmentation result label, the media segmentation result image and the intima segmentation result image of the blood vessel segment of interest.

[0127] S4: Update the parameters of the deep learning segmentation model to be trained according to the three-dimensional vascular image containing the vascular segment of interest, the media segmentation result image, the intima segmentation result image, the media segmentation result label, the intima segmentation result label and the loss function, until the deep learning segmentation model to be trained meets the preset training conditions, thereby obtaining a trained deep learning segmentation model.

[0128] During the entire training process, the media segmentation result label and the intima segmentation result label are the segmentation labels. The media segmentation result label is the standard media segmentation result, that is, the expected media segmentation result. The intima segmentation result label is the standard intima segmentation result, that is, the expected intima segmentation result. During training, the three-dimensional vascular image and its corresponding media segmentation result label and intima segmentation result label are input into the deep learning segmentation model to update the parameters of the deep learning segmentation model and learn the segmentation process until the preset training conditions are met.

[0129] The preset training condition refers to the loss function decreasing to convergence (i.e., the loss function no longer decreases when training is continued). When the preset training condition is met, it is considered that the deep learning segmentation model has been able to accurately segment the media segmentation results and the endometrium segmentation results. The preset training condition can be a specific number of training times, which can be determined based on the amount of data used for training and the specific strategy during training.

[0130] By adopting the above technical scheme, the training method of the deep learning segmentation model of the present invention is combined with the cross-attention mechanism, and through the interaction of the middle membrane feature data and the inner membrane feature data, the trained deep learning segmentation model can more accurately and simultaneously segment the middle membrane segmentation result image and the inner membrane segmentation result image of the blood vessel segment of interest. The present invention trains a deep learning segmentation model with an end-to-end processing framework, and directly inputs a three-dimensional blood vessel image containing the blood vessel segment of interest into the deep learning segmentation model, so that the middle membrane and the inner membrane of the blood vessel can be processed simultaneously, and the corresponding middle membrane segmentation results and inner membrane segmentation results can be obtained. In the above embodiments, the deep learning segmentation model of the present invention can optimize and guide the precise segmentation of the inner membrane by utilizing the relative stability of the middle membrane and the potential topological relationship between the middle membrane and the inner membrane, combined with the cross-attention mechanism, and can also promote the precise segmentation of the middle membrane at the same time, and improve the segmentation accuracy of the middle membrane and the inner membrane. In addition, the deep learning segmentation model of the present invention with an end-to-end deep learning framework can avoid complex artificial feature design and multi-stage processing, and simplify the workflow.

[0131] Furthermore, in the above embodiment, the loss function in step S3 includes a cross entropy loss function, a Dice loss function and a consistency loss function.

[0132] Specifically, the cross entropy loss function is used to measure the difference between the actual output (from the model) and the target output (true label). In the present invention, it specifically measures the difference between the media segmentation results and the inner membrane segmentation results actually segmented by the deep learning segmentation model and the media segmentation labels and the inner membrane segmentation labels (i.e., the media segmentation results and the inner membrane segmentation results expected to be obtained).

[0133] In the image segmentation task, the Dice loss function can better handle the problem of large difference in the number of pixels between the foreground and background of the image, and improve the segmentation accuracy. In the present invention, the foreground is the vascular media and the vascular intima, and the background is the surrounding environment where the blood vessel is located. The Dice loss function is conducive to accurately distinguishing the vascular intima, the vascular media and the environment where the blood vessel is located, and improving the segmentation accuracy of the vascular intima and the vascular media.

[0134] Fig.18 The figure shows a plan view cut along the center line of the blood vessel, where the yellow part represents the intima of the blood vessel and the orange part represents the tunica media of the blood vessel. Fig.18 It can be seen that the middle membrane is thicker than the inner membrane, and the area displayed on the image is larger and more continuous, which is more conducive to segmentation.

[0135] The segmentation result of the middle membrane is relatively stable because the lumen is thicker and the diameter changes less. The consistency loss function (i.e., consistency constraint) is proposed in the process of simultaneous segmentation of the middle membrane and the intima in order to use the continuity of the middle membrane to constrain the continuity of the intima. Since the segmentation of the middle membrane is relatively stable, establishing the relationship between the middle membrane and the intima through consistency constraint helps to ensure the coordination between the segmentation results of the intima and the middle membrane. Figure 2 , the consistency loss function is obtained by the following steps:

[0136] S310: obtaining media skeletonization result data of the media segmentation result image by a soft skeleton extraction method according to the media segmentation result image, and obtaining endo skeletonization result data of the endo segmentation result image by a soft skeleton extraction method according to the endo segmentation result image.

[0137] Among them, the soft skeleton extraction method is the soft-skeleton algorithm. The soft-skeleton algorithm is a differentiable method for calculating the skeletonization result of the segmentation result. The result obtained by soft-skeleton is similar to that of the conventional skeletonization algorithm, but soft-skeleton adopts a differentiable operation, which is convenient for calculating the loss function.

[0138] S320: Determine a consistency loss function according to the middle membrane skeletonization result data and the inner membrane skeletonization result data.

[0139] Among them, a more specific way to determine the consistency loss function is: calculate the dot product through the middle membrane skeletonization result data and the inner membrane skeletonization result data, and then divide it by the sum of the pixel points of the middle membrane skeletonization result data, so as to calculate the continuity of the inner membrane relative to the middle membrane. The continuity of the inner membrane relative to the middle membrane is the consistency loss function.

[0140] The middle membrane is thicker, and it is easier to maintain the continuity of the middle membrane segmentation results. Therefore, the consistency loss function can be used to make the segmentation results of the middle membrane and the inner membrane consistent in continuity, so as to promote the continuity of the inner membrane segmentation results.

[0141] The above method is used to determine the consistency loss function. First, the soft-skeleton is used to obtain the center line of the middle membrane, and the consistency loss function is used to calculate the consistency of the connectivity between the inner membrane and the middle membrane. This method is not easily affected by the different sizes of blood vessels in places with different blood vessel thicknesses, which helps to segment the middle membrane and the inner membrane more accurately.

[0142] Among them, the consistency loss function can be specifically:

[0143] S l =SoftSkeleton(P l )(Formula 1)

[0144] S w =SoftSkeleton(P w )(Formula 2)

[0145]

[0146] Among them, P I represents the image of the intima segmentation result, S I represents the inner membrane skeletonization result data, P W represents the result image of the middle membrane segmentation, S W Represents the skeletonization result data of the middle membrane, SoftSkeleton represents the soft skeleton extraction method (i.e., the function of the aforementioned soft-skeleton algorithm), Consistency lumen-wall represents the consistency loss function.

[0147] In the above embodiments, the loss function may specifically be:

[0148] Total Loss = Dice Lumen +Dice Wall +CELumen +CE Wall +Consistency lumen-wall (male

[0149] Formula 4)

[0150]

[0151] Among them, Total Loss represents the loss function, Dice Lumen +Dice Wall represents the Dice loss function, CE Lumen +CE Wall represents the cross entropy loss function, Consistency lumen-wall represents the consistency loss function, G L Indicates the result label of the inner membrane segmentation, P L represents the image of the intima segmentation result, S L Indicates the inner membrane skeletonization result data, G W Indicates the result label of the middle membrane segmentation, P W represents the result image of the middle membrane segmentation, S W Represents the medial membrane skeletonization result data.

[0152] Further, refer to Figure 3 , the aforementioned step S1 specifically includes the following steps:

[0153] S11: Acquire an unprocessed three-dimensional vascular image of a vascular segment of interest.

[0154] Fig.13 FIG. 4 shows an original three-dimensional coronary CTA section image, which is an example of an unprocessed three-dimensional blood vessel image.

[0155] S12: performing normalization processing and data enhancement processing on the unprocessed three-dimensional vascular image to obtain a three-dimensional vascular image.

[0156] Normalization and data enhancement can improve image quality and data diversity. Specifically, normalization refers to scaling the image pixel values ​​to the range of 0 to 1, which is more conducive to model training and prediction. Data enhancement includes random rotation and random noise addition. Data enhancement is only used during model training. Data enhancement can be used to increase data as much as possible, that is, by randomly changing the contrast of the image, randomly cropping, scaling, grayscale conversion, and adding noise to the image. These changed data are equivalent to new data for the deep learning segmentation model, so data diversity can be improved, so that the trained model has better generalization performance.

[0157] Fig.15The 3D blood vessel image obtained after normalization and data enhancement processing of the original 3D coronary CTA section image is shown. In this embodiment, the data enhancement specifically adds random noise and random contrast transformation to the normalized original 3D coronary CTA section image.

[0158] The deep learning segmentation model also includes a feature extractor, and the loss function is also used to train the feature extractor. Figure 4 Combined with Fig.12 , the aforementioned step S2 further includes the following steps:

[0159] S21: Using an attention mechanism to extract vascular features from a three-dimensional vascular image containing a vascular segment of interest through a feature extractor to obtain three-dimensional vascular image feature data of the vascular segment of interest.

[0160] Among them, the attention mechanism includes a spatial attention mechanism and / or a channel attention mechanism.

[0161] The spatial attention mechanism is performed on the three-dimensional spatial dimension of the image, which enables the deep learning segmentation model to focus on specific key areas in the image and ignore unimportant parts. The spatial attention mechanism enables the deep learning segmentation model to automatically learn the importance of different spatial positions in the image, and improves the deep learning segmentation model's ability to represent spatial information. In the present invention, the spatial attention mechanism is used to extract the spatial feature data of the vessel of interest in the three-dimensional vascular image containing the vessel of interest, which is conducive to improving the subsequent segmentation accuracy of the tunica media and tunica intima.

[0162] The channel attention mechanism can assign different weights to different feature channels of the image, so that the deep learning segmentation model can pay more attention to the feature channels with more important information content, while suppressing less important feature channels. In the present invention, the channel attention mechanism is used to extract the channel feature data (i.e., feature channels) of the vascular segment of interest in the three-dimensional vascular image containing the vascular segment of interest, and can adaptively adjust the weights of different feature channels to enhance the deep learning segmentation model's ability to characterize the key features of the target blood vessel, thereby improving the performance of the deep learning segmentation model, which is beneficial to improving the subsequent segmentation accuracy of the middle membrane and the inner membrane.

[0163] Specifically, continue to refer to Fig.12The feature extractor is a shared encoder, which is used to encode the features of the three-dimensional vascular image to obtain a feature vector F1. The feature vector F1 includes an intima feature vector for representing the intima feature and a media feature vector for representing the media feature. Then, the channel attention mechanism and the spatial attention mechanism are calculated for the feature vector F1, so that the feature vector F1 is further encoded into a feature vector F2. The feature vector F2 is the feature data of the three-dimensional vascular image. The feature vector F2 is further decoded by the intima segmenter and the media segmenter to obtain the media feature and the intima feature respectively, and the cross-attention mechanism is calculated for the media feature and the intima feature. The intima segmenter and the media segmenter promote each other to learn the intima feature and the media feature respectively, and finally the intima segmentation result image and the media segmentation result image with high segmentation accuracy can be obtained respectively.

[0164] Furthermore, in the above embodiments, the feature extractor, the inner membrane segmenter and the middle membrane segmenter can all be any backbone. The backbone refers to the main feature extraction part in the neural network, which can be a pre-trained model based on a large image dataset (such as ImageNet), which can extract specific features of the image from the input data. The backbone can be composed of a series of convolutional layers, pooling layers, normalization layers and activation function layers.

[0165] In the above embodiments, the present invention is guided by the topological structure of the tunica media and can achieve higher accuracy in endothelial segmentation in three-dimensional coronary CTA images, especially three-dimensional coronary CTA images with high plaque load. The present invention uses channel attention, spatial attention, and cross-attention mechanisms to achieve efficient feature interaction between the endothelial membrane and the tunica media, thereby improving the segmentation accuracy of the endothelial membrane and the tunica media. The present invention can also optimize the continuity and accuracy of endothelial segmentation through the above-mentioned deep learning segmentation model. Since the extraction of the vascular centerline is obtained by skeletonizing the segmentation results of the vascular endothelial membrane, the present invention can also improve the accuracy of subsequent vascular centerline extraction and optimize the extraction quality of the vascular centerline.

[0166] In addition, through the optimized intima segmentation and centerline extraction technology, the risk assessment and pathological analysis of heart disease can be more accurate, the misdiagnosis rate and missed diagnosis rate can be reduced, and the clinical application value of CCTA (coronary computed tomography angiography) analysis can be optimized, which is conducive to the accurate assessment of heart lesions and subsequent treatment planning, and has significant social and economic benefits.

[0167] Second, refer to Fig. 9The embodiment of the present invention discloses a training device 1 for a deep learning segmentation model, and the deep learning segmentation model is trained by the training method of the deep learning segmentation model in any of the aforementioned embodiments. The deep learning segmentation model includes a media segmenter and an endothelium segmenter, the media segmenter is used to segment the media segmentation result image, and the endothelium segmenter is used to segment the endothelium segmentation result image. The training device 1 for the deep learning segmentation model includes an image acquisition module 11, a pre-segmentation module 12, a loss function determination module 13 and a training module 14.

[0168] The image acquisition module 11 is connected to the pre-segmentation module 12 , the pre-segmentation module 12 is connected to the loss function determination module 13 , and the loss function determination module 13 is connected to the training module 14 .

[0169] Continue to refer Fig. 9 The image acquisition module 11 is used to acquire a three-dimensional blood vessel image including a blood vessel segment of interest, and to acquire a media segmentation result label and an intima segmentation result label of the annotated blood vessel segment of interest.

[0170] The pre-segmentation module 12 is used to process the three-dimensional vascular image containing the vascular segment of interest through the deep learning segmentation model to be trained to obtain the media segmentation result image and the intima segmentation result image.

[0171] The loss function determination module 13 is used to determine the loss function according to the three-dimensional blood vessel image of the blood vessel segment of interest, the media segmentation result image and the intima segmentation result image.

[0172] The training module 14 is used to update the parameters of the deep learning segmentation model to be trained according to the three-dimensional vascular image containing the vascular segment of interest, the media segmentation result image, the endothelium segmentation result image, the media segmentation result label, the endothelium segmentation result label and the loss function, until the deep learning segmentation model to be trained meets the preset training conditions, thereby obtaining a trained deep learning segmentation model.

[0173] By adopting the above-mentioned technical scheme, the training device 1 of the deep learning segmentation model of this embodiment is combined with the cross-attention mechanism, through which the middle membrane features and the inner membrane features interact, so that the trained deep learning segmentation model can more accurately and simultaneously segment the middle membrane segmentation result image and the inner membrane segmentation result image of the blood vessel segment of interest, thereby improving the segmentation accuracy of the middle membrane and the inner membrane.

[0174] Among them, the pre-segmentation module 12 processes the three-dimensional vascular image containing the vascular segment of interest through the deep learning segmentation model to be trained, specifically by using the middle membrane segmenter and the endothelium segmenter to segment the three-dimensional vascular image containing the vascular segment of interest using the cross-attention mechanism to obtain the middle membrane segmentation result image and the endothelium segmentation result image; the process of segmenting the three-dimensional vascular image containing the vascular segment of interest using the cross-attention mechanism specifically includes: extracting the middle membrane features of the three-dimensional vascular image containing the vascular segment of interest through the middle membrane segmenter to obtain the middle membrane feature data; extracting the endothelium features of the three-dimensional vascular image containing the vascular segment of interest through the endothelium segmentor to obtain the endothelium feature data; the middle membrane segmentor performs feature fusion on the middle membrane feature data and the endothelium feature data through the cross-attention mechanism to obtain the middle membrane segmentation result image; the endothelium segmentor performs feature fusion on the middle membrane feature data and the endothelium feature data through the cross-attention mechanism to obtain the endothelium segmentation result image.

[0175] By adopting the above-mentioned technical scheme, the training device 1 of the deep learning segmentation model of this embodiment is combined with the cross-attention mechanism. Through the interaction of the middle membrane feature data and the inner membrane feature data, the trained deep learning segmentation model can more accurately and simultaneously segment the middle membrane segmentation result image and the inner membrane segmentation result image of the blood vessel segment of interest, thereby improving the segmentation accuracy of the middle membrane and the inner membrane.

[0176] Furthermore, the image acquisition module 11 can also be used to perform the aforementioned steps S11 and S12. The pre-segmentation module 12 can also be used to perform the aforementioned step S21. The loss function determination module 13 can also be used to perform the aforementioned steps S310 and S320.

[0177] In a third aspect, an embodiment of the present invention discloses a blood vessel image segmentation method, wherein a blood vessel image is segmented by a deep learning segmentation model, wherein the deep learning segmentation model includes a media segmentor and an intima segmentor, wherein the media segmentor is used to segment a media segmentation result image, and the intima segmentor is used to segment an intima segmentation result image. The deep learning segmentation model is obtained by training the deep learning segmentation model training method in any of the aforementioned embodiments.

[0178] refer to Figure 5 , the blood vessel image segmentation method specifically includes the following steps:

[0179] S01: Acquire three-dimensional blood vessel image feature data of a blood vessel segment of interest.

[0180] Among them, the three-dimensional blood vessel image feature data can be Fig.12 The eigenvector F2 in .

[0181] S02: Using a deep learning segmentation model and a cross-attention mechanism, the three-dimensional vascular image feature data of the vascular segment of interest is segmented to obtain a media segmentation result image and an intima segmentation result image of the vascular segment of interest.

[0182] By adopting the above technical solution, the blood vessel image segmentation method of the present invention can improve the segmentation accuracy of the middle membrane and the inner membrane by adopting a cross-attention mechanism and making full use of the interactive relationship between the middle membrane and the inner membrane.

[0183] Among them, the process of segmenting the three-dimensional vascular image feature data by using the cross-attention mechanism specifically includes: extracting the media features of the three-dimensional vascular image feature data by the media segmenter to obtain the media feature data; extracting the intima features of the three-dimensional vascular image feature data by the intima segmenter to obtain the intima feature data. The media segmenter performs feature fusion on the media feature data and the intima feature data by the cross-attention mechanism to obtain the media segmentation result image. The intima segmenter performs feature fusion on the media feature data and the intima feature data by the cross-attention mechanism to obtain the intima segmentation result image.

[0184] Fig.16 A schematic diagram showing the superposition of the media segmentation result image and the intima segmentation result image on the original three-dimensional blood vessel image, Fig.17 The corresponding 3D visualization results of the middle membrane segmentation results and the inner membrane segmentation results are shown. Fig.16 and Fig.17 In the figure, the green part represents the segmentation result of the middle membrane, and the blue part represents the segmentation result of the inner membrane. It should be noted that the blue part is also part of the middle membrane. Since it is a superimposed image, part of the green part has been covered by the blue part.

[0185] By adopting the above technical scheme, the vascular image segmentation method of the present invention adopts a cross-attention mechanism to enable the middle membrane features and the inner membrane features to interact, so that the deep learning segmentation model can more accurately and simultaneously segment the middle membrane segmentation result image and the inner membrane segmentation result image of the vascular segment of interest, thereby improving the segmentation accuracy of the middle membrane and the inner membrane.

[0186] Further, refer to Figure 6 The aforementioned step S01 specifically includes the following steps:

[0187] S011: Acquire a three-dimensional blood vessel image including a blood vessel segment of interest.

[0188] Specifically, the three-dimensional blood vessel image is a three-dimensional coronary CTA image.

[0189] S012: Acquire three-dimensional vascular image feature data of the vascular segment of interest based on the three-dimensional vascular image containing the vascular segment of interest, including: extracting vascular features from the three-dimensional vascular image of the vascular segment of interest through a feature extractor using an attention mechanism to obtain the three-dimensional vascular image feature data of the vascular segment of interest.

[0190] The attention mechanism includes a spatial attention mechanism and / or a channel attention mechanism. The spatial attention mechanism is used to extract the spatial feature data of the vessel of interest of the vessel segment of interest in the three-dimensional vessel image. The channel attention mechanism is used to extract the channel feature data of the vessel of interest of the vessel segment of interest in the three-dimensional vessel image. For more details, please refer to the description of the spatial attention mechanism and the channel attention mechanism above, which will not be repeated here.

[0191] Further, refer to Figure 7 The aforementioned step S011 specifically includes the following steps:

[0192] S0111: Acquire an unprocessed three-dimensional vascular image of a vascular segment of interest.

[0193] Unprocessed 3D blood vessel images are, for example, Fig.13 The original three-dimensional coronary CTA section image is shown.

[0194] S0112: Perform normalization processing on the unprocessed three-dimensional blood vessel image to obtain a three-dimensional blood vessel image of the blood vessel segment of interest.

[0195] The normalized three-dimensional vascular image is, for example, Fig.14 Shown is a three-dimensional coronary CTA section image.

[0196] It should be noted that when actually using the trained deep learning segmentation model, there is no need to perform data augmentation on the unprocessed 3D vascular images during preprocessing, only normalization is required.

[0197] refer to Figure 8 After executing the aforementioned steps S01 and S02, step S03 may be further executed: performing image enhancement optimization operations on the media segmentation result image and the endothelium segmentation result image to obtain enhanced final media segmentation result image and final endothelium segmentation result image.

[0198] The image enhancement optimization operation includes morphological operation and / or connectivity analysis, the morphological operation is selected from one or more of the following: corrosion operation, expansion operation, opening operation, closing operation, morphological gradient, top hat and black hat, and the connectivity analysis is selected from one or more of the following: connected component labeling, region growing, boundary tracing, hole filling, skeleton extraction and region segmentation. The above operations can fill the holes inside the blood vessels in the segmented image, or remove the noise generated during segmentation, etc., which can achieve the purpose of optimizing the image. The appropriate operation can be selected according to the actual segmentation situation to optimize the segmented image.

[0199] By further performing image enhancement and optimization operations on the media segmentation result images and the intima segmentation result images, the accuracy and robustness of the media and intima segmentation can be further improved.

[0200] Fourthly, reference Fig.10 The embodiment of the present invention discloses a blood vessel image segmentation device 2, which segments the blood vessel image through a deep learning segmentation model. The deep learning segmentation model includes a media segmentor and an intima segmentor. The media segmentor is used to segment the media segmentation result image, and the intima segmentor is used to segment the intima segmentation result image. The blood vessel image segmentation device 2 includes a feature acquisition module 21 and an image segmentation module 22. The feature acquisition module 21 is connected to the image segmentation module 22.

[0201] The feature acquisition module 21 is used to acquire the three-dimensional blood vessel image feature data of the blood vessel segment of interest.

[0202] The image segmentation module 22 is used to segment the three-dimensional vascular image feature data of the vascular segment of interest using a cross-attention mechanism through a deep learning segmentation model to obtain a media segmentation result image and an intima segmentation result image of the vascular segment of interest.

[0203] By adopting the above technical solution, the blood vessel image segmentation device 2 of the present invention can improve the segmentation accuracy of the middle membrane and the inner membrane by adopting a cross-attention mechanism and making full use of the interactive relationship between the middle membrane and the inner membrane.

[0204] Among them, the image segmentation module adopts the cross-attention mechanism to segment the three-dimensional vascular image feature data, which specifically includes: extracting the middle membrane features of the three-dimensional vascular image feature data through the middle membrane segmentor to obtain the middle membrane feature data; extracting the inner membrane features of the three-dimensional vascular image feature data through the inner membrane segmentor to obtain the inner membrane feature data; the middle membrane segmentor performs feature fusion on the middle membrane feature data and the inner membrane feature data through the cross-attention mechanism to obtain the middle membrane segmentation result image; the inner membrane segmentor performs feature fusion on the middle membrane feature data and the inner membrane feature data through the cross-attention mechanism to obtain the inner membrane segmentation result image.

[0205] By adopting the above technical scheme, the vascular image segmentation device 2 of the present invention adopts a cross-attention mechanism to enable the middle membrane features and the inner membrane features to interact, so that the deep learning segmentation model can more accurately and simultaneously segment the middle membrane segmentation result image and the inner membrane segmentation result image of the vascular segment of interest, thereby improving the segmentation accuracy of the middle membrane and the inner membrane.

[0206] Furthermore, the feature acquisition module 21 can also be used to execute the aforementioned steps S011, S012, S0111, and S0112. The image segmentation module 22 can also be used to execute the aforementioned step S03.

[0207] Fifth, reference Fig.11 The embodiment of the present invention further discloses an electronic device 3, including a memory 31 and a processor 32, wherein the memory 31 stores at least one instruction, and when the at least one instruction is executed by the processor 32, the training method of the deep learning segmentation model in any of the aforementioned embodiments and / or the vascular image segmentation method in any of the aforementioned embodiments are implemented. The memory 31 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application, a boot loader, and other programs.

[0208] In this embodiment, the electronic device 3 can combine the cross-attention mechanism, through the interaction of the middle membrane feature data and the inner membrane feature data, so that the trained deep learning segmentation model can more accurately and simultaneously segment the middle membrane segmentation result image and the inner membrane segmentation result image of the blood vessel segment of interest, thereby improving the segmentation accuracy of the middle membrane and the inner membrane.

[0209] In a sixth aspect, an embodiment of the present invention further discloses a computer-readable storage medium, in which at least one instruction is stored, and when the at least one instruction is executed, the training method of the deep learning segmentation model in any of the aforementioned embodiments and / or the vascular image segmentation method in any of the aforementioned embodiments is implemented.

[0210] By adopting the above technical solution, the computer-readable storage medium can be combined with the cross-attention mechanism, and through the interaction of the middle membrane feature data and the inner membrane feature data, the trained deep learning segmentation model can more accurately segment the middle membrane segmentation result image and the inner membrane segmentation result image of the blood vessel segment of interest, thereby improving the segmentation accuracy of the middle membrane and the inner membrane.

[0211] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0212] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0213] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0214] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0215] Although the present invention has been illustrated and described with reference to certain preferred embodiments of the present invention, it should be understood by those skilled in the art that the above is a further detailed description of the present invention in conjunction with specific embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. Those skilled in the art may make various changes in form and details, including making several simple deductions or substitutions, without departing from the spirit and scope of the present invention.

Claims

1. A training method for a deep learning segmentation model, characterized in that: The deep learning segmentation model includes a media segmenter and an intima segmenter, wherein the media segmenter is used to segment a media segmentation result image, and the intima segmenter is used to segment an intima segmentation result image. The training method includes: Acquire a three-dimensional blood vessel image including a blood vessel segment of interest; acquire a media segmentation result label and an intima segmentation result label of the annotated blood vessel segment of interest; Processing the three-dimensional blood vessel image including the blood vessel segment of interest by using the deep learning segmentation model to be trained to obtain the media segmentation result image and the intima segmentation result image; Determine a loss function according to the media segmentation result label, the intima segmentation result label, the media segmentation result image, and the intima segmentation result image of the blood vessel segment of interest; According to the three-dimensional vascular image including the vascular segment of interest, the media segmentation result image, the intima segmentation result image, the media segmentation result label, the intima segmentation result label and the loss function, parameters of the deep learning segmentation model to be trained are updated until the deep learning segmentation model to be trained meets the preset training conditions, thereby obtaining the trained deep learning segmentation model; The media segmenter and the endothelium segmenter use a cross-attention mechanism to segment the three-dimensional blood vessel image including the blood vessel segment of interest to obtain the media segmentation result image and the endothelium segmentation result image.

2. The deep learning segmentation model training method according to claim 1, characterized in that: The process of segmenting the three-dimensional blood vessel image including the blood vessel segment of interest by using the cross attention mechanism specifically includes: Extracting the media features of the three-dimensional vascular image including the vascular segment of interest by the media segmenter to obtain media feature data; Extracting the intimal features of the three-dimensional blood vessel image including the blood vessel segment of interest by the intimal segmentation device to obtain intimal feature data; The middle membrane segmenter performs feature fusion on the middle membrane feature data and the inner membrane feature data through the cross attention mechanism to obtain the middle membrane segmentation result image; The endothelium segmentor performs feature fusion on the middle membrane feature data and the endothelium feature data through the cross-attention mechanism to obtain the endothelium segmentation result image.

3. The deep learning segmentation model training method according to claim 1, characterized in that: The loss functions include a cross entropy loss function, a Dice loss function and a consistency loss function.

4. The deep learning segmentation model training method according to claim 3, characterized in that: The consistency loss function is obtained in the following way: According to the middle membrane segmentation result image, a soft skeleton extraction method is used to obtain the middle membrane skeletonization result data of the middle membrane segmentation result image; according to the inner membrane segmentation result image, an inner membrane skeletonization result data of the inner membrane segmentation result image is obtained by the soft skeleton extraction method; The consistency loss function is determined according to the middle membrane skeletonization result data and the inner membrane skeletonization result data.

5. The deep learning segmentation model training method according to claim 3, characterized in that: The loss function is specifically: Total Loss=Dice Lumen +Dice Wall +CE Lumen +CE Wall +Consistency lumen-wall Among them, Total Loss represents the loss function, Dice Lumen +Dice Wall represents the Dice loss function, CE Lumen +CE Wall Represents the cross entropy loss function, Consistency lumen-wall represents the consistency loss function, G L represents the inner membrane segmentation result label, P L represents the endometrial segmentation result image, S L represents the inner membrane skeletonization result data, G W Indicates the result label of the middle membrane segmentation, P W represents the middle membrane segmentation result image, S W Represents the media skeletonization result data.

6. The deep learning segmentation model training method according to claim 4, characterized in that: The consistency loss function is specifically: S l =SoftSkeleton(P l ) S w =SoftSkeleton(P w ) Among them, P I represents the endometrium segmentation result image, S I represents the inner membrane skeletonization result data, P W represents the middle membrane segmentation result image, S W represents the skeletonization result data of the middle membrane, SoftSkeleton represents the soft skeleton extraction method, Consistency lumen-wall Denotes the consistency loss function.

7. The deep learning segmentation model training method according to claim 1, characterized in that: The acquiring of a three-dimensional blood vessel image including a blood vessel segment of interest specifically includes: Acquiring an unprocessed three-dimensional blood vessel image of the blood vessel segment of interest; The unprocessed three-dimensional blood vessel image is subjected to normalization processing and data enhancement processing to obtain the three-dimensional blood vessel image.

8. The deep learning segmentation model training method according to claim 1, characterized in that: The deep learning segmentation model further includes a feature extractor, and the loss function is further used to train the feature extractor. The three-dimensional blood vessel image including the blood vessel segment of interest is processed by the deep learning segmentation model to be trained, specifically including: Using the attention mechanism to extract blood vessel features from the three-dimensional blood vessel image containing the blood vessel segment of interest through the feature extractor to obtain three-dimensional blood vessel image feature data of the blood vessel segment of interest; Wherein, the attention mechanism includes a spatial attention mechanism and / or a channel attention mechanism; The spatial attention mechanism is used to extract the vessel of interest spatial feature data of the vessel of interest segment in the three-dimensional vessel image containing the vessel of interest segment; The channel attention mechanism is used to extract the vessel of interest channel feature data of the vessel of interest segment in the three-dimensional vessel image containing the vessel of interest segment.

9. A training device for a deep learning segmentation model, characterized in that: The deep learning segmentation model includes a media segmenter and an intima segmenter, wherein the media segmenter is used to segment a media segmentation result image, and the intima segmenter is used to segment an intima segmentation result image. The training device includes: An image acquisition module, used to acquire a three-dimensional blood vessel image including a blood vessel segment of interest, and to acquire a media segmentation result label and an intima segmentation result label of the blood vessel segment of interest that have been annotated; A pre-segmentation module, used for processing the three-dimensional blood vessel image including the blood vessel segment of interest by using the deep learning segmentation model to be trained to obtain the media segmentation result image and the intima segmentation result image; A loss function determination module, used to determine a loss function according to the media segmentation result label, the intima segmentation result label, the media segmentation result image and the intima segmentation result image of the blood vessel segment of interest; A training module, used for updating parameters of the deep learning segmentation model to be trained according to the three-dimensional vascular image including the vascular segment of interest, the media segmentation result image, the intima segmentation result image, the media segmentation result label, the intima segmentation result label and the loss function, until the deep learning segmentation model to be trained meets the preset training conditions, thereby obtaining the trained deep learning segmentation model; The media segmenter and the endothelium segmenter use a cross-attention mechanism to segment the three-dimensional blood vessel image including the blood vessel segment of interest to obtain the media segmentation result image and the endothelium segmentation result image.

10. The deep learning segmentation model training device according to claim 9, characterized in that: In the process of the pre-segmentation module processing the three-dimensional blood vessel image including the blood vessel segment of interest by the deep learning segmentation model to be trained, the three-dimensional blood vessel image including the blood vessel segment of interest is segmented by the media segmenter and the intima segmenter using a cross attention mechanism to obtain the media segmentation result image and the intima segmentation result image; The process of segmenting the three-dimensional blood vessel image including the blood vessel segment of interest by using the cross attention mechanism specifically includes: Extracting the media features of the three-dimensional vascular image including the vascular segment of interest by the media segmenter to obtain media feature data; Extracting the intimal features of the three-dimensional blood vessel image including the blood vessel segment of interest by the intimal segmentation device to obtain intimal feature data; The middle membrane segmenter performs feature fusion on the middle membrane feature data and the inner membrane feature data through the cross attention mechanism to obtain the middle membrane segmentation result image; The endothelium segmentor performs feature fusion on the middle membrane feature data and the endothelium feature data through the cross-attention mechanism to obtain the endothelium segmentation result image.

11. A blood vessel image segmentation method, characterized in that: The blood vessel image is segmented by a deep learning segmentation model, wherein the deep learning segmentation model includes a media segmentor and an intima segmentor, wherein the media segmentor is used to segment a media segmentation result image, and the intima segmentor is used to segment an intima segmentation result image. The blood vessel image segmentation method includes: Acquire three-dimensional vascular image feature data of a vascular segment of interest; The deep learning segmentation model adopts a cross-attention mechanism to segment the three-dimensional vascular image feature data of the vascular segment of interest, thereby obtaining the media segmentation result image and the intima segmentation result image of the vascular segment of interest.

12. The blood vessel image segmentation method according to claim 11, characterized in that: The process of segmenting the three-dimensional vascular image feature data using the cross attention mechanism specifically includes: Extracting the middle membrane features of the three-dimensional blood vessel image feature data by the middle membrane segmentor to obtain middle membrane feature data; Extracting the intima features of the three-dimensional blood vessel image feature data by the intima segmenter to obtain intima feature data; The middle membrane segmenter performs feature fusion on the middle membrane feature data and the inner membrane feature data through the cross attention mechanism to obtain the middle membrane segmentation result image; The endothelium segmentor performs feature fusion on the middle membrane feature data and the endothelium feature data through the cross-attention mechanism to obtain the endothelium segmentation result image.

13. The blood vessel image segmentation method according to claim 11, characterized in that: The deep learning segmentation model further includes a feature extractor, and the acquisition of three-dimensional vascular image feature data of the vascular segment of interest specifically includes: Acquiring a three-dimensional blood vessel image including the blood vessel segment of interest; Acquiring the three-dimensional blood vessel image feature data of the blood vessel segment of interest according to the three-dimensional blood vessel image including the blood vessel segment of interest, including: extracting blood vessel features from the three-dimensional blood vessel image of the blood vessel segment of interest by the feature extractor using an attention mechanism to obtain the three-dimensional blood vessel image feature data of the blood vessel segment of interest; Wherein, the attention mechanism includes a spatial attention mechanism and / or a channel attention mechanism; The spatial attention mechanism is used to extract the spatial feature data of the vessel of interest of the vessel segment of interest in the three-dimensional vessel image; The channel attention mechanism is used to extract the vessel channel feature data of the vessel segment of interest in the three-dimensional vessel image.

14. The blood vessel image segmentation method according to claim 11, characterized in that: The acquiring of the three-dimensional blood vessel image including the blood vessel segment of interest specifically includes: Acquiring an unprocessed three-dimensional blood vessel image of the blood vessel segment of interest; The unprocessed three-dimensional blood vessel image is normalized to obtain the three-dimensional blood vessel image of the blood vessel segment of interest.

15. The blood vessel image segmentation method according to claim 11, characterized in that: Also includes: Performing image enhancement and optimization operations on the media segmentation result image and the intima segmentation result image to obtain enhanced final media segmentation result image and final intima segmentation result image; Wherein, the image enhancement optimization operation includes morphological operation and / or connectivity analysis, and the morphological operation is selected from one or more of the following: corrosion operation, expansion operation, opening operation, closing operation, morphological gradient, top hat and black hat; the connectivity analysis is selected from one or more of the following: connected component labeling, region growing, boundary tracing, hole filling, skeleton extraction and region segmentation.

16. A blood vessel image segmentation device, characterized in that: The blood vessel image is segmented by a deep learning segmentation model, wherein the deep learning segmentation model includes a media segmentor and an intima segmentor, wherein the media segmentor is used to segment a media segmentation result image, and the intima segmentor is used to segment an intima segmentation result image, and the blood vessel image segmentation device includes: A feature acquisition module, used to acquire three-dimensional vascular image feature data of a vascular segment of interest; An image segmentation module is used to segment the three-dimensional vascular image feature data of the vascular segment of interest using a cross-attention mechanism through the deep learning segmentation model to obtain the media segmentation result image and the intima segmentation result image of the vascular segment of interest.

17. The blood vessel image segmentation device according to claim 16, characterized in that: The process in which the image segmentation module segments the three-dimensional blood vessel image feature data using the cross attention mechanism specifically includes: Extracting the middle membrane features of the three-dimensional blood vessel image feature data by the middle membrane segmentor to obtain middle membrane feature data; Extracting the intima features of the three-dimensional blood vessel image feature data by the intima segmenter to obtain intima feature data; The middle membrane segmenter performs feature fusion on the middle membrane feature data and the inner membrane feature data through the cross attention mechanism to obtain the middle membrane segmentation result image; The endothelium segmentor performs feature fusion on the middle membrane feature data and the endothelium feature data through the cross-attention mechanism to obtain the endothelium segmentation result image.

18. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, and when the at least one instruction is executed by the processor, implements the training method of the deep learning segmentation model as described in any one of claims 1-8, and / or the vascular image segmentation method as described in any one of claims 11-15.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which, when executed, implements the training method of the deep learning segmentation model as described in any one of claims 1-8, and / or the vascular image segmentation method as described in any one of claims 11-15.

Images