IVOCT vascular branch opening point detection method based on deep learning

Abstract

The present invention relates to an IVOCT vascular branch opening point detection method based on deep learning, comprising the following steps: (1) IVOCT image preprocessing; (2) vascular branch recognition, wherein the resolution of the preprocessed IVOCT image in the training set is adjusted and the image is input into a vascular branch recognition model to train the vascular branch recognition model; the vascular branch recognition model is a Vision Transformer model based on a self-attention mechanism; (3) vascular branch opening point detection: the first step is to construct a vascular branch opening point detection dataset; the second step is to train the branch opening point detection model, wherein the resolution of the image in the training set obtained in the first step is adjusted and the image is input into the branch opening point detection model; the branch opening point detection model is a key point detection model based on a Transformer, comprising three modules: an encoder, a decoder, and a post-processing module.

Description

Technical Field

[0001] The present invention relates to intravascular optical coherence tomography (IVOCT), which covers the field of endoscopic OCT imaging of cardiovascular and cerebral blood vessels, and in particular to a deep learning-based IVOCT vascular branch recognition and branch opening point detection method. Background Art

[0002] Intravascular optical coherence tomography (IVOCT) is a high-resolution endovascular imaging technique that measures backscattered light from blood vessels, enabling high-resolution imaging of the microstructure of in vivo vascular walls with a resolution one to two orders of magnitude higher than that of intravascular ultrasound (IVUS). Vascular branching information is of great value in fluid simulation and numerical analysis of vascular models, registration of different vascular images, and stent implantation. Therefore, accurate identification of vascular branches is crucial in the treatment of coronary artery disease.

[0003] At present, the branch vessel recognition methods proposed by domestic and foreign scholars can be mainly divided into algorithms based on machine learning and deep learning and traditional computer vision algorithms. Among them, Macedo et al. [1] Using the method based on Ughi et al. [2] A set of operators that extract geometric features from the lumen contour to identify branches. Algorithms based on deep learning of neural networks have achieved superior performance than machine learning algorithms. Macedo et al. and Porto et al. [3,4] The performance of neural network and SVM classifiers in branch detection tasks was evaluated using the same dataset and it was found that the results obtained using neural network were better than those obtained using SVM. Miyagawa et al. [5] Convolutional neural networks are applied to branch recognition, and transfer learning methods are used to transfer the knowledge of lumen segmentation to branch recognition to overcome the problem of small training sets.

[0004] Traditional computer vision algorithms: Wang et al. [6] A fully automated method for extracting the center of the main vessel based on distance transformation and then identifying vessel branches in IVOCT was proposed. They first used the Dijkstra algorithm to detect the various parts of the catheter and the lumen contour based on the gradient of the IVOCT image. Then, based on the lumen contour and the extracted main vessel center, they set a distance threshold and established a distance map to detect vessel branches. Cao et al. [7]Based on the work of Wang et al., the accuracy of detecting the boundary between the main vessel and the branch was further improved. They used the algorithm of Wang et al. to screen images that may be branches, and then calculated the angle between the normal vector of the image contour point and the vector pointing to the center of the main vessel, and used the change of this angle to identify the opening point. [8] The characteristics of the IVOCT pullback process are utilized to extract adjacent longitudinal sections, and the branch image is judged based on the outward discontinuity of the branch area.

[0005] All of the above methods perform branch identification based on the results of lumen contour segmentation. However, correctly extracting the lumen contour is a challenging problem, especially in the presence of complex blood artifacts and low brightness of the blood vessel wall, which can cause serious errors in the extracted blood vessel contour, resulting in the algorithm being unable to correctly identify branches and reducing the accuracy of branch identification.

[0006] References:

[0007] [1]CDNPorto,CFFCFilho,MMGMacedo,MAGutierrez andM.GFCosta.Classification of bifurcations regions in IVOCT images using support vector machine and artificial neural network models[J].Int.Soc.Opt.Photon.vol.3,Mar.2017.

[0008] [2]EWDijkstra.A note on two problems in connexion with graphs[J].Numer.Math,Dec.1959vol.1,no.1:269-271.

[0009] [3]MMGMacedo,WVN MZGalon,CKTakimura,PALemos andM.A.Gutierrez.A bifurcation identifier for IV-OCT using orthogonal leastsquares and supervised machine learning[J].Comput.Med.Imag.Graph,2015.vol.46:237-248.

[0010] [4]G.J.Ughi,T.Adriaenssens,W.Desmet and J.D’hooge.Fully automaticthree-dimensional visualization of intravascular optical coherence tomographyimages:Methods and feasibility in vivo[J].Biomed.Opt.Express,Dec.2012vol.3,no.12:3291-3303.

[0011] [5]M.Miyagawa,M.G.F.Costa,M.A.Gutierrez,J.P.G.F.Costa and C.F.F.CostaFilho.Detecting Vascular Bifurcation in IVOCT Images Using ConvolutionalNeural Networks With Transfer Learning[J].IEEE Access,2019.vol.7,pp.66167-66175.

[0012] [6]A.Wang,J.Eggermont,J.H.C.Reiber and J.Dijkstra.Fully automatedside branch detection in intravascular optical coherence tomography pullbackruns[J].Biomed.Opt.Express,2014vol.5,no.9:3160-3173.

[0013] [7]Y.Cao et al.Automatic Side Branch Ostium Detection and MainVascular Segmentation in Intravascular Optical Coherence Tomography Images[C].In IEEE Journal of Biomedical and Health Informatics,Sept.2018vol.22,no.5:1531-1539.

[0014] [8] F. Zhu, Y. Yu, Z. Ding, Q. Li, S. Zhou, K. Tao, H. Kuang, and T. Liu. Automatic bifurcation detection utilizing pullback characteristics of bifurcation inintravascular optical coherence tomography [J]. Opt. Express, 2022, vol. 30, 31381-31395. Summary of the Invention

[0015] The purpose of this invention is to provide a deep learning-based IVOCT vascular branch recognition and branch opening point detection method to overcome the difficulty of existing automatic IVOCT vascular branch recognition and branch opening point detection methods in recognizing images containing severe blood artifacts, unclear vessel walls, and when the probe is retracted into a plastic catheter. The present invention first performs a logarithmic transformation on the original IVOCT image and converts it to rectangular coordinates to obtain the network input image. The vascular branch recognition model is used to identify whether the image contains branches. The image containing branches is then input into the branch opening point detection model to detect the opening points of vascular branches. The technical solution is as follows:

[0016] A deep learning-based IVOCT blood vessel branch opening point detection method includes the following steps:

[0017] (1) IVOCT image preprocessing:

[0018] Performing logarithmic transformation on the collected IVOCT images, and transforming the logarithmic transformed IVOCT images into a rectangular coordinate system to obtain a preprocessed data set;

[0019] Divide the preprocessed IVOCT image sequence into training set and test set as needed;

[0020] (2) Identification of vascular branches:

[0021] Adjust the resolution of the pre-processed IVOCT images in the training set and input them into the vascular branch recognition model to train the vascular branch recognition model;

[0022] The vascular branch recognition model is a Vision Transformer model based on the self-attention mechanism. The model includes three modules: the Embedding layer, the Transformer encoder, and the classification layer. The Embedding layer is used to divide an image into several blocks and map and tile the image blocks into a sequence through the convolution layer. The mapped and tiled sequence is input into the Transformer encoder after position encoding. The Transformer encoder is composed of several stacked Transformer modules, each of which includes a layer normalization module, a multi-head attention module, and a multi-layer perceptron module. The Transformer encoder is used to learn the long-range dependencies of the image and output image features. The learned image features are input into the fully connected layer for weighted summation to obtain the final classification result.

[0023] After the vascular branch recognition model training is completed, the test set is input into the vascular branch recognition model to obtain the classification results of the test set;

[0024] (3) Detection of the opening point of blood vessel branches:

[0025] Step 1: Construct a dataset for detecting blood vessel branch opening points. The method is as follows:

[0026] Based on the vascular branch recognition model obtained in step (2), images with branches are extracted as training sets and test sets for vascular branch opening point detection. The branch opening points are the two dividing points where the branches and the main blood vessels intersect. The branch opening point coordinates are used as key point coordinates to construct the annotation data.

[0027] Step 2: Train the branch opening point detection model as follows:

[0028] Adjust the resolution of the training set images obtained in the first step and input them into the branch opening point detection model;

[0029] The branch opening point detection model is a key point detection model based on Transformer, including three modules: encoder, decoder and post-processing; the encoder includes an embedding layer and several Transformer modules stacked together, the embedding layer divides an image into several blocks of images, and maps and flattens these image blocks into a sequence through a convolution layer; each Transformer module includes a layer normalization module, a multi-head attention module and a multi-layer perceptron module; the encoder is used to learn the long-distance dependency of the image and output image features; the learned image features are dimensionally adjusted after layer normalization to obtain a feature map; the decoder includes two deconvolution modules and a predictor, the deconvolution module includes a deconvolution layer, a batch normalization layer and a ReLU activation layer, the feature map output by the encoder is sequentially input into the two deconvolution modules to enlarge the size of the feature map, and the enlarged feature map is input into a convolution layer with a kernel size of 1×1 to obtain a heat map predicted by the model;

[0030] Generate a two-dimensional Gaussian distribution heat map centered on the key point coordinates as a supervisory signal, calculate the MSE loss with the heat map predicted by the model, use the AdamW algorithm to optimize the model parameters to reduce the loss, and stop training when the model reaches the predetermined number of iterations;

[0031] The post-processing module uses an unbiased data processing method to extract the key point coordinates from the model-predicted heat map and make corrections; after the model training is completed, the test set is input into the branch opening point detection model, and the model outputs the coordinates corresponding to the two opening points.

[0032] Furthermore, after step (1) and before step (2), the images containing branches in the training set preprocessed in step (1) are subjected to enhancement processing including rotation, flipping and color jittering to obtain a data-enhanced training set.

[0033] Furthermore, after the first step and before the second step of step (3), the training set in the first step is rotated, flipped, and color-jittered to obtain a data-enhanced training set; at the same time, the corresponding labeled data is also subjected to the same rotation and flipping operations to ensure that the coordinates of the labeled data and the branch opening points after data enhancement are consistent.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] This method overcomes the difficulties existing automated IVOCT vascular branch recognition and branch opening point detection methods face in recognizing images containing severe blood artifacts, unclear vessel walls, and when the probe is retracted into a plastic catheter. The dataset used eliminates the need for manual removal of image fragments containing severe blood artifacts and when the probe is retracted into the plastic catheter, nor does it require vessel contour recognition or manual definition of vessel features. The Transformer-based deep learning model better understands image context, and the self-attention mechanism better captures long-range dependencies. The algorithm's strong generalization capabilities improve the accuracy of vascular branch recognition and opening point detection in IVOCT images. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flowchart of the IVOCT vascular branch recognition and branch opening point detection method based on deep learning.

[0037] Figure 2 Schematic diagram of the Vision Transformer model structure

[0038] Figure 3 Transformer module structure diagram

[0039] Figure 4 Decoder module structure diagram

[0040] Figure 5 IVOCT vascular branch identification and opening point detection results DETAILED DESCRIPTION

[0041] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. The described specific embodiments are only used to explain the present invention and are not intended to limit the present invention.

[0042] like Figure 1 As shown, the present invention proposes an IVOCT vascular branch identification and branch opening point detection method comprising the following steps:

[0043] The present invention proposes a deep learning-based method for IVOCT vascular branch identification and branch opening point detection. The method first performs a logarithmic transformation on the original IVOCT image and converts it to rectangular coordinates to obtain the network input image. Data enhancement is performed using horizontal flipping, rotation, and color dithering. A vascular branch identification model is used to identify whether the image contains branches. The image containing branches is then input into a branch opening point detection model to detect the coordinates of the vascular branch opening points. The method comprises the following steps:

[0044] 1. A deep learning-based IVOCT vascular branch recognition and branch opening point detection method, comprising the following steps:

[0045] (1) IVOCT image preprocessing:

[0046] The collected IVOCT images are logarithmically transformed to highlight the details of the low grayscale parts of the image, and the images are transformed into a rectangular coordinate system to facilitate image annotation, and finally the preprocessed data set is obtained.

[0047] The preprocessed IVOCT image sequence is divided into training set and test set as needed.

[0048] (2) Identification of vascular branches:

[0049] Step 1: Data Augmentation

[0050] In order to avoid the imbalance between samples with branches and samples without branches, the images containing branches in the preprocessed training set are rotated, flipped, and color-dithered to obtain a data-augmented training set, thereby generating realistic IVOCT images and improving the generalization ability of the algorithm.

[0051] Step 2: Train the model

[0052] The resolution of the training set after data enhancement obtained in the first step is adjusted to 224×224, and the single-channel grayscale image is adjusted to a three-channel image through a copy operation, and input into the vascular branch recognition model to train the vascular branch recognition model.

[0053] The vascular branch recognition model is a Vision Transformer model based on the self-attention mechanism, as shown in the following diagram: Figure 2 As shown in Figure 2, the model consists of three modules: Embedding layer, Transformer encoder and classification layer.

[0054] First, the image is input into the Embedding layer. The Embedding layer takes an image Divide into 196 image blocks, map these image blocks through the convolution layer and flatten them into a sequence A classification feature vector, class token, is then added to the sequence to extract image features for classification. Finally, the resulting 197×768 sequence is added to the positional encoding of the same dimension so that the model can learn the positional relationships between different image patches.

[0055] The position-encoded sequence is input into the Transformer encoder. The Transformer encoder consists of 12 stacked Transformer modules, each of which includes a layer normalization module, a multi-head attention module, and a multi-layer perceptron module. Layer normalization normalizes each sample and accelerates training. The Transformer encoder learns long-range image dependencies and outputs image features.

[0056] Finally, the output classification feature vector class token is extracted and input into the fully connected layer for weighted summation to obtain the final classification result.

[0057] Step 3: Test the model

[0058] After the training is completed, the test set is input into the branch recognition model to obtain the classification results of the test set.

[0059] (3) Detection of the opening point of blood vessel branches:

[0060] Step 1: Construct a dataset for detecting blood vessel branch openings

[0061] All images with branches are extracted from the preprocessed training set and test set obtained in (1) as the training set and test set for vascular branch opening point detection.

[0062] The branch opening points are the two dividing points where the branch and the main vessel intersect, and the branch opening point coordinates are used to construct the annotation data.

[0063] Step 2: Data Enhancement

[0064] The training set from the first step is rotated, flipped, and color-dithered to obtain the data-augmented training set. The corresponding labeled data is also rotated and flipped in the same way to ensure that the coordinates of the branch opening points of the labeled data and the data-augmented data are consistent.

[0065] Step 3: Train the branch opening point detection model

[0066] The training set image resolution obtained in the second step is resized to 256×192 through zero padding, and the single-channel grayscale image is resized to a three-channel image through a copy operation. This is then fed into the branch opening point detection model. The branch opening point detection model is a Transformer-based key point detection model consisting of three modules: encoder, decoder, and post-processing.

[0067] The encoder is composed of an Embedding layer and 12 Transformer modules stacked together. The Embedding layer converts an image Divide into 192 image blocks, map these image blocks through the convolution layer and flatten them into a sequence Finally, the resulting 192×768 sequence is added to the positional encoding of the same dimension so that the model can learn the positional relationship between different image patches.

[0068] The position-encoded sequence is sequentially input into 12 Transformer modules. The Transformer modules are the same as those in (1). The output image features are normalized to 16×12×768 dimensions, which is denoted as

[0069] The encoder output F out Input to the decoder. The decoder includes two deconvolution modules and a predictor, and its structural diagram is shown as follows: Figure 4 As shown in Figure 2. The deconvolution module includes a deconvolution layer, a batch normalization layer, and a ReLU activation layer. The feature map F output by the encoder out The feature map is then fed into a deconvolution layer, a batch normalization layer, and a ReLU activation layer to double the size of the feature map. The feature map is then fed into a second deconvolution module to quadruple the size of the feature map. Finally, the feature map is fed into a convolution layer with a kernel size of 1×1 to generate a heat map.

[0070] H=Conv 1×1 (Deconv(Deconv(F out ))) (1)

[0071] A two-dimensional Gaussian distribution heat map K is generated with the branch opening point coordinates as the center as the supervision signal, and the MSE loss is calculated with the heat map predicted by the model. The AdamW algorithm is used to optimize the model parameters to reduce the loss. The training is stopped when the model reaches the predetermined number of iterations.

[0072]

[0073] where x i ,y i (i=1, 2) are the coordinates of the branch opening point, and σ is the standard deviation of the Gaussian function.

[0074] The post-processing module uses an unbiased data processing method (UDP) to extract the pixel coordinates with the highest score from the heat map and correct the deviation caused by image flipping, and finally obtains the coordinates of the branch opening point.

[0075] Step 4: Test the model

[0076] After the model training is completed, the test set is input into the branch opening point detection model, and the model outputs the coordinates corresponding to the two branch opening points.

[0077] The present invention overcomes the problem that existing automatic IVOCT vascular branch recognition and branch opening point detection methods have difficulty in recognizing images containing severe blood artifacts, unclear vascular walls, and when the probe is withdrawn into the plastic catheter. The dataset used does not require manual removal of image fragments containing severe blood artifacts and when the probe is withdrawn into the plastic catheter, nor does it require vascular contour recognition and manual definition of vascular features. The use of a Transformer-based deep learning model can better understand image context information, and the self-attention mechanism can better capture long-distance dependencies. The algorithm has strong generalization capabilities, which improves the accuracy of IVOCT image vascular branch recognition and opening point detection. The test set size used for branch recognition of the present invention is 5399 images, and the recognition accuracy is 93.14%. The dataset size used for opening point detection is 689 images, and the opening distance error of the detection result is 0.4548mm.

[0078] The blood vessel branches and their opening points identified by the Transformer-based deep learning method of the present invention are as follows: Figure 5 As shown, it can be clearly seen that the method provided by the present invention can overcome the situation when the blood vessels contain complex blood artifacts and the blood vessel walls are unclear, and at the same time, it does not require manual extraction of blood vessel features and correctly identifies blood vessel branches and branch opening points.

Claims

1. A deep learning-based IVOCT vascular branch opening point detection method, comprising the following steps: (1) IVOCT image preprocessing Performing logarithmic transformation on the collected IVOCT images, and transforming the logarithmic transformed IVOCT images into a rectangular coordinate system to obtain a preprocessed data set; Divide the preprocessed IVOCT image sequence into training set and test set as needed; (2) Identification of vascular branches: Adjust the resolution of the pre-processed IVOCT images in the training set and input them into the vascular branch recognition model to train the vascular branch recognition model; The vascular branch recognition model is a Vision Transformer model based on the self-attention mechanism. The model includes three modules: the Embedding layer, the Transformer encoder, and the classification layer. The Embedding layer is used to divide an image into several blocks and map and tile the image blocks into a sequence through the convolution layer. The mapped and tiled sequence is input into the Transformer encoder after position encoding. The Transformer encoder is composed of several stacked Transformer modules, each of which includes a layer normalization module, a multi-head attention module, and a multi-layer perceptron module. The Transformer encoder is used to learn long-range image dependencies and output image features. The learned image features are input into the fully connected layer for weighted summation to obtain the final classification result; After the vascular branch recognition model training is completed, the test set is input into the vascular branch recognition model to obtain the classification results of the test set; (3) Detection of the opening point of blood vessel branches: Step 1: Construct a dataset for detecting blood vessel branch opening points. The method is as follows: Based on the vascular branch recognition model obtained in step (2), images with branches are extracted as training sets and test sets for vascular branch opening point detection. The branch opening points are the two dividing points where the branches and the main blood vessels intersect. The branch opening point coordinates are used as key point coordinates to construct the annotation data. Step 2: Train the branch opening point detection model as follows: Adjust the resolution of the training set images obtained in the first step and input them into the branch opening point detection model; The branch opening point detection model is a Transformer-based key point detection model, including three modules: encoder, decoder, and post-processing. The encoder is composed of an embedding layer and several stacked Transformer modules. The embedding layer divides an image into several blocks and maps and flattens these blocks into a sequence through a convolutional layer. Each Transformer module includes a layer normalization module, a multi-head attention module, and a multi-layer perceptron module. The encoder is used to learn long-range dependencies between images and output image features. The learned image features are normalized and then dimensionally adjusted to obtain a feature map. The decoder includes two deconvolution modules and a predictor. The deconvolution module includes a deconvolution layer, a batch normalization layer, and a ReLU activation layer. The feature map output by the encoder is sequentially input into the two deconvolution modules to enlarge the feature map size. The enlarged feature map is then input into a convolution layer with a kernel size of 1×1 to obtain a heat map predicted by the model. Generate a two-dimensional Gaussian distribution heat map centered on the key point coordinates as a supervisory signal, calculate the MSE loss with the heat map predicted by the model, use the AdamW algorithm to optimize the model parameters to reduce the loss, and stop training when the model reaches the predetermined number of iterations; The post-processing module uses unbiased data processing methods to extract key point coordinates from the model-predicted heat map and make corrections; After the model training is completed, the test set is input into the branch opening point detection model, and the model outputs the coordinates corresponding to the two opening points.

2. The IVOCT blood vessel branch opening point detection method according to claim 1, characterized in that: After step (1) and before step (2), the images containing branches in the training set preprocessed in step (1) are subjected to enhancement processing including rotation, flipping and color jittering to obtain a data-enhanced training set.

3. The IVOCT blood vessel branch opening point detection method according to claim 1, characterized in that: After the first step and before the second step in step (3), the training set in the first step is rotated, flipped, and color-jittered to obtain the data-enhanced training set; at the same time, the corresponding labeled data is also subjected to the same rotation and flipping operations to ensure that the coordinates of the labeled data and the branch opening points after data enhancement are consistent.

Images