Novel esophageal cavity internal and external blood vessel classification method based on endoscopic image
Through deep learning technology, the classification of endoscopic images of internal and external esophageal lumen was solved, and the problem of difficulty in predicting external vascular in the middle and external esophageal cavity was achieved, and efficient and accurate vascular typing and treatment strategy adjustment was achieved.
Patent Information
- Application Number
- CN202411178308.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-20
AI Technical Summary
The prior art is difficult to accurately predict the condition of blood vessels inside and outside the esophageal cavity during endoscopy, which affects the efficiency and safety of endoscopic treatment. The process of finding the extraluminal vascular mass through contrast enhancement CT imaging examination is cumbersome and time-consuming.
Deep learning method is used to analyze the endoscopic images of patients with esophageal varicose veins, and segment the endoscopic image automatically recognizes the endoscopic image, so as to achieve the prediction of blood vessels inside and outside the esophageal cavity without CT images. Classification, vascular segmentation and classification are performed using 3D U-Net and Transformer image classification models.
Standardized vascular grading is achieved, subjective deviations from doctors are reduced, segmentation accuracy and efficiency are improved, and prediction and classification of intraluminal blood vessels can be completed in endoscopy without CT imaging, and the treatment strategy can be assisted.
Smart Images
Figure CN119169673A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for classifying blood vessels inside and outside the esophagus cavity, and in particular to a novel method for classifying blood vessels inside and outside the esophagus cavity based on endoscopic images. Background Art
[0002] The esophageal varicose veins that can be seen and treated endoscopically are limited to the mucosa and submucosal layer, while there may be varicose vascular masses outside the esophagus, which seriously increase the risk of endoscopic treatment and affect the efficacy of endoscopic treatment. These extraluminal vascular masses must be detected through contrast-enhanced CT imaging examinations. The operation is more troublesome and time-consuming.
[0003] There is an urgent need in the market to be able to predict the status of the extraluminal vascular masses in the esophagus based on the endoscopic characteristics of esophageal varices during endoscopic examination, classify the blood vessels inside and outside the esophageal cavity, and adjust the endoscopic treatment strategy in a timely manner. Summary of the invention
[0004] Purpose of the invention: The purpose of the invention is to provide a new method for classifying esophageal extraluminal vessels based on endoscopic images. The invention uses a deep learning method to perform medical image analysis on endoscopic images of the esophagus of patients with esophageal varices, classify the severity of extraluminal esophageal varices, and classify the intraluminal and extraluminal vessels of esophageal and gastric varices.
[0005] The present invention completes segmentation by analyzing the relevant ROI of portal vein contrast-enhanced CT of patients with esophageal varices, including the esophagus, intra-esophageal vessels, and extra-esophageal vessels. After obtaining the ROI area, the vessels inside and outside the esophageal cavity are analyzed and divided into intra-cavity varicose veins, mixed varicose veins, and extra-cavity varicose veins. Figure 1 shown.
[0006] After classifying the intraluminal and extraluminal blood vessels of the patient's esophagus according to contrast-enhanced CT, the corresponding esophageal endoscopic images are obtained, the model is trained, and the construction of the endoscopic image automatic recognition model is completed. The tested and verified endoscopic image automatic recognition model can achieve the prediction and classification of esophageal varicose veins inside and outside the esophageal cavity without the need for patients to undergo CT imaging examinations, as long as the esophageal varicose vein images during endoscopic examinations are used.
[0007] Compared with the difficulty of evaluating the classification of esophageal varicose veins by naked eye observation, the three-dimensional segmentation results can fully consider the vascular morphology of the esophageal cavity in multiple two-dimensional planes, such as Figure 3As shown, pixel-level labels can standardize the classification. After the data pairing module completes the pairing, the esophageal intraluminal and extraluminal vascular classification is obtained through CT images, and the corresponding endoscopic images are labeled at the same time. After obtaining the endoscopic images and the corresponding CT-guided classification labels, an image classification model is constructed and trained, and intraluminal and extraluminal vascular prediction can be performed from the endoscopic images, realizing the intraluminal and extraluminal vascular classification of esophageal varices.
[0008] Technical solution: The novel endoscopic image-based classification method for esophageal intraluminal and extraluminal blood vessels of the present invention comprises the following steps:
[0009] (1) Esophageal varices patient data pairing module, used to pair contrast-enhanced CT and esophageal endoscopy images of patients with esophageal varices;
[0010] (2) A deep learning esophageal intraluminal and extraluminal vein segmentation module is used to segment the esophagus, intraluminal vessels, and extraluminal vessels on enhanced CT and output the segmentation results;
[0011] (3) The module for detecting and grading extraluminal esophageal varicose veins in endoscopic images guided by contrast-enhanced CT is used to analyze the CT segmentation results and obtain the gold standard for CT-guided extraluminal esophageal vascular detection and classification. The endoscopic images and labels are input into the Transformer image classification model for training, and a model is obtained that can complete the detection of extraluminal esophageal vessels by endoscopic images without the need for CT images.
[0012] Furthermore, the pairing method in step (1) is as follows: after completing standardization processing such as resampling and normalization of the CT data, the region of interest is marked, and the endoscopic image and CT of the same patient are matched into a CT-endoscopic data pair.
[0013] Furthermore, the region of interest includes: esophagus, varicose vessels in the esophageal cavity, and varicose vessels outside the esophageal cavity.
[0014] Furthermore, the enhanced CT and corresponding pixel-level labels in module 1 are obtained, and a deep learning neural network is trained to automatically segment the CT data. After the obtained segmentation network is tested and meets the usage standards, it is used to obtain the ROI.
[0015] Furthermore, the present invention uses 3D U-Net to segment the enhanced CT.
[0016] Furthermore, the 3D U-Net consists of an encoder, a decoder, and a bottleneck layer.
[0017] Furthermore, the encoder extracts features from the input image by stacking multiple convolutional layers. Each convolutional layer expresses a certain perspective or abstract level of the input image. The convolutional block consists of a 3*3*3 convolutional layer, BatchNormalization, and Leaky ReLU. The two convolutional blocks are connected by residual operations to alleviate the gradient vanishing problem. After feature extraction, the spatial resolution of the feature map is gradually reduced by downsampling, while the depth of the feature map is increased.
[0018] The decoder accepts the abstract feature map from the encoder and reconstructs the structure of the original input image. Through upsampling operations, it gradually increases the spatial resolution of the feature map and recovers the details and spatial information lost in the encoder.
[0019] The bottleneck layer retains the most important semantic information through further convolution, transition decoder and encoder, improves the feature expression ability of the network, and captures a wider range of contextual information in the input image.
[0020] Furthermore, the method of step (3) is:
[0021] A. Segmentation: Input contrast-enhanced CT into the segmentation model to obtain the results of the esophagus, esophageal varicose vessels, and esophageal varicose vessels;
[0022] B. Quantification: Quantification is performed on the segmentation results. The volume of varicose vessels only outside the esophageal cavity and varicose vessels only inside the esophageal cavity is calculated, that is, the number of pixels multiplied by the spacing of the voxels. Patients are classified according to the ratio of the volume of varicose vessels inside and outside the esophageal cavity. The ratio is less than 0.1 for extracavitary type, the volume ratio is greater than 0.1 and less than 10 for mixed type, and the ratio is greater than 10 for intracavitary type.
[0023] C. Image classification: After obtaining the extraluminal vascular severity grading labels guided by CT image segmentation results for patients with esophageal varices, the images and labels are input into the Transformer-based image classification network for training. The Transformer divides the input image into fixed-size tiles, then flattens each tile into a one-dimensional vector and passes it to the Transformer encoder as a sequence input; the vector of each tile first passes through a learnable embedding layer and retrograde encoding to retain the spatial information of each tile; the self-attention mechanism is used in the encoder to capture the dependencies between different positions in the sequence; the output of the encoder passes through the classifier to output the final category prediction.
[0024] Compared with the prior art, the present invention has the following beneficial effects:
[0025] 1. Standardized grading. The three-dimensional segmentation results fully consider the esophageal varicose veins in multiple planes and complete the severity grading by quantifying the target. The labels obtained by this detection and grading are more standardized than the two-dimensional observation evaluation and reduce the subjective experience bias of the observer.
[0026] 2. 3D segmentation is completed through deep learning neural network. Compared with manual segmentation, the end-to-end segmentation network greatly reduces the labeling time; compared with traditional segmentation methods, it reduces the operation steps and greatly improves the segmentation accuracy.
[0027] 3. After the model is established, CT images are not needed. The prediction and classification of the blood vessels inside and outside the esophageal varices of patients can be completed only by endoscopic images during the examination. Based on the Transformer image classification model, learn more about the implicit features of esophageal varices under endoscopy, and realize timely confirmation and adjustment of treatment strategies during endoscopic examinations. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 The varicose vein analysis results of the present invention are divided into intracavitary varicose veins, mixed varicose veins and extracavitary varicose veins;
[0029] Figure 2 It is a schematic diagram of the classification method module;
[0030] Figure 3 This is the blood vessel segmentation result based on three-dimensional segmentation;
[0031] Figure 4 Schematic diagram of the deep learning module for segmenting blood vessels inside and outside the esophagus;
[0032] Figure 5 Schematic diagram of quantification after segmentation. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be further described below.
[0034] The novel endoscopic image-based classification method for esophageal intraluminal and extraluminal blood vessels of this embodiment includes the following steps:
[0035] 1) Esophageal varices patient data matching module
[0036] Contrast-enhanced CT and endoscopic images of patients with esophageal varices were collected. After the CT data was resampled, normalized, and standardized, the doctors completed the annotation of the regions of interest (for training the segmentation model), including the esophagus, esophageal varicose vessels, and esophageal varicose vessels. The endoscopic images and CT of the same patient were matched into a CT-endoscopic data pair, and the grading results of the quantified enhanced CT segmentation results were the gold standard for the corresponding endoscopic images.
[0037] 2) Deep learning esophageal intraluminal and extraluminal blood vessel segmentation module
[0038] The enhanced CT and corresponding pixel-level labels in module 1 are obtained, and a deep learning neural network is trained to automatically segment the CT data. After the obtained segmentation network is tested and meets the usage standards, it is used to obtain ROI.
[0039] In addition, there is a jump connection between the encoder and decoder at the corresponding level, which fuses the feature map in the encoder and the feature map in the decoder, retaining more spatial information and details and improving the segmentation precision and accuracy.
[0040] 3) Detection and grading of extraluminal esophageal varices using enhanced CT-guided endoscopic images
[0041] ① Segmentation. Input contrast-enhanced CT into the segmentation model to obtain the results of the esophagus, esophageal varicose vessels, and esophageal varicose vessels.
[0042] ② Quantification. Quantify the segmentation results. Figure 5 , calculate the volume of varicose vessels only outside the esophageal cavity and varicose vessels only inside the esophageal cavity, that is, the number of pixels multiplied by the spacing of voxels. Patients were classified according to the ratio of varicose vessels inside and outside the esophageal cavity. The ratio was less than 0.1 for extracavitary type, the volume ratio was greater than 0.1 and less than 10 for mixed type, and the ratio was greater than 10 for intracavitary type.
[0043] ③ Image classification. After obtaining the extraluminal vascular severity grading labels guided by CT image segmentation results for patients with esophageal varices, the images and labels are input into the Transformer-based image classification network for training. Transformer divides the input image into fixed-size tiles, then flattens each tile into a one-dimensional vector and passes it to the Transformer encoder as a sequence input; the vector of each tile first passes through a learnable embedding layer and reverse encoding to retain the spatial information of each tile; the self-attention mechanism is used in the encoder to capture the dependencies between different positions in the sequence; the output of the encoder passes through the classifier to output the final category prediction.
[0044] In actual operation, a new classification method for esophageal intraluminal and extraluminal blood vessels based on endoscopic images was obtained. In actual use, the endoscopic images were directly input into the Transformer network to obtain the classification of esophageal intraluminal and extraluminal varicose veins, which assisted in determining treatment decisions.
[0045] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.
Claims
1. A new method for classifying esophageal intraluminal and extraluminal blood vessels based on endoscopic images, characterized in that: Classification is achieved through collaboration of three modules, including: (1) Esophageal varices patient data pairing module, used to pair contrast-enhanced CT and esophageal endoscopy images of patients with esophageal varices; (2) A deep learning esophageal intraluminal and extraluminal vein segmentation module is used to segment the esophagus, intraluminal vessels, and extraluminal vessels on enhanced CT and output the segmentation results; (3) The module for detecting and grading extraluminal esophageal varices in contrast-enhanced CT-guided endoscopic images is used to analyze the CT segmentation results. The endoscopic images are used to obtain the gold standard for CT-guided prediction and classification of extraluminal esophageal vessels. The endoscopic images and labels are input into the Transformer image classification model for training to obtain the final category prediction.
2. The novel endoscopic image-based classification method for esophageal intraluminal and extraluminal blood vessels according to claim 1, characterized in that: The pairing method in step (1) is as follows: after completing standardization processing such as resampling and normalization of the CT data, the region of interest is marked, and the endoscopic image and CT of the same patient are matched into a CT-endoscopic data pair.
3. The novel endoscopic image-based classification method for esophageal intraluminal and extraluminal blood vessels according to claim 2, characterized in that: The region of interest includes: esophagus, varicose vessels in the esophageal cavity, and varicose vessels outside the esophageal cavity.
4. The novel esophageal intraluminal and extraluminal blood vessel classification method based on endoscopic images according to claim 1, characterized in that: The enhanced CT and corresponding pixel-level labels in module (1) are obtained, and a deep learning neural network is trained to automatically segment the CT data. After the obtained segmentation network is tested and meets the usage standards, it is used to obtain the ROI.
5. The novel endoscopic image-based classification method for esophageal intraluminal and extraluminal blood vessels according to claim 4, characterized in that: The present invention uses 3D U-Net to segment enhanced CT.
6. The novel endoscopic image-based classification method for esophageal intraluminal and extraluminal blood vessels according to claim 5, characterized in that: The 3D U-Net consists of an encoder, a decoder, and a bottleneck layer.
7. The novel endoscopic image-based classification method for esophageal intraluminal and extraluminal blood vessels according to claim 6, characterized in that: The encoder extracts features from the input image by stacking multiple convolutional layers. Each convolutional layer expresses a certain perspective or abstract level of the input image. The convolutional block consists of a 3*3*3 convolutional layer, Batch Normalization, and Leaky ReLU. The two convolutional blocks are connected by residual operations to alleviate the gradient vanishing problem. After feature extraction, the spatial resolution of the feature map is gradually reduced through downsampling, while the depth of the feature map is increased. The decoder accepts the abstract feature map from the encoder and reconstructs the structure of the original input image. Through upsampling operations, it gradually increases the spatial resolution of the feature map and recovers the details and spatial information lost in the encoder. The bottleneck layer retains the most important semantic information through further convolution, transition decoder and encoder, improves the feature expression ability of the network, and captures a wider range of contextual information in the input image.
8. The novel esophageal intraluminal and extraluminal blood vessel classification method based on endoscopic images according to any one of claims 1 to 7, characterized in that: The method of step (3): A. Segmentation: Input contrast-enhanced CT into the segmentation model to obtain the results of the esophagus, esophageal varicose vessels, and esophageal varicose vessels; B. Quantification: Quantification is performed on the segmentation results. The volume of varicose vessels only outside the esophageal cavity and varicose vessels only inside the esophageal cavity is calculated, that is, the number of pixels multiplied by the spacing of the voxels. Patients are classified according to the ratio of the volume of varicose vessels inside and outside the esophageal cavity. The ratio is less than 0.1 for extracavitary type, the volume ratio is greater than 0.1 and less than 10 for mixed type, and the ratio is greater than 10 for intracavitary type. C. Image classification: After obtaining the extraluminal vascular severity grading labels guided by CT image segmentation results for patients with esophageal varices, the images and labels are input into the Transformer-based image classification network for training. The Transformer divides the input image into fixed-size tiles, then flattens each tile into a one-dimensional vector and passes it to the Transformer encoder as a sequence input; the vector of each tile first passes through a learnable embedding layer and retrograde encoding to retain the spatial information of each tile; the self-attention mechanism is used in the encoder to capture the dependencies between different positions in the sequence; the output of the encoder passes through the classifier to output the final category prediction.
Citation Information
Cited By
Colorectal cancer postoperative risk assessment method, device and equipment and storage medium
CN121545753A