Automatic detection method and system for coronary artery stenosis

Abstract

An automatic method and system for detecting coronary artery stenosis solves the problems of cumulative error relying on segmentation structures and difficulty in extracting inter-frame correlation information from coronary angiography sequences, thereby quickly and effectively obtaining accurate stenosis detection results and providing more precise stenosis locations for subsequent coronary artery lesion analysis. The method includes: (1) semantic feature map and stenosis proposal extraction, automatically extracting semantic feature maps and stenosis proposals from each frame of the input coronary angiography sequence; (2) spatiotemporal feature aggregation, using semantic feature maps and stenosis proposals to generate spatiotemporal RoI features of the coronary angiography sequence, and performing spatiotemporal feature aggregation by learning the contextual correlation of spatiotemporal RoI features to generate enhanced RoI features; (3) stenosis result prediction, inputting the enhanced RoI features into a multi-task output network to predict the stenosis detection result of the coronary angiography sequence.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and more particularly to an automatic detection method and system for coronary artery stenosis. Background Technology

[0002] Coronary artery disease is a leading cause of death in humans. It is caused by the accumulation of atherosclerotic plaques in the inner walls of the coronary arteries, leading to narrowing of the coronary artery lumen, which restricts blood flow to the myocardium, causing myocardial ischemia and increasing the risk of myocardial infarction. Research on vascular stenosis detection algorithms based on angiographic images is mainly divided into two categories: detection based on coronary artery segmentation structure and detection based on the original image. The method based on coronary artery segmentation structure requires first segmenting the coronary artery lumen structure from the angiographic image using a coronary artery segmentation algorithm. Then, based on prior geometric knowledge of the blood vessels, it analyzes the changing trend of the vessel lumen radius in the segmented structure to determine whether stenosis exists in that vessel. However, this method only considers the geometric morphology of the segmented structure, thus heavily relying on the accuracy of the segmentation results, leading to error accumulation. Furthermore, this method does not consider the vessel location information in the original angiographic image; if the original image contains overlapping vessels, it can lead to false positives or false negatives. The second category is learning-based methods. In recent years, with the rise of deep learning methods, more and more deep learning-based coronary artery stenosis detection algorithms have been proposed. Deep learning-based detectors can effectively extract stenosis-related depth features from coronary angiography images, which helps improve the accuracy of stenosis detection. However, most deep learning-based detectors perform stenosis detection based on single-frame coronary angiography images, without considering the contextual relationships between frames within the same coronary angiography sequence. This leads to inconsistencies in detection results between consecutive frames within the same sequence, limiting the accuracy of coronary artery stenosis detection. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, the technical problem to be solved by the present invention is to provide an automatic method for detecting coronary artery stenosis. This method solves the problems of cumulative error relying on segmentation structures and difficulty in extracting inter-frame correlation information from coronary angiography sequences, thereby quickly and effectively obtaining accurate stenosis detection results and providing more precise stenosis location for subsequent coronary artery lesion analysis.

[0004] The technical solution of this invention is: an automatic detection method for coronary artery stenosis, comprising the following steps:

[0005] (1) Semantic feature map and stenosis proposal extraction: Semantic feature map and stenosis proposal are automatically extracted from each frame of the input coronary angiography sequence;

[0006] (2) Spatiotemporal feature aggregation: Spatiotemporal RoI features of coronary angiography sequences are generated using semantic feature maps and stenosis proposal boxes. Spatiotemporal features are aggregated by learning the contextual association of spatiotemporal RoI features to generate enhanced RoI features.

[0007] (3) Stenosis result prediction: The enhanced RoI features are input into the multi-task output network to predict the stenosis detection results of the coronary angiography sequence.

[0008] This invention first extracts semantic feature maps and stenosis proposal boxes from each frame of coronary angiography images. Then, it aggregates the spatiotemporal context information of the sequence images to enhance stenosis-related features. Finally, it predicts the stenosis detection result for each frame of the image. Therefore, it solves the problem of cumulative error dependent on segmentation structure and the problem of difficulty in extracting inter-frame correlation information of coronary angiography sequences. This allows for the rapid and effective acquisition of accurate stenosis detection results, providing more precise stenosis locations for subsequent coronary artery lesion analysis.

[0009] It also provides an automated coronary artery stenosis detection system, which includes:

[0010] The extraction module is configured to extract semantic feature maps and stenosis proposals, automatically extracting semantic feature maps and stenosis proposals for each frame of the input coronary angiography sequence;

[0011] The aggregation module is configured to perform spatiotemporal feature aggregation, generate spatiotemporal RoI features of coronary angiography sequences using semantic feature maps and stenosis proposal boxes, and perform spatiotemporal feature aggregation by learning the contextual association of spatiotemporal RoI features to generate enhanced RoI features;

[0012] The prediction module, configured to perform narrow outcome prediction, inputs enhanced RoI features into...

[0013] In a multi-task output network, the stenosis detection results of coronary angiography sequences are predicted. Attached Figure Description

[0014] Figure 1 This is a flowchart of the automatic detection method for coronary artery stenosis according to the present invention.

[0015] Figure 2 This is a flowchart of semantic feature map and narrow suggestion box extraction in this invention.

[0016] Figure 3 This is a flowchart of the spatiotemporal tokenization submodule in this invention.

[0017] Figure 4 This is a flowchart of the feature aggregation module in this invention.

[0018] Figure 5This is a flowchart of the narrow result prediction in this invention. Detailed Implementation

[0019] To address the issues of cumulative error due to reliance on segmentation structures and difficulty in extracting inter-frame correlation information from coronary angiography sequences in the prior art, this invention proposes a detection framework based on Transformer spatiotemporal feature aggregation to automatically detect coronary artery stenosis directly from coronary angiography sequences. First, a backbone network and a region proposal network are used to extract semantic feature maps and stenosis proposal boxes for each frame of the coronary angiography sequence. Then, a Transformer spatiotemporal feature aggregation network is used to obtain the long-range spatiotemporal context of the sequence images, enhancing stenosis-related features. Finally, a multi-task output network is used to predict stenosis information in the sequence images, thereby quickly and effectively obtaining accurate stenosis detection results.

[0020] As mentioned above, in order to achieve the above functions, it involves semantic feature map and narrow proposal box extraction based on backbone network and regional proposal network, narrow related feature enhancement based on Transformer spatiotemporal feature aggregation network, and multi-task output network, etc.

[0021] like Figure 1 As shown, this automated method for detecting coronary artery stenosis includes the following steps:

[0022] (1) Semantic feature map and stenosis proposal extraction, from the input coronary angiography sequence.

[0023] The semantic feature map and narrow proposal box are extracted from each frame of the image.

[0024] (2) Spatiotemporal feature aggregation: Spatiotemporal RoI features of coronary angiography sequences are generated using semantic feature maps and stenosis proposal boxes. Spatiotemporal features are aggregated by learning the contextual association of spatiotemporal RoI features to generate enhanced RoI features.

[0025] (3) Stenosis result prediction: The enhanced RoI features are input into the multi-task output network to predict the stenosis detection results of the coronary angiography sequence.

[0026] This invention first extracts semantic feature maps and stenosis proposal boxes from each frame of coronary angiography images. Then, it aggregates the spatiotemporal context information of the sequence images to enhance stenosis-related features. Finally, it predicts the stenosis detection result for each frame of the image. Therefore, it solves the problem of cumulative error dependent on segmentation structure and the problem of difficulty in extracting inter-frame correlation information of coronary angiography sequences. This allows for the rapid and effective acquisition of accurate stenosis detection results, providing more precise stenosis locations for subsequent coronary artery lesion analysis.

[0027] Preferably, such as Figure 2As shown, step (1) includes:

[0028] (1.1) For the input coronary angiography sequence, a backbone network composed of ResNet-50 and feature pyramid is used to extract multi-scale features of each frame of the image and generate semantic feature maps.

[0029] (1.2) Input the generated feature map into the region proposal network to generate narrow proposal boxes of different aspect ratios and sizes to adapt to narrow lesions of different scales and proportions.

[0030] Preferably, such as Figure 3 As shown, step (2) includes:

[0031] (2.1) For the narrow proposal box in the current frame of the sequence image, move it in the four directions of up, down, left and right within the local window to generate spatial RoIs, and include the RoI represented by the narrow proposal box in the spatial RoIs; use the same method to translate the narrow proposal box of the current frame on other frames to generate temporal RoIs; combine the spatial RoIs and temporal RoIs to get the spatiotemporal RoIs related to the narrow proposal box in the current frame;

[0032] (2.2) By using the RoI Align operation, spatiotemporal RoIs are used to extract spatiotemporal RoI features of the same size on the detected semantic feature map;

[0033] (2.3) Vectorize the spatiotemporal RoI features and perform linear projection to generate spatiotemporal tokens related to the narrow proposal box on the current frame.

[0034] Preferably, such as Figure 4 As shown, step (2) further includes:

[0035] (2.4) Encode the obtained spatiotemporal tokens by position, add them to the spatiotemporal tokens, and record the spatiotemporal position of the tokens;

[0036] (2.5) Input the spatiotemporal tokens with added position encoding into the Transformer encoder to learn the long-range contextual association of the spatiotemporal tokens. The Transformer encoder consists of four consecutive Transformer layers.

[0037] (2.6) The output of the Transformer encoder is fused to inject the features of the neighboring spatiotemporal RoI represented by the spatiotemporal tokens into the features of the RoI represented by the narrow proposal box, so as to obtain the token after spatiotemporal feature aggregation.

[0038] (2.7) Linear projection and tensor quantization are performed on the tokens after spatiotemporal feature aggregation to obtain enhanced RoI features representing narrow proposal boxes.

[0039] Preferably, such as Figure 5 As shown, step (3) includes:

[0040] (3.1) Global feature modeling and dimensionality reduction are performed on the extracted enhanced RoI features through a fully connected layer;

[0041] (3.2) Input the dimensionality-reduced features into the stenosis location regression output branch and the stenosis proposal box classification output branch respectively, correct the detection results of the stenosis proposal box, and predict the final coronary artery stenosis detection results.

[0042] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. When executed, the program includes the steps of the methods of the above embodiments. The storage medium can be ROM / RAM, magnetic disk, optical disk, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes an automatic coronary artery stenosis detection system, which is typically represented in the form of functional modules corresponding to the steps of the method. The system includes:

[0043] The extraction module is configured to extract semantic feature maps and stenosis proposals, automatically extracting semantic feature maps and stenosis proposals for each frame of the input coronary angiography sequence;

[0044] The aggregation module, configured to perform spatiotemporal feature aggregation, generates spatiotemporal RoI features of coronary angiography sequences using semantic feature maps and stenosis proposal boxes, and learns the spatiotemporal RoIs.

[0045] The contextual association of features is used to aggregate spatiotemporal features, generating enhanced RoI features;

[0046] The prediction module, configured to predict stenosis results, inputs enhanced RoI features into a multi-task output network to predict stenosis detection results from coronary angiography sequences.

[0047] Preferably, the extraction module performs:

[0048] (1.1) For the input coronary angiography sequence, a backbone network composed of ResNet-50 and feature pyramid is used to extract multi-scale features of each frame of the image and generate semantic feature maps.

[0049] (1.2) Input the generated feature map into the region proposal network to generate narrow proposal boxes of different aspect ratios and sizes to adapt to narrow lesions of different scales and proportions.

[0050] Preferably, the aggregation module performs:

[0051] (2.1) For the narrow proposal box in the current frame of the sequence image, move it in the four directions of up, down, left and right within the local window to generate spatial RoIs, and include the RoI represented by the narrow proposal box in the spatial RoIs; use the same method to translate the narrow proposal box of the current frame on other frames to generate temporal RoIs; combine the spatial RoIs and temporal RoIs to get the spatiotemporal RoIs related to the narrow proposal box in the current frame;

[0052] (2.2) By using the RoI Align operation, spatiotemporal RoIs are used to extract spatiotemporal RoI features of the same size on the detected semantic feature map;

[0053] (2.3) Vectorize the spatiotemporal RoI features and perform linear projection to generate spatiotemporal tokens related to the narrow proposal box on the current frame.

[0054] Preferably, the aggregation module further performs:

[0055] (2.4) Encode the obtained spatiotemporal tokens by position, add them to the spatiotemporal tokens, and record the spatiotemporal position of the tokens;

[0056] (2.5) Input the spatiotemporal tokens with added position encoding into the Transformer encoder to learn the long-range contextual association of the spatiotemporal tokens. The Transformer encoder consists of four consecutive Transformer layers.

[0057] (2.6) The output of the Transformer encoder is fused to inject the features of the neighboring spatiotemporal RoI represented by the spatiotemporal tokens into the features of the RoI represented by the narrow proposal box, so as to obtain the token after spatiotemporal feature aggregation.

[0058] (2.7) Linear projection and tensor quantization are performed on the tokens after spatiotemporal feature aggregation to obtain enhanced RoI features representing narrow proposal boxes.

[0059] Preferably, the prediction module performs:

[0060] (3.1) Global feature modeling and dimensionality reduction are performed on the extracted enhanced RoI features through a fully connected layer;

[0061] (3.2) Input the dimensionality-reduced features into the stenosis location regression output branch and the stenosis proposal box classification output branch respectively, correct the detection results of the stenosis proposal box, and predict the final coronary artery stenosis detection results.

[0062] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for automatic detection of coronary stenosis, characterized in that: It comprises the following steps: (1) semantic feature map and narrow suggestion box extraction, automatically extracting the semantic feature map and narrow suggestion box of each frame image from the input coronary angiography sequence; (2) spatio-temporal feature aggregation, generating spatio-temporal RoI features of the coronary angiography sequence by using the semantic feature map and the narrow suggestion box, and performing spatio-temporal feature aggregation by learning the context association of the spatio-temporal RoI features to generate enhanced RoI features; (3) narrow result prediction, inputting the enhanced RoI features into a multi-task output network to predict the narrow detection result of the coronary angiography sequence; The step (2) comprises: (2.1) for the narrow suggestion box of the current frame in the sequence image, moving in the up, down, left and right four directions within the local window to generate spatial RoIs, and the RoI represented by the narrow suggestion box is also included in the spatial RoIs; In the same way, the narrow suggestion box of the current frame is translated on other frames to generate time sequence RoIs; The spatial RoIs and time sequence RoIs are combined as the spatio-temporal RoIs related to the narrow suggestion box of the current frame; (2.2) by RoI Align operation, using spatio-temporal RoIs to extract spatio-temporal RoI features with the same size on the detected semantic feature map; (2.3) vectorizing the spatio-temporal RoI features and performing linear projection to generate spatio-temporal tokens related to the narrow suggestion box of the current frame; (2.4) position coding is performed on the obtained spatio-temporal tokens, which are added to the spatio-temporal tokens to record the spatio-temporal position of the tokens; (2.5) input the spatio-temporal tokens added with position coding into the Transformer encoder to learn the long-range context association of the spatio-temporal tokens, wherein the Transformer encoder is composed of four consecutive Transformer layers; (2.6) the output result of the Transformer encoder is fused to inject the adjacent spatio-temporal RoI features represented by the spatio-temporal tokens into the RoI features represented by the narrow suggestion box, and obtain the token after spatio-temporal feature aggregation; (2.7) linear projection and tensorization are performed on the token after spatio-temporal feature aggregation to obtain the enhanced RoI features representing the narrow suggestion box.

2. The method of claim 1, wherein: The step (1) comprises: (1.1) for the input coronary angiography sequence, a backbone network composed of ResNet-50 and feature pyramid is used to extract multi-scale features of each frame image to generate semantic feature maps; (1.2) input the generated feature map into the region proposal network to generate narrow suggestion boxes with different aspect ratios and sizes to adapt to narrow lesions of different scales and proportions.

3. The method of claim 2, wherein: The step (3) comprises: (3.1) performing global feature modeling and dimension reduction on the extracted enhanced RoI features through a fully connected layer; (3.2) input the reduced features into the narrow position regression output branch and the narrow suggestion box classification output branch respectively to correct the detection result of the narrow suggestion box and predict the final coronary narrow detection result.

4. A system for automatic detection of coronary stenosis, characterized by: It comprises: an extraction module configured to extract semantic feature maps and narrow suggestion boxes, the extraction module being configured to automatically extract semantic feature maps and narrow suggestion boxes of each frame of image from an input coronary angiography sequence; an aggregation module configured to perform spatio-temporal feature aggregation, the aggregation module being configured to generate spatio-temporal RoI features of the coronary angiography sequence by using the semantic feature maps and the narrow suggestion boxes, and to perform spatio-temporal feature aggregation by learning context association of the spatio-temporal RoI features to generate enhanced RoI features; a prediction module configured to perform narrow result prediction, the prediction module being configured to input the enhanced RoI features into a multi-task output network to predict narrow detection results of the coronary angiography sequence. The aggregation module performs: (2.1) for the narrow suggestion box of the current frame in the sequence image, moving in the up, down, left and right four directions within the local window to generate spatial RoIs, and the RoI represented by the narrow suggestion box is also included in the spatial RoIs; In the same way, the narrow suggestion box of the current frame is translated on other frames to generate time sequence RoIs; the spatial RoIs and the time sequence RoIs are combined as the spatio-temporal RoIs related to the narrow suggestion box on the current frame; (2.2) by RoI Align operation, the spatio-temporal RoIs are used to extract spatio-temporal RoI features of the same size on the detected semantic feature maps; (2.3) the spatio-temporal RoI features are vectorized and linearly projected to generate spatio-temporal tokens related to the narrow suggestion box on the current frame; (2.4) the obtained spatio-temporal tokens are positionally encoded and added to the spatio-temporal tokens to record the spatio-temporal position of the tokens; (2.5) the spatio-temporal tokens added with the positional encoding are input into a Transformer encoder to learn the long-range context association of the spatio-temporal tokens, wherein the Transformer encoder is composed of four consecutive Transformer layers; (2.6) the output results of the Transformer encoder are fused to inject the adjacent spatio-temporal RoI features represented by the spatio-temporal tokens into the RoI features represented by the narrow suggestion box to obtain the tokens after spatio-temporal feature aggregation; (2.7) the tokens after spatio-temporal feature aggregation are linearly projected and tensorized to obtain the enhanced RoI features representing the narrow suggestion box.

5. The coronary stenosis automatic detection system of claim 4, wherein: The extraction module performs: (1.1) for the input coronary angiography sequence, a backbone network composed of ResNet-50 and a feature pyramid is used to extract multi-scale features of each frame of image to generate semantic feature maps; (1.2) the generated feature maps are input into a region proposal network to generate narrow suggestion boxes of different aspect ratios and sizes to adapt to narrow lesions of different scales and proportions.

6. The coronary stenosis automatic detection system of claim 5, wherein: The prediction module performs: (3.1) global feature modeling and dimensionality reduction of the extracted enhanced RoI features are performed by a fully connected layer; (3.2) the dimensionally reduced features are input into a narrow position regression output branch and a narrow suggestion box classification output branch respectively to correct the detection results of the narrow suggestion boxes and predict the final coronary narrow detection results.

Images