A multi-scale interactive plaque and vessel segmentation method

Abstract

The present application belongs to the technical field of deep learning, computer vision, medical image segmentation, and discloses a multi-scale interactive plaque and blood vessel segmentation method. The present application first constructs a multi-level interactive integration module, fuses the features extracted in the encoder, and efficiently integrates the features from different levels and different channels through a channel selection module. Then a feature polishing module is constructed, the integrated features are finely adjusted and sent to the upsampling process. Finally, through the corresponding scale decoding module, the resolution is restored layer by layer by using the upsampling method, and the full-resolution prediction result is output from the top convolution block, thereby realizing accurate segmentation of the region of interest of different scales.

Description

Technical Field

[0001] This invention relates to the fields of deep learning, computer vision, and medical image segmentation, and particularly to a multi-scale interactive method for segmenting plaques and blood vessels. Background Technology

[0002] The heart, a vital organ, is responsible for pumping blood, ensuring that all tissues receive sufficient oxygen and nutrients and removing carbon dioxide and other waste products. The coronary arteries, located around the heart, supply this vital blood; therefore, their health determines the heart's pumping capacity. Coronary artery disease, also known as coronary angina, is the most common cardiovascular disease, primarily caused by plaque buildup in the coronary arteries, leading to narrowing of the vessels and impaired blood flow. Atherosclerotic plaques are the pathological basis of coronary angina, with approximately half of vulnerable plaque rupture occurring in lesions where the coronary artery diameter is less than 50% narrowed. Therefore, accurately determining the plaque type and degree of stenosis is crucial. Currently, non-invasive computed tomography (CTA) angiography is mainly used to indicate the degree of coronary artery stenosis; however, coronary CTA has relatively low spatial resolution and is susceptible to artifacts.

[0003] Existing coronary artery plaque segmentation methods can be categorized into manual, semi-automatic, and fully automatic segmentation based on the degree of human intervention. These methods can be broadly classified into five types: threshold-based, edge-based, region-based, graph-theory-based, and energy functional-based. However, all of these methods fall short in the field of interactive fine-grained segmentation. This invention, based on the "encoder-decoder" structure, introduces a multi-scale interactive fine-grained segmentation algorithm, effectively addressing this deficiency and thereby improving detection efficiency and accuracy. Summary of the Invention

[0004] The purpose of this invention is to overcome the problems of noise and redundant information introduced by current feature processing design methods, and to provide a segmentation scheme that can process multi-level features to achieve high-precision image segmentation.

[0005] The technical solution of the present invention is as follows: a multi-scale interactive plaque and blood vessel segmentation method, wherein the encoder extracts the coded features of the corresponding size of the input image from 6 different layers as output; the coded features of different layers are input into the decoder to obtain 6 decoded images P0 to P5, and loss calculation is performed with the obtained real images G0 to G5; the encoder and decoder are updated according to the result of the loss calculation.

[0006] The decoder includes a multi-level interactive integration module, a feature polishing module, and a feature alignment upsampling unit;

[0007] In the multi-level interactive integration module, the encoded features with the same resolution as the current level are used as the main features f.main Other hierarchical encoding features serve as auxiliary features. The system captures valuable cues from auxiliary features that are relevant to the main feature based on similarity; it then performs an adaptive combination operation on the main and auxiliary features; the resulting feature f is obtained from this adaptive combination operation. aux With main feature f main To conduct a comprehensive integration;

[0008] The integrated features are input into the feature polishing module for processing. After fine-tuning, the sample is fed into the feature alignment upsampling unit, as shown in the following formula;

[0009]

[0010] This is the output of the feature polishing module;

[0011] Following the top-down path, the resolution is restored layer by layer through the feature alignment upsampling unit, and the full-resolution prediction result is output from the top convolutional block. In the feature alignment upsampling unit, a deformable convolution is introduced to align the distribution of features in adjacent layers. First, traditional bilinear interpolation is used to upsample low-resolution features to the same resolution. Then, the outputs of the feature polishing module and the next-level feature alignment upsampling module are concatenated and fed into two independent convolutional branches to predict the two adaptive parameters of the deformable convolution inputs, namely the offset and spatial modulation, which control the degree of spatial offset of low-resolution features during the upsampling process. The calculation process is expressed as follows:

[0012]

[0013] Δp k and Δm k w represents the offset and modulation amount at position k during the local operation at the corresponding output position p, respectively. k Corresponding input position p+p k +Δp k ;x represents the feature after the output of the feature polishing module and the feature alignment upsampling module of the next level are concatenated.

[0014] The adaptive combination operation includes spatial attention operation, average pooling operation, and channel selection;

[0015] The input encoded features are subjected to spatial attention and average pooling operations, and the number of elements in the resulting feature map is set to K.

[0016] In the channel selection, the channels of the four auxiliary features are expanded to twice their original size. Then, half of each feature is taken and combined to generate channel weights for the remaining four auxiliary features. The channel weights are multiplied by the corresponding features along the channel and then added together.

[0017] The output f of the channel selection module aux With main feature f main To conduct a comprehensive integration,

[0018]

[0019]

[0020] Here W main , and Represents a linear transformation matrix. It is a feature after integration.

[0021] The encoder is composed of a feature extraction network, which uses the classic Res2Net network as its backbone.

[0022] The beneficial effects of this invention are as follows: This invention proposes a multi-scale interactive plaque and vessel segmentation method. First, a multi-level interactive integration module is constructed to fuse features extracted from the encoder, and a channel selection module efficiently integrates features from different levels and channels. Next, a feature polishing module is constructed to finely adjust the integrated features before feeding them into the upsampling process. Finally, through a corresponding scale decoding module, the resolution is restored layer by layer using upsampling, and the full-resolution prediction result is output from the top convolutional block, thereby achieving accurate segmentation of regions of interest at different scales. This method effectively utilizes features from other levels, strengthens foreground cues, and suppresses background interference, providing a more accurate and efficient new method for plaque and vessel segmentation. Attached Figure Description

[0023] Figure 1 The overall framework for multi-scale interactive plaque and vessel segmentation methods;

[0024] Figure 2 It is a multi-level interactive integration module;

[0025] Figure 3 Select the module for the channel;

[0026] Figure 4 It is a feature polishing module;

[0027] Figure 5 It is a feature alignment upsampling module;

[0028] Figures 6(a)-6(d) Original image of a cerebral artery;

[0029] Figures 7(a)-7(d) This is the result of cerebral artery segmentation. Detailed Implementation

[0030] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0031] A multi-scale interactive plaque and vessel segmentation method, such as Figure 1 As shown, the leftmost column represents the feature extraction encoder, which uses the feature extraction structure of an existing classification model. The entire module in the middle represents the decoder. To supplement the detailed information in the decoding process using shallow features, multiple encoded features with different resolutions are extracted from the encoder and fed into the decoder. These features are then fused in a multi-level interactive integration module, further adjusted in a feature polishing module, and then sent to the corresponding scale decoder. Following the top-down path, the resolution is restored layer by layer through a feature alignment and upsampling module, and the full-resolution prediction result is output from the top convolutional block. The rightmost module is the supervision module, which calculates the loss between p0 to p5 and G0 to G5 to update the model and make the model output more accurate.

[0032] The encoder's feature extraction network uses the classic Res2Net network as its backbone, extracting features of the corresponding size of the input image from six different layers as output.

[0033] In the multi-level interactive integration module, the encoded features with the same resolution as the current level are designated as "primary features," while features from other levels are designated as "auxiliary features." Clues from the auxiliary features that are more valuable to the primary features are captured based on similarity. For ease of description, the primary feature is denoted as f. main The four auxiliary features are represented as Since auxiliary features have different resolutions and channel numbers, directly integrating them with the main features would result in excessive computation and parameter count. Therefore, an adaptive combination operation was designed. Specifically, the four auxiliary features are first compressed to ensure a consistent channel count. Then, spatial modulation is performed using a spatial attention mechanism to predict an independent spatial modulation map for each input. In the formula, v represents the Sigmoid operation, and Φ(*,θ) represents a convolutional layer with parameter θ. These modulated weight maps are multiplied onto the original feature map. Then, spatial adaptive average pooling is used to achieve uniform spatial resolution. This operation pools all the input feature maps, and the number of elements in the resulting feature map is set to K.

[0034] To efficiently integrate features from different levels and channels, a channel selection module is introduced, such as... Figure 3 As shown, the processing method is to first expand the channels of the four features to twice their original size, then take half of each feature and concatenate them to generate channel weights for the remaining four features. These weights are multiplied by the corresponding features along the channels and then added; the output f of the channel selection module... auxWith main feature f main To conduct a comprehensive integration,

[0035]

[0036]

[0037] Here W main , and Represents a linear transformation matrix. It is a feature after integration.

[0038] Step (3) Constructing the feature polishing module

[0039] The feature polishing module performs fine adjustments on the integrated features before sending them into the upsampling process. The specific formula is as follows.

[0040]

[0041] Output for the module.

[0042] During the upsampling process, deformable convolution is introduced to align the distribution of features from adjacent layers. First, traditional bilinear interpolation is used to upsample low-resolution features to the same resolution. Then, the two input features are concatenated and fed into two independent convolutional branches to predict the adaptive parameters of the two inputs of the deformable convolution: the offset and the spatial modulation. This allows control over the degree of spatial offset of the low-resolution features during the upsampling process. The calculation process can be represented as:

[0043]

[0044] Here Δp k and Δm k These represent the offset and modulation amount at position k during the local computation at the corresponding output position p, respectively. Δm k It is a value between 0 and 1, while Δp k Then it is an unconstrained real value. Because w k The corresponding input position p+p k +Δp k It is a floating-point number, and here a bilinear interpolation strategy is used to approximate the result at that location using its four nearest integer coordinates. This targeted design can reduce the distribution differences when fusing features at different scales and promote the efficient top-down transmission of contextual information.

[0045] Finally, following the top-down path, the resolution is restored layer by layer through feature-aligned upsampling units, and the full-resolution prediction result is output from the top convolutional block.

[0046] Taking 3D cerebral artery blood vessels as an example, the original segmentation image and the result are given. The original image is shown in Figure 6, where Figures 6(a), 6(b), and 6(d) are images in the x, y, and z directions, respectively, and Figure 6(c) is the label. The output result of the original image after being processed by the encoder and decoder of this network is shown in Figure 7, where Figures 7(a), 7(b), and 7(d) are images in the x, y, and z directions, respectively, and Figure 7(c) is the label. It can be seen that the 3D cerebral artery blood vessel labels processed by the network of this invention are clearer and more accurate, and the blood vessel status in each direction is more obvious.

Claims

1. A multi-scale interactive plaque and vessel segmentation method, characterized in that, The encoder extracts the coded features of the corresponding size of the input image from 6 different layers as output; the coded features of different layers are input into the decoder to obtain 6 decoded images P0 to P5, and loss is calculated with the obtained real images G0 to G5. The encoder and decoder are updated according to the result of the loss calculation. The decoder includes a multi-level interactive integration module, a feature polishing module, and a feature alignment upsampling unit; In the multi-level interactive integration module, the encoded features with the same resolution as the current level are used as the main features f. main Other hierarchical encoding features serve as auxiliary features. The system captures valuable cues from auxiliary features that are relevant to the main feature based on similarity; it then performs an adaptive combination operation on the main and auxiliary features; the resulting feature f is obtained from this adaptive combination operation. aux With main feature f main To conduct a comprehensive integration; The integrated features are input into the feature polishing module for processing. After fine-tuning, the sample is fed into the feature alignment upsampling unit, as shown in the following formula; This is the output of the feature polishing module; Following the top-down path, the resolution is restored layer by layer through the feature alignment upsampling unit, and the full-resolution prediction result is output from the top convolutional block. In the feature alignment upsampling unit, a deformable convolution is introduced to align the distribution of features in adjacent layers. First, traditional bilinear interpolation is used to upsample low-resolution features to the same resolution. Then, the outputs of the feature polishing module and the next-level feature alignment upsampling module are concatenated and fed into two independent convolutional branches to predict the two adaptive parameters of the deformable convolution inputs, namely the offset and spatial modulation, which control the degree of spatial offset of low-resolution features during the upsampling process. The calculation process is expressed as follows: Δp k and Δm k w represents the offset and modulation amount at position k during the local operation at the corresponding output position p, respectively. k Corresponding input position p+p k +Δp k ;x represents the feature after the output of the feature polishing module and the feature alignment upsampling module of the next level are concatenated.

2. The multi-scale interactive plaque and vessel segmentation method according to claim 1, characterized in that, The adaptive combination operation includes spatial attention operation, average pooling operation, and channel selection; The input encoded features are subjected to spatial attention and average pooling operations, and the number of elements in the resulting feature map is set to K. In the channel selection, the channels of the four auxiliary features are expanded to twice their original size. Then, half of each feature is taken and combined to generate channel weights for the remaining four auxiliary features. The channel weights are multiplied by the corresponding features along the channel and then added together. The output f of the channel selection module aux With main feature f main To conduct a comprehensive integration, Here W main , and Represents a linear transformation matrix. It is a feature after integration.

3. The multi-scale interactive plaque and vessel segmentation method according to claim 1 or 2, characterized in that, The encoder is composed of a feature extraction network, which uses the classic Res2Net network as its backbone.

Images