A carotid plaque ultrasonic quantitative evaluation method and system based on DP-LSTM

Abstract

The application discloses a carotid plaque ultrasonic quantitative evaluation method and system based on a DP-LSTM, relates to the field of image processing, and comprises the following steps: acquiring a carotid artery dynamic ultrasonic image and a carotid artery high-resolution blood vessel magnetic resonance image corresponding to each time point; extracting multi-scale information of the carotid artery dynamic ultrasonic image, capturing corresponding time sequence features through a long short-term neural network after fusion between bottom layer time sequence features and high layer time sequence features; performing spatial feature extraction on a plaque target in the carotid artery high-resolution blood vessel magnetic resonance image after identification and cutting; performing cross-modal feature fusion on the time sequence features of the carotid artery dynamic ultrasonic image and the corresponding time sequence spatial features of the carotid artery high-resolution blood vessel magnetic resonance image in a multi-modal fusion manner; and performing quantitative evaluation on the carotid artery plaque according to the fused feature map. The application improves the evaluation precision of the carotid artery plaque and reduces the misjudgment risk caused by image quality and noise.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a method and system for quantitative assessment of carotid artery plaques using DP-LSTM ultrasound. Background Technology

[0002] Atherosclerosis is one of the main causes of morbidity and recurrence of LAA (large artery atherosclerosis) ischemic stroke. The carotid artery is a window reflecting systemic atherosclerosis; incorporating carotid plaque parameters into cardiovascular disease prediction models increases their predictive ability. Furthermore, studies have shown that plaque stability is an independent risk factor for cardiovascular and cerebrovascular events. Therefore, carotid plaque assessment is crucial for the prognosis and risk stratification of LAA stroke. Currently, carotid plaque assessment mainly relies on morphological characteristics, with pathological examination being the gold standard. However, pathological examination requires plaque removal and is not suitable for early plaque evaluation. Carotid ultrasound examination has advantages such as being non-invasive, simple, and allowing dynamic monitoring, and is widely used in carotid plaque assessment. However, current methods for effectively extracting information from ultrasound images have the following limitations:

[0003] 1. Image quality and noise: Dynamic ultrasound images can be affected by various noises and artifacts, which can interfere with the extraction of plaque features. Physicians may be unable to accurately identify subtle changes during analysis, thus affecting the accuracy of the assessment.

[0004] 2. The complexity of temporal changes: Atherosclerotic plaques may exhibit different morphological and structural changes at different time points. Traditional assessment methods may fail to effectively capture these changes, leading to insufficient understanding of the dynamic characteristics of plaques and consequently affecting the prediction of disease progression. Summary of the Invention

[0005] To address the shortcomings of existing carotid ultrasound images in plaque morphology recognition, this invention proposes a DP-LSTM-based method for quantitative assessment of carotid plaques via ultrasound, comprising the following steps:

[0006] S1: Acquire dynamic ultrasound images of the carotid artery and high-resolution magnetic resonance images of the carotid artery at each time point.

[0007] S2: Multi-scale information of carotid artery dynamic ultrasound images is extracted by hollow spatial pyramid pooling. After fusing the low-level and high-level temporal features, the corresponding temporal features are captured by a long short-term neural network.

[0008] S3: Spatial feature extraction of plaque targets in high-resolution vascular magnetic resonance images of the carotid artery is performed by a dual-path attention segmentation network;

[0009] S4: The temporal features of dynamic ultrasound images of the carotid artery are fused with the corresponding spatial features of high-resolution magnetic resonance imaging of the carotid artery across modalities using a multimodal fusion method.

[0010] S5: Quantitative assessment of carotid plaques based on the feature map obtained after cross-modal feature fusion.

[0011] Furthermore, in step S2, the temporal features of the carotid artery dynamic ultrasound image are extracted by a deep residual convolutional network and combined with information at each scale to obtain the temporal features from the bottom layer to the top layer.

[0012] Furthermore, in step S3, the dual-path attention segmentation network extracts information about the changes between different sizes and sub-regions based on the extraction of global context prior information, thereby realizing the extraction of spatial features of the patch target.

[0013] Furthermore, in step S3, an attention mechanism is added before the path attention segmentation network to guide the attention of patch target cutting.

[0014] Furthermore, in step S4, the multimodal fusion method is as follows: each layer of the current modality features is connected to each layer of another modality features in a feedforward manner. For each layer of features of the current modality, all features of the previous layers are used as its input, and its own feature map is used as the input of all subsequent layers of other modalities.

[0015] This invention also proposes a DP-LSTM-based ultrasound quantitative assessment system for carotid plaques, comprising:

[0016] The image source acquisition module is used to acquire dynamic ultrasound images of the carotid artery and high-resolution vascular magnetic resonance images of the carotid artery corresponding to each time point.

[0017] The temporal capture module is used to extract multi-scale information from dynamic ultrasound images of the carotid artery through hollow spatial pyramid pooling. After fusing the low-level and high-level temporal features, the corresponding temporal features are captured through a long short-term neural network.

[0018] The spatial extraction module is used to extract spatial features from plaque targets in high-resolution carotid artery vascular magnetic resonance images by using a dual-path attention segmentation network;

[0019] The feature fusion module is used to perform cross-modal feature fusion between the temporal features of dynamic ultrasound images of the carotid artery and the corresponding spatial features of high-resolution magnetic resonance imaging of the carotid artery through multimodal fusion.

[0020] The quantitative assessment module is used to quantitatively assess carotid plaques based on the feature map obtained after cross-modal feature fusion.

[0021] Furthermore, in the temporal capture module, the temporal features of the carotid artery dynamic ultrasound image are extracted through a deep residual convolutional network, and combined with information at various scales to obtain temporal features from the bottom layer to the top layer.

[0022] Furthermore, in the spatial extraction module, the dual-path attention segmentation network extracts information on the changes between different sizes and sub-regions based on the extraction of global context prior information, thereby realizing the extraction of spatial features of the patch target.

[0023] Furthermore, in the spatial extraction module, an attention mechanism is added before the path attention segmentation network to guide the attention of patch target cutting.

[0024] Furthermore, in the feature fusion module, the multimodal fusion method is as follows: each layer of the current modality features is connected to each layer of another modality features in a feedforward manner. For each layer of features in the current modality, all features of the previous layers are used as its input, and its own feature map is used as the input of all subsequent layers of other modalities.

[0025] Compared with the prior art, the present invention has at least the following beneficial effects:

[0026] (1) The DP-LSTM-based ultrasound quantitative assessment method and system for carotid plaques described in this invention improves the assessment accuracy of carotid plaques by capturing the spatial temporal features and motion transformation temporal features of plaques, helps doctors obtain more accurate carotid plaque identification data support, and reduces the risk of misjudgment caused by image quality and noise.

[0027] (2) By using long short-term neural networks to memorize the temporal information between temporal features, we can better understand the trend of plaque changes over time, thus providing a more reliable basis for disease prediction;

[0028] (3) By deeply fusing data from ultrasound and magnetic resonance images, this cross-modal feature fusion can not only improve the accuracy of diagnosis, but also help doctors better understand the performance of plaques under different imaging techniques, thereby making a more comprehensive clinical judgment. Attached Figure Description

[0029] Figure 1 This is a step-by-step diagram of a DP-LSTM-based ultrasound quantitative assessment method for carotid plaques.

[0030] Figure 2 This is a block diagram of a DP-LSTM-based carotid plaque ultrasound quantitative assessment system.

[0031] Figure 3 This is a schematic diagram of the DP-LSTM network structure;

[0032] Figure 4 A schematic diagram of Skip-cross fusion for comparative learning. Detailed Implementation

[0033] The following are specific embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments. Example 1

[0034] Carotid artery ultrasound plays a crucial role in clinical diagnosis and treatment decisions. Through dynamic image sequences, physicians can obtain valuable information about plaque changes, hemodynamics, and physiological responses. This information helps to more accurately assess the risk of carotid artery plaques and provides a scientific basis for developing appropriate treatment plans. However, dynamic ultrasound images can be affected by various noises and artifacts, which can interfere with the extraction of plaque features. Physicians may not be able to accurately identify subtle changes during analysis, thus affecting the accuracy of the assessment. Traditional assessment methods may not be able to effectively capture these changes, leading to insufficient understanding of the dynamic characteristics of plaques and consequently affecting the prediction of disease progression. Therefore, such as... Figure 1 As shown, this invention proposes a DP-LSTM-based ultrasound quantitative assessment method for carotid plaques, aiming to integrate temporal and spatial features to improve assessment accuracy. The method specifically includes the following steps:

[0035] S1: Acquire dynamic ultrasound images of the carotid artery and high-resolution magnetic resonance images of the carotid artery at each time point.

[0036] S2: Multi-scale information of carotid artery dynamic ultrasound images is extracted by hollow spatial pyramid pooling. After fusing the low-level and high-level temporal features, the corresponding temporal features are captured by a long short-term neural network.

[0037] S3: Spatial feature extraction of plaque targets in high-resolution vascular magnetic resonance images of the carotid artery is performed by a dual-path attention segmentation network;

[0038] S4: The temporal features of dynamic ultrasound images of the carotid artery are fused with the corresponding spatial features of high-resolution magnetic resonance imaging of the carotid artery across modalities using a multimodal fusion method.

[0039] S5: Quantitative assessment of carotid plaques based on the feature map obtained after cross-modal feature fusion.

[0040] During carotid ultrasound, a dynamic image is acquired, consisting of a series of carotid ultrasound images acquired at consecutive time points. This image possesses both temporal and cardiac motion characteristics. To more comprehensively capture the dynamic changes of plaques, this invention incorporates a portion of the DeepLabv3+ network structure into the Long Short-Term Memory (LSTM) neural network, forming the DP-LSTM network structure proposed in this invention. Its specific network structure is as follows: Figure 3 As shown, firstly, the temporal features of the input carotid artery dynamic ultrasound image are extracted using a ResNet18 deep residual convolutional network in the encoder. Then, multi-scale information is acquired using Spatial Pyramid Pooling (ASPP) introduced by DeepLabv3+, and combined with the extracted temporal features to obtain multi-scale spatial temporal features from the bottom to the top. Finally, the bottom-level and high-level temporal features are further fused in the decoder, and the temporal information between the temporal features is memorized using an LSTM. In this way, DP-LSTM can capture the temporal features of carotid artery plaques at various scales and during motion transformations in dynamic ultrasound images.

[0041] Here, we obtain temporal features by performing convolution operations (ResNet18 deep residual convolutional network) and pooling operations (Aperture Spatial Pyramid Pooling with Dirt) on single-frame images. This transforms the image's temporal features into condensed data by reducing width and height and increasing depth. LSTM then fully utilizes the time series to analyze the input. Unlike traditional neural networks that disregard the input at time t and time t+1, LSTM rationally uses the input at time t+n to process the feature information at time t, capturing the temporal dimension of ultrasound, and extracting the last frame from the video as the final effective temporal feature output by the DP-LSTM model.

[0042] To compensate for the limitations of ultrasound images in recognizing dynamic features such as changes in morphology and structure at different time points by supplementing spatial features, this invention introduces high-resolution carotid artery vascular magnetic resonance images for spatial feature extraction. However, carotid plaques are relatively small targets in high-resolution carotid artery vascular magnetic resonance images, and the identification and segmentation of carotid plaques in images with various complex structures (trachea, spine, neck muscles, neck glands, etc.) presents certain challenges. Therefore, this invention introduces the dual-path attention segmentation network DPAM-PSPNet for image segmentation.

[0043] PSPNet is a network architecture for semantic segmentation that introduces a Pyramid Pooling Module to capture contextual information at different scales. The Pyramid Pooling Module acquires contextual information at different levels by performing multi-scale pooling operations globally, and then fuses this information to enhance the recognition ability of objects at different scales. DPAM is a method for improving feature extraction, which enhances feature aggregation through two paths (dense connections and residual connections). This method aims to improve feature propagation and fusion, thereby improving the model's performance. DPAM helps the model better learn multi-level features in images, and due to the presence of dense connections, it promotes feature reuse, making the model more efficient.

[0044] DPAM-PSPNet combines the advantages of PSPNet and DPAM, utilizing DPAM to enhance the feature aggregation capabilities of PSPNet. In this invention, the characteristics of DPAM-PSPNet are used to effectively extract global contextual prior information, thereby obtaining information with different sizes and variations across different sub-regions, thus improving the performance of small target object and content recognition in complex medical image scenes.

[0045] Finally, considering the high sensitivity of both ultrasound and magnetic resonance imaging (HR-VWI) data for carotid plaque assessment, this invention achieves deep fusion of the two modalities through contrastive learning and a Skip-cross approach. Specifically, as... Figure 4 As shown, in a feedforward manner, each layer of the current modality features is connected to each layer of another modality. For each layer of the current modality, all features from previous layers are used as its input, and its own feature map is used as the input to all subsequent layers of other modalities. Specifically applied in this application, Skip-cross connects the temporal feature map of the carotid ultrasound image to the corresponding position of the spatial feature map of the high-resolution MRI of the carotid artery, achieving dense cross-modal connections across all feature maps. In this process, the Skip-cross module adaptively identifies the optimal combination period for the two image features. Then, Skip-cross fusion networks (SkipcrossNets) fully utilize the multimodal information from the encoder stage, performing feature fusion in Skip-cross so that the feature maps of each modality cross-influence each other during resolution restoration, ensuring that the fusion of spatial and temporal features does not only occur in the last layer of the network, truly achieving deep fusion of cross-modal features. Finally, the carotid plaque can be quantitatively assessed based on the feature map output by deep fusion. Example 2

[0046] To better understand this invention, this embodiment describes it through a system structure, such as... Figure 2 As shown, a DP-LSTM-based ultrasound quantitative assessment system for carotid plaques includes:

[0047] The image source acquisition module is used to acquire dynamic ultrasound images of the carotid artery and high-resolution vascular magnetic resonance images of the carotid artery corresponding to each time point.

[0048] The temporal capture module is used to extract multi-scale information from dynamic ultrasound images of the carotid artery through hollow spatial pyramid pooling. After fusing the low-level and high-level temporal features, the corresponding temporal features are captured through a long short-term neural network.

[0049] The spatial extraction module is used to extract spatial features from plaque targets in high-resolution carotid artery vascular magnetic resonance images by using a dual-path attention segmentation network;

[0050] The feature fusion module is used to perform cross-modal feature fusion between the temporal features of dynamic ultrasound images of the carotid artery and the corresponding spatial features of high-resolution magnetic resonance imaging of the carotid artery through multimodal fusion.

[0051] The quantitative assessment module is used to quantitatively assess carotid plaques based on the feature map obtained after cross-modal feature fusion.

[0052] Furthermore, in the temporal capture module, the temporal features of the carotid artery dynamic ultrasound image are extracted through a deep residual convolutional network, and combined with information at various scales to obtain the temporal features from the bottom layer to the top layer.

[0053] Furthermore, in the spatial extraction module, the dual-path attention segmentation network extracts information based on global context prior information to obtain information on the changes between different sizes and sub-regions, thereby realizing the extraction of spatial features of the patch target.

[0054] Furthermore, in the spatial extraction module, an attention mechanism is added before the path attention segmentation network to guide the attention of patch target cutting.

[0055] Furthermore, in the feature fusion module, the multimodal fusion method is as follows: each layer of the current modality features is connected to each layer of another modality features in a feedforward manner. For each layer of features in the current modality, all features of the previous layers are used as its input, and its own feature map is used as the input of all subsequent layers of other modalities.

[0056] In summary, the DP-LSTM-based ultrasound quantitative assessment method and system for carotid artery plaques described in this invention improves the assessment accuracy of carotid artery plaques by capturing their spatial and temporal features and motion transformation temporal features. This helps physicians obtain more accurate data support for carotid artery plaque identification and reduces the risk of misjudgment caused by image quality and noise. By using a long short-term neural network to remember the temporal information between temporal features, the changing trend of plaques over time can be better understood, thus providing a more reliable basis for disease prediction.

[0057] By deeply fusing data from ultrasound and magnetic resonance imaging, this cross-modal feature fusion can not only improve diagnostic accuracy but also help doctors better understand the plaque’s behavior under different imaging techniques, thereby making a more comprehensive clinical judgment.

[0058] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0059] Furthermore, in this invention, descriptions involving terms such as "first," "second," and "a" are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0060] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0061] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

Claims

1. A method for quantitative assessment of carotid artery plaques using ultrasound based on DP-LSTM, characterized in that, Including the following steps: S1: Acquire dynamic ultrasound images of the carotid artery and high-resolution magnetic resonance images of the carotid artery at each time point. S2: Multi-scale information of carotid artery dynamic ultrasound images is extracted by hollow spatial pyramid pooling. After fusing the low-level and high-level temporal features, the corresponding temporal features are captured by a long short-term neural network. S3: Spatial feature extraction of plaque targets in high-resolution vascular magnetic resonance images of the carotid artery is performed by a dual-path attention segmentation network; S4: The temporal features of dynamic ultrasound images of the carotid artery are fused with the corresponding spatial features of high-resolution magnetic resonance imaging of the carotid artery across modalities using a multimodal fusion method. S5: Quantitative assessment of carotid plaques based on the feature map obtained after cross-modal feature fusion; In step S4, the multimodal fusion method is as follows: each layer of the current modality features is connected to each layer of another modality features in a feedforward manner. For each layer of features of the current modality, all features of the previous layers are used as its input, and its own feature map is used as the input of all subsequent layers of other modalities.

2. The method for quantitative assessment of carotid plaques based on DP-LSTM as described in claim 1, characterized in that, In step S2, the temporal features of the carotid artery dynamic ultrasound image are extracted by a deep residual convolutional network and combined with information at each scale to obtain the temporal features from the bottom layer to the top layer.

3. The method for quantitative assessment of carotid artery plaques based on DP-LSTM as described in claim 1, characterized in that, In step S3, the dual-path attention segmentation network extracts information about the changes between different sizes and sub-regions based on the extraction of global context prior information, thereby extracting the spatial features of the patch target.

4. The method for quantitative assessment of carotid plaques based on DP-LSTM as described in claim 3, characterized in that, In step S3, an attention mechanism is added before the path attention segmentation network to guide the attention of patch target cutting.

5. A DP-LSTM-based ultrasound quantitative assessment system for carotid artery plaques, characterized in that, include: The image source acquisition module is used to acquire dynamic ultrasound images of the carotid artery and high-resolution vascular magnetic resonance images of the carotid artery corresponding to each time point. The temporal capture module is used to extract multi-scale information from dynamic ultrasound images of the carotid artery through hollow spatial pyramid pooling. After fusing the low-level and high-level temporal features, the corresponding temporal features are captured through a long short-term neural network. The spatial extraction module is used to extract spatial features from plaque targets in high-resolution carotid artery vascular magnetic resonance images by using a dual-path attention segmentation network; The feature fusion module is used to perform cross-modal feature fusion between the temporal features of dynamic ultrasound images of the carotid artery and the corresponding spatial features of high-resolution magnetic resonance imaging of the carotid artery through multimodal fusion. The quantitative assessment module is used to perform quantitative assessment of carotid plaques based on the feature map obtained after cross-modal feature fusion. In the feature fusion module, the multimodal fusion method is as follows: each layer of the current modality features is connected to each layer of another modality features in a feedforward manner. For each layer of features in the current modality, all features of the previous layers are used as its input, and its own feature map is used as the input of all subsequent layers of other modalities.

6. The DP-LSTM-based carotid artery plaque ultrasound quantitative assessment system as described in claim 5, characterized in that, In the temporal capture module, the temporal features of the carotid artery dynamic ultrasound image are extracted through a deep residual convolutional network, and combined with information at various scales to obtain the temporal features from the bottom layer to the top layer.

7. The DP-LSTM-based carotid artery plaque ultrasound quantitative assessment system as described in claim 5, characterized in that, In the spatial extraction module, the dual-path attention segmentation network extracts information based on global context prior information to obtain information on the changes between different sizes and sub-regions, thereby realizing the extraction of spatial features of the patch target.

8. The carotid artery plaque ultrasound quantitative assessment system based on DP-LSTM as described in claim 7, characterized in that, In the spatial extraction module, an attention mechanism is added before the path attention segmentation network to guide the attention of patch target cutting.

Images