Carotid plaque ultrasonic quantitative evaluation method and system based on DP-LSTM

The spatial timing characteristics and motion transformation timing characteristics of plaques were captured in carotid ultrasound images through DP-LSTM, combined with the dual-path attention segmentation network and multimodal fusion technology, and the carotid plaque evaluation in the prior art was solved. Accurate problems improve evaluation accuracy and diagnostic accuracy.

CN118968575AActive Publication Date: 2024-11-15NEW SMART MEDICAL TECH (NINGBO) CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411455757.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-18
Publication Date
2024-11-15
Estimated Expiration
2044-10-18

AI Technical Summary

Technical Problem

The prior art is difficult to accurately identify subtle changes and capture dynamic features of plaques in carotid ultrasound images, resulting in inaccurate evaluation and affecting disease progression prediction.

Method used

Using a DP-LSTM-based method, multi-scale information is extracted through hollow space pyramid pooling, combined with long and short-term neural networks to capture timing features, and cross-modal feature fusion is carried out through dual-path attention segmentation network and multimodal fusion technology. Improve the accuracy of quantitative evaluation of plaques.

Benefits of technology

Improve the evaluation accuracy of carotid artery plaques, reduce the risk of misjudgment caused by image quality and noise, better understand the changing trends of plaques over time, provide a more reliable basis for disease prediction, and improve diagnosis accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968575A_ABST
    Figure CN118968575A_ABST
Patent Text Reader

Abstract

The invention discloses a DP-LSTM-based carotid plaque ultrasonic quantitative evaluation method and system, and relates to the technical field of image processing, and the method comprises the steps: obtaining a carotid artery dynamic ultrasonic image and a carotid artery high-resolution blood vessel magnetic resonance image corresponding to each time point; multi-scale information of the carotid artery dynamic ultrasound image is extracted, and after fusion between bottom-layer time sequence features and high-layer time sequence features is carried out, capturing of corresponding time sequence features is carried out through a long-short-term neural network; spatial feature extraction under identification and cutting is carried out on a plaque target in the carotid artery high-resolution blood vessel magnetic resonance image; performing cross-modal feature fusion on the time sequence features of the carotid artery dynamic ultrasound image and the spatial features of the corresponding time sequence of the carotid artery high-resolution blood vessel magnetic resonance image through a multi-modal fusion mode; and performing quantitative evaluation on the carotid plaque according to the fused feature map. The carotid plaque evaluation precision is improved, and the misjudgment risk caused by image quality and noise is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a DP-LSTM-based carotid artery plaque ultrasonic quantitative assessment method and system. Background Art

[0002] Atherosclerosis is one of the main causes of LAA-type (large artery atherosclerosis) ischemic stroke and its recurrence. The carotid artery is a window that reflects systemic atherosclerosis. Incorporating the corresponding parameters of carotid plaques into the cardiovascular disease prediction model will increase the predictive ability of the model. In addition, studies have shown that plaque stability is an independent risk factor for cardiovascular and cerebrovascular events. Therefore, the evaluation of carotid plaques is of great significance for the prognosis and risk stratification of LAA-type stroke. At present, the evaluation of carotid plaques is mainly based on morphological characteristics, and the gold standard is pathological examination. Pathological examination requires the removal of plaque specimens and is not suitable for early plaque evaluation. Carotid ultrasound examination has the advantages of being non-invasive, simple, and dynamically monitored, and is widely used in the evaluation of carotid plaques. However, there are currently the following deficiencies in the effective extraction of information from ultrasound images: 1. Image quality and noise: Dynamic ultrasound images may be affected by various noises and artifacts, which interfere with the extraction of plaque features. Doctors may not be able to accurately identify subtle changes during analysis, thus affecting the accuracy of the assessment; 2. Complexity of temporal changes: Atherosclerotic plaques may present different morphological and structural changes at different time points. Traditional evaluation methods may not be able to effectively capture these changes, resulting in insufficient understanding of the dynamic characteristics of plaques, which in turn affects the prediction of disease progression. Summary of the invention

[0003] In order to solve the shortcomings of existing carotid artery ultrasound images in identifying plaque morphology, the present invention proposes a DP-LSTM-based carotid artery plaque ultrasound quantitative assessment method, comprising the steps of: S1: Acquire dynamic ultrasound images of the carotid artery and high-resolution vascular magnetic resonance images of the carotid artery corresponding to each time point; S2: Extract multi-scale information of carotid artery dynamic ultrasound images through void space pyramid pooling, and after fusing the underlying temporal features with the high-level temporal features, capture the corresponding temporal features through long-term and short-term neural networks; S3: Spatial feature extraction under recognition and cutting of plaque targets in high-resolution vascular magnetic resonance images of carotid arteries using a dual-path attention segmentation network; S4: The temporal features of the carotid artery dynamic ultrasound image and the spatial features of the corresponding temporal features of the carotid artery high-resolution vascular magnetic resonance image are fused across modalities by means of multimodal fusion; S5: Quantitative assessment of carotid artery plaques based on feature maps after cross-modal feature fusion.

[0004] Furthermore, in the step S2, the time series features of the carotid artery dynamic ultrasound image are extracted by a deep residual convolutional network, and are combined with each scale information to obtain the time series features from the bottom layer to the high layer.

[0005] Furthermore, in the S3 step, the dual-path attention segmentation network obtains information on changes between each size and each sub-region based on the extraction of global context prior information, thereby realizing the extraction of spatial features of the patch target.

[0006] Furthermore, in the S3 step, an attention mechanism is added before the path attention segmentation network to guide the attention of patch target cutting.

[0007] Furthermore, in the S4 step, the multimodal fusion method is specifically as follows: each layer of the current modality feature is connected to each layer of the other modality feature in a feedforward manner, and for each layer of the current modality feature, all previous layers of features are used as its input, and its own feature map is used as the input of all subsequent layers of other modalities.

[0008] The present invention also proposes a DP-LSTM-based carotid artery plaque ultrasound quantitative assessment system, comprising: An image source acquisition module, used to acquire carotid artery dynamic ultrasound images and carotid artery high-resolution vascular magnetic resonance images corresponding to each time point; The time series capture module is used to extract multi-scale information of the carotid artery dynamic ultrasound image through void space pyramid pooling. After fusing the bottom-level time series features with the high-level time series features, the corresponding time series features are captured through the long-term and short-term neural network. A spatial extraction module is used to extract spatial features of plaque targets in carotid artery high-resolution vascular magnetic resonance images by identifying and cutting them through a dual-path attention segmentation network; A feature fusion module is used to perform cross-modal feature fusion of the temporal features of the carotid artery dynamic ultrasound image and the spatial features of the corresponding temporal features of the carotid artery high-resolution vascular magnetic resonance image by means of multimodal fusion; The quantitative evaluation module is used to perform quantitative evaluation of carotid artery plaques based on feature maps after cross-modal feature fusion.

[0009] Furthermore, in the timing capture module, the timing features of the carotid artery dynamic ultrasound image are extracted through a deep residual convolutional network, and combined with each scale information to obtain the timing features from the bottom layer to the high layer.

[0010] Furthermore, in the spatial extraction module, the dual-path attention segmentation network obtains information on changes between each size and each sub-region based on the extraction of global context prior information, thereby realizing the extraction of spatial features of the patch target.

[0011] Furthermore, in the spatial extraction module, an attention mechanism is added before the path attention segmentation network to guide the attention of patch target cutting.

[0012] Furthermore, in the feature fusion module, the multimodal fusion method is specifically as follows: each layer of the current modality feature is connected to each layer of the other modality feature in a feedforward manner, and for each layer of the current modality feature, all previous layers of features are used as its input, and its own feature map is used as the input of all subsequent layers of other modalities.

[0013] Compared with the prior art, the present invention has at least the following beneficial effects: (1) The DP-LSTM-based carotid plaque ultrasound quantitative assessment method and system described in the present invention improves the assessment accuracy of carotid plaque by capturing the spatial temporal characteristics and motion transformation temporal characteristics of the plaque, helps doctors obtain more accurate carotid plaque identification data support, and reduces the risk of misjudgment caused by image quality and noise; (2) By memorizing the temporal information between temporal features using long-term and short-term neural networks, we can better understand the changing trends of plaques over time, thus providing a more reliable basis for disease prediction; (3) Through the deep fusion of ultrasound image and magnetic resonance image data, this cross-modal feature fusion can not only improve the accuracy of diagnosis, but also help doctors better understand the performance of plaques under different imaging technologies, thereby making more comprehensive clinical judgments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A step diagram of a DP-LSTM-based method for quantifying and evaluating carotid artery plaques using ultrasound; Figure 2 This is a module diagram of a DP-LSTM-based carotid artery plaque ultrasound quantitative assessment system; Figure 3 It is a schematic diagram of the DP-LSTM network structure; Figure 4 Schematic diagram of contrastive learning fusion Skip-cross. DETAILED DESCRIPTION

[0015] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments. Embodiment 1

[0016] Carotid ultrasound examination plays an important role in clinical diagnosis and treatment decisions. Through dynamic image sequences, doctors can obtain valuable information about plaque changes, hemodynamics, and physiological responses. This information helps to more accurately assess the risk of carotid plaques and provide a scientific basis for formulating reasonable treatment plans. However, dynamic ultrasound images may be affected by various noises and artifacts, which interfere with the extraction of plaque features. Doctors may not be able to accurately identify subtle changes during analysis, which affects the accuracy of the assessment. Traditional evaluation methods may not be able to effectively capture these changes, resulting in insufficient understanding of the dynamic characteristics of plaques, which in turn affects the prediction of disease progression. For this reason, Figure 1 As shown, the present invention proposes a DP-LSTM-based carotid artery plaque ultrasound quantitative assessment method, which aims to integrate temporal features with spatial features to improve the accuracy of assessment, and specifically includes the following steps: S1: Acquire dynamic ultrasound images of the carotid artery and high-resolution vascular magnetic resonance images of the carotid artery corresponding to each time point; S2: Extract multi-scale information of carotid artery dynamic ultrasound images through void space pyramid pooling, and after fusing the underlying temporal features with the high-level temporal features, capture the corresponding temporal features through long-term and short-term neural networks; S3: Spatial feature extraction under recognition and cutting of plaque targets in high-resolution vascular magnetic resonance images of carotid arteries using a dual-path attention segmentation network; S4: The temporal features of the carotid artery dynamic ultrasound image and the spatial features of the corresponding temporal features of the carotid artery high-resolution vascular magnetic resonance image are fused across modalities by means of multimodal fusion; S5: Quantitative assessment of carotid artery plaques based on feature maps after cross-modal feature fusion.

[0017] During the carotid ultrasound process, a dynamic image consisting of a series of carotid ultrasound images collected at consecutive time points is obtained, which has time characteristics and cardiac motion characteristics. In order to more comprehensively capture the dynamic changes of plaques, the present invention adds part of the network structure of DeepLabv3+ to the network based on the long short-term memory neural network LSTM, forming the DP-LSTM network structure proposed by the present invention. Its specific network structure is as follows Figure 3As shown in the figure, firstly, the deep residual convolution network ResNet18 in the encoder is used to extract the time series features of the input carotid artery dynamic ultrasound image, and then the multi-scale information is acquired through the hollow space pyramid pooling ASPP introduced by DeepLabv3+, and combined with the extracted time series features, the time series features of the bottom to high-level multi-scale space are obtained. Finally, in the decoder, the bottom-level time series features are further fused with the high-level time series features, and the time information between the time series features is memorized by LSTM. In this way, DP-LSTM can capture the time series features of the carotid plaque in each scale space and the time series features during motion transformation in the dynamic ultrasound image.

[0018] Here, we obtain the time series features by performing convolution operations (deep residual convolution network ResNet18) and pooling operations (atrous spatial pyramid pooling ASPP) on a single frame image, and transform the graphic time series features of the image into condensed data by reducing the length and width and increasing the depth, while LSTM makes full use of the time series to analyze the input. Compared with traditional neural networks, which consider the input content at time t and the input content at time t+1 to be irrelevant, LSTM will reasonably use the input at time t+n to process the feature information at time t to capture the time dimension information of ultrasound, and take out the last frame in the video frame as the final effective time series feature output by the DP-LSTM model.

[0019] In order to make up for the lack of dynamic feature recognition of ultrasound images with different morphological and structural changes at different time points by supplementing spatial features, the present invention introduces carotid high-resolution vascular magnetic resonance images to extract spatial features. However, for carotid high-resolution vascular magnetic resonance images, carotid plaques are relatively small targets, and the identification and segmentation of carotid plaques are somewhat difficult in images with a variety of complex structures (trachea, spine, neck muscles, neck glands, etc.). Therefore, the present invention introduces the dual-path attention segmentation network DPAM-PSPNet to perform image segmentation tasks.

[0020] Among them, PSPNet is a network architecture for semantic segmentation, which introduces the Pyramid Pooling Module to capture contextual information of different scales. The Pyramid Pooling Module obtains contextual information of different levels by performing multi-scale pooling operations on a global scale, and then fuses this information to enhance the recognition of objects of different scales. DPAM is a method for improving feature extraction, which enhances feature aggregation through two paths (dense connections and residual connections). This method aims to improve the transfer and fusion of features, thereby improving the expressiveness of the model. DPAM can help the model better learn multi-level features in the image, and due to the existence of dense connections, it can promote feature reuse, making the model more efficient.

[0021] DPAM-PSPNet combines the advantages of PSPNet and DPAM, and uses DPAM to enhance the feature aggregation capability in PSPNet. In the present invention, the characteristics of DPAM-PSPNet are used to effectively extract global contextual prior information, thereby obtaining information with different sizes and changes between different sub-regions, thereby improving the performance of small target object and content recognition in complex scenes of medical images.

[0022] Finally, considering that both ultrasound image and magnetic resonance image (HR-VWI image) data are highly sensitive to carotid artery plaque assessment, the present invention achieves deep fusion of the two modality data by contrastive learning fusion Skip-cross. Specifically, Figure 4 As shown, each layer of the current modality feature is connected to each layer of the other modality feature in a feedforward manner. For each layer of the current modality feature, the features of all previous layers are used as its input, and its own feature map is used as the input of all subsequent layers of other modalities. Specifically applied in this application, Skip-cross connects the temporal feature map of the carotid ultrasound image to the corresponding position of the spatial feature map of the carotid high-resolution nuclear magnetic resonance, and all feature maps achieve dense cross-modal connection. In this process, the Skip-cross module is used to adaptively identify the optimal combination period of the two image features. Then, the Skip-cross fusion network (SkipcrossNets) makes full use of the multimodal information of the encoder stage and performs feature fusion in Skip-cross, so that in the process of restoring the resolution, the feature maps of each modality cross-influence each other, ensuring that the fusion of spatial and temporal features occurs not only in the last layer of the network, but also truly realizes the deep fusion of cross-modal features. Finally, the quantitative evaluation of carotid plaques can be performed based on the feature map output after deep fusion. Embodiment 2

[0023] In order to better understand the present invention, this embodiment describes the present invention in the form of a system structure. Figure 2 As shown, a DP-LSTM-based carotid artery plaque ultrasound quantitative evaluation system includes: An image source acquisition module, used to acquire carotid artery dynamic ultrasound images and carotid artery high-resolution vascular magnetic resonance images corresponding to each time point; The time series capture module is used to extract multi-scale information of the carotid artery dynamic ultrasound image through void space pyramid pooling. After fusing the bottom-level time series features with the high-level time series features, the corresponding time series features are captured through the long-term and short-term neural network. A spatial extraction module is used to extract spatial features of plaque targets in carotid artery high-resolution vascular magnetic resonance images by identifying and cutting them through a dual-path attention segmentation network; A feature fusion module is used to perform cross-modal feature fusion of the temporal features of the carotid artery dynamic ultrasound image and the spatial features of the corresponding temporal features of the carotid artery high-resolution vascular magnetic resonance image by means of multimodal fusion; The quantitative evaluation module is used to perform quantitative evaluation of carotid artery plaques based on feature maps after cross-modal feature fusion.

[0024] Furthermore, in the timing capture module, the timing features of the carotid artery dynamic ultrasound image are extracted through a deep residual convolutional network, and combined with the information of each scale to obtain the timing features from the bottom layer to the high layer.

[0025] Furthermore, in the spatial extraction module, the dual-path attention segmentation network extracts global context prior information to obtain information about changes between each size and each sub-region, thereby realizing the extraction of spatial features of the patch target.

[0026] Furthermore, in the spatial extraction module, an attention mechanism is added before the path attention segmentation network to guide the attention of patch target cutting.

[0027] Furthermore, in the feature fusion module, the multimodal fusion method is specifically as follows: each layer of the current modality feature is connected to each layer of the other modality feature in a feedforward manner. For each layer of the current modality feature, the features of all previous layers are used as its input, and its own feature map is used as the input of all subsequent layers of other modalities.

[0028] In summary, the DP-LSTM-based ultrasound quantitative evaluation method and system for carotid plaques described in the present invention improves the evaluation accuracy of carotid plaques by capturing the spatial temporal characteristics and motion transformation temporal characteristics of plaques, helps doctors obtain more accurate carotid plaque identification data support, and reduces the risk of misjudgment caused by image quality and noise. By memorizing the time information between temporal characteristics through long-term and short-term neural networks, the changing trend of plaques over time can be better understood, thereby providing a more reliable basis for disease prediction.

[0029] Through the deep fusion of ultrasound image and magnetic resonance image data, this cross-modal feature fusion can not only improve the accuracy of diagnosis, but also help doctors better understand the performance of plaques under different imaging technologies, thereby making more comprehensive clinical judgments.

[0030] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0031] In addition, in the present invention, descriptions such as "first", "second", "one", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0032] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0033] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A DP-LSTM-based method for ultrasonic quantitative evaluation of carotid plaque, characterized in that: Includes steps: S1: Acquire dynamic ultrasound images of the carotid artery and high-resolution vascular magnetic resonance images of the carotid artery corresponding to each time point; S2: Extract multi-scale information of carotid artery dynamic ultrasound images through void space pyramid pooling, and after fusing the underlying temporal features with the high-level temporal features, capture the corresponding temporal features through long-term and short-term neural networks; S3: Spatial feature extraction under recognition and cutting of plaque targets in high-resolution vascular magnetic resonance images of carotid arteries using a dual-path attention segmentation network; S4: The temporal features of the carotid artery dynamic ultrasound image and the spatial features of the corresponding temporal features of the carotid artery high-resolution vascular magnetic resonance image are fused across modalities by means of multimodal fusion; S5: Quantitative assessment of carotid artery plaques based on feature maps after cross-modal feature fusion.

2. The method for ultrasonic quantitative evaluation of carotid plaque based on DP-LSTM as claimed in claim 1, characterized in that: In the step S2, the time series features of the carotid artery dynamic ultrasound image are extracted through a deep residual convolutional network, and combined with each scale information to obtain the time series features from the bottom layer to the high layer.

3. The method for ultrasonic quantitative evaluation of carotid plaque based on DP-LSTM as claimed in claim 1, characterized in that: In the step S3, the dual-path attention segmentation network extracts global context prior information, obtains information on changes between each size and each sub-region, and realizes the extraction of spatial features of the patch target.

4. The method for ultrasonic quantitative assessment of carotid plaque based on DP-LSTM as claimed in claim 3, characterized in that: In the S3 step, an attention mechanism is added before the path attention segmentation network to guide the attention of patch target cutting.

5. The method for ultrasonic quantitative evaluation of carotid plaque based on DP-LSTM as claimed in claim 1, characterized in that: In the step S4, the multimodal fusion method is specifically as follows: each layer of the current modality feature is connected to each layer of the other modality feature in a feedforward manner, and for each layer of the current modality feature, all the features of the previous layers are used as its input, and its own feature map is used as the input of all subsequent layers of other modalities.

6. A DP-LSTM-based carotid artery plaque ultrasound quantitative assessment system, characterized in that: include: An image source acquisition module, used to acquire carotid artery dynamic ultrasound images and carotid artery high-resolution vascular magnetic resonance images corresponding to each time point; The time series capture module is used to extract multi-scale information of the carotid artery dynamic ultrasound image through void space pyramid pooling. After fusing the bottom-level time series features with the high-level time series features, the corresponding time series features are captured through the long-term and short-term neural network. A spatial extraction module is used to extract spatial features of plaque targets in carotid artery high-resolution vascular magnetic resonance images by identifying and cutting them through a dual-path attention segmentation network; A feature fusion module is used to perform cross-modal feature fusion of the temporal features of the carotid artery dynamic ultrasound image and the spatial features of the corresponding temporal features of the carotid artery high-resolution vascular magnetic resonance image by means of multimodal fusion; The quantitative evaluation module is used to perform quantitative evaluation of carotid artery plaques based on feature maps after cross-modal feature fusion.

7. The DP-LSTM-based carotid artery plaque ultrasonic quantitative assessment system according to claim 6, characterized in that: In the timing capture module, the timing features of the carotid artery dynamic ultrasound image are extracted through a deep residual convolutional network, and combined with each scale information to obtain the timing features from the bottom layer to the high layer.

8. The DP-LSTM-based carotid artery plaque ultrasonic quantitative assessment system according to claim 6, characterized in that: In the spatial extraction module, the dual-path attention segmentation network extracts global context prior information, obtains information on changes between each size and each sub-region, and realizes the extraction of spatial features of patch targets.

9. The DP-LSTM-based carotid artery plaque ultrasonic quantitative assessment system according to claim 8, characterized in that: In the spatial extraction module, an attention mechanism is added before the path attention segmentation network to guide the attention of patch target cutting.

10. The DP-LSTM-based carotid artery plaque ultrasonic quantitative assessment system according to claim 6, characterized in that: In the feature fusion module, the multimodal fusion method is specifically as follows: each layer of the current modality feature is connected to each layer of the other modality feature in a feedforward manner, and for each layer of the current modality feature, all the features of the previous layers are used as its input, and its own feature map is used as the input of all subsequent layers of other modalities.

Citation Information

Patent Citations

  • Carotid plaque tracking and echo classification framework and method based on ultrasonic video

    CN115381492A

  • Brain dysfunction auxiliary evaluation method based on multi-modal data fusion

    CN115553752A

  • Retina image blood vessel segmentation method, system and device and medium

    CN117495881A

  • Depression assessment method and device based on audio-visual multi-modal data fusion

    CN118173267A

  • Ultrasonic image plaque segmentation method, system and equipment based on deep learning

    CN118397015A