Ultrasound video based carotid plaque tracking and echogenicity classification framework and method

Abstract

The application discloses a carotid plaque tracking and echo classification framework and method of an ultrasonic video, a multiscale hollow encoder, an internal and external feature decoupler and a tracking network are combined to form a carotid plaque tracking framework, and a carotid plaque echo classification framework is established based on feature recombination and a double-channel 3D-Attention framework; an ultrasonic video image is input into the tracking framework to position a plaque position, feature recombination is used to select a plaque contour as local feature input of the echo classification framework, and an original ultrasonic video image is used as global feature input of the echo classification framework. The tracking and classification framework combines deep learning technologies such as feature hierarchical level fusion, multiscale context temporal feature extraction and a three-dimensional attention mechanism, not only solves the optimization problem caused by the change of the state between ultrasonic sections in the carotid ultrasonic video, and fully combines the context features and the vascular environment features of the plaque, so that the features corresponding to each type of state reflecting the instability of the plaque are fully reflected.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer, and particularly relates to a neural network framework for tracking and echo classification of carotid plaque in medical images. BACKGROUND

[0002] Cardiovascular disease is the main disease with the highest incidence and mortality in the world, and is also an important health problem currently faced by the world. The rupture and shedding of unstable plaque is the main cause of acute cardiovascular emergency, and carotid plaque is formed by the deposition of blood lipids, connective tissue matrix and smooth muscle cells, macrophages and various components, and the damage of plaque will form thrombus, which is easy to cause acute cardiovascular events such as stroke. In recent decades, with the development of computer technology, magnetic resonance imaging (MRI), computed tomography (CT), ultrasound and other computer-aided medical imaging technologies have been widely used in clinical practice. Medical ultrasound imaging technology is widely used in the diagnosis of carotid plaque due to its convenience and non-radiation examination. In the early clinical risk assessment of carotid plaque, it is often affected by many factors, among which the echo type reflected by the plaque composition is usually the key observation factor of the doctor. In clinical ultrasound diagnosis, carotid plaque is usually divided into high echo, mixed echo and low echo plaque types, which can reflect the stability of the plaque. Although early diagnosis can be found to intervene and treat cardiovascular diseases, in clinical practice, different results are often obtained due to different judgments of doctors, and there is a lack of unified standard. Therefore, it is necessary to establish a more robust, lower labor cost and more repeatable diagnosis process to improve the diagnosis efficiency and accuracy, and it is necessary to assist clinical plaque diagnosis based on computer technology.

[0003] So far, various methods have studied the application of machine learning and deep learning in ultrasound carotid plaque echo classification, but the research objects mainly face ultrasound images, so the changes of pixels in the time domain and the correlation between video frames are ignored. In clinical practice, doctors do not use a single image, but make comprehensive diagnosis combined with the dynamic morphological features of the plaque in the video. This shows that in-depth mining of ultrasound video can better capture the plaque features and more accurately classify the plaque. There are two main challenges in the research of carotid plaque echo classification based on ultrasound video: on the one hand, how to distinguish the tissues similar to the plaque. On the other hand, how to select the key ultrasound sections reflecting the properties of the plaque from the video. The traditional method based on detection is not suitable for tracking scenes with real-time changes in target semantics. Figure 1 and Figure 2 As shown in FIGS. 1 and 2, the interference of similar tissues or inaccurate ultrasound sections may lead to incorrect diagnosis. SUMMARY

[0004] The technical problem to be solved by the present application is how to locate the plaque and exclude the confused tissues to complete the plaque classification.

[0005] To achieve the above object, the concept of the present application is: by tracking module based on multi-scale hollow encoder and feature decoupling, combined with feature level fusion, multi-scale context temporal feature extraction, three-dimensional attention mechanism and other deep learning technologies, an accurate and stable carotid plaque tracking classification framework is established. Based on the optimization of the tracking algorithm, the classification module can relieve the influence of similar tissues from the blood vessels and irregular changes in the shape of the plaque in the ultrasound image. Further, it can efficiently combine the context features between different ultrasound sections and extract key temporal features, reduce feature redundancy, and complete accurate and robust carotid ultrasound video plaque tracking and echo classification. The framework can be widely used for tracking and echo classification of various carotid plaques with multi-position distribution and multi-morphology change.

[0006] The technical scheme of the present application:

[0007] The carotid plaque tracking and echo classification framework based on ultrasound video includes a multi-scale hollow encoder, an internal and external feature decoupler and a tracking network to form a carotid plaque ultrasound video tracking framework, and an echo classification framework for carotid plaques is established based on feature reorganization and a dual-channel 3D-Attention framework; the ultrasound video image is input into the tracking framework to locate the plaque position, the plaque contour is selected as the local feature input of the echo classification framework based on feature reorganization, and the original ultrasound video image is used as the global feature input of the echo classification framework.

[0008] The multi-scale hollow encoder includes:

[0009] A1. Receive the current frame (t), the previous frame (t-1) and the next frame (t+1) as input, C2, C3 and C4 represent the output features of ResNet50;

[0010] A2. C2 and C4 are followed by three residual blocks, wherein a 1x1 convolutional layer is applied to reduce the channel dimension, a 3x3 dilated convolution with different dilation rates is added to refine the semantic context, and output features with multiple receptive fields are generated;

[0011] A3. Add C3 skip connection after the residual block after C2 and C4, the subnet after the encoder layer adopts the design paradigm of JDE, and is independently trained at each scale.

[0012] The internal and external feature decoupler includes:

[0013] B1. Given input Where i represents the number of channels, H and W represent the length and width of the feature map, and the calculation of the global vector can be represented as:

[0014]

[0015] where X g = W g ⊙ X, W g represent learnable linear projections;

[0016] B2. The internal weight vector v interior and the external weight vector

[0017] v interior = σ(W i1 ⊙ (δ(W i2 ⊙ v))) (2)

[0018] v exterior = σ(W e1 ⊙ (δ(W e2 ⊙ v))) (3)

[0019] where W i1 , W i2 , W e1 and W e2 represent four learnable weight matrices, σ(·) and δ(·) are sigmoid and relu activation functions respectively;

[0020] B3. Finally, the original features are re-weighted on each channel:

[0021]

[0022]

[0023] where X exterior is used to distinguish the state of the patch to assist detection, X interior is used to distinguish the identity of the patch to assist tracking.

[0024] The tracking network includes:

[0025] C1. The tracking network consists of two parts, the detection subnet (box subnet and class subnet) and the linking subnet (Embedding). The detection subnet predicts the location and echo state of the patch, and the linking subnet completes the tracking trajectory according to the correlation between the detection results in each frame.

[0026] C2. The detection subnet consists of two branches, where the class subnet contains four convolutional layers, each consisting of a 1x1 convolution and a 3x3 convolution; the box regression subnet has the same structure as the first three layers of the class subnet.

[0027] C3. The first two layers of the linking subnet consist of 3x3 convolutional layers, and the last layer uses a fully connected layer.

[0028] C4. Identity-based Embedding features and bounding boxes are used to calculate the correlation matrix by cosine similarity, and then the Hungarian algorithm and Kalman filtering are used to match the plaques between different frames and complete the establishment of the tracking trajectory.

[0029] Feature reorganization includes:

[0030] D1. For each plaque, select the highest-scoring profile of each type of plaque from the ultrasound sections selected by the tracking framework, in addition to selecting the feature map of the penultimate layer in the profile classification subnet and the link subnet as supplementary features, and finally splicing the tracking profile and the supplementary features as local features of the dual-channel time-series plaque echo classification model;

[0031] D2. Select all unselected ultrasound sections as global information, focusing on the changes in vascular profiles and echo attenuation as global features of the dual-channel time-series plaque echo classification model.

[0032] The dual-channel 3D-Attention framework includes:

[0033] E1. The global channel and the local channel each contain three convolutional blocks, each of which consists of two 3D-Conv layers and a global pooling layer;

[0034] E2. Add se-block to the global channel for 3D-level expansion to learn the weight of each frame feature in each channel and select features;

[0035] E3. A connection is added between the two channels to integrate the information learned together;

[0036] E4. The information learned together is followed by a fully connected layer for classification, consisting of two fully connected layers and a layer for plaque echo classification.

[0037] An ultrasound video-based carotid plaque tracking and echo classification method includes:

[0038] A. Perform lesion labeling and establish labels for the ultrasound video;

[0039] B. Data preprocessing, such as normalization, resolution processing, and data enhancement;

[0040] C. Establish a carotid plaque tracking framework and input the data obtained in step B into the framework for plaque positioning;

[0041] D. Establish a carotid plaque echo classification framework based on tracking, and input the plaques located in step C as local features,

[0042] The video data obtained in step B is input as global features.

[0043] The carotid plaque tracking framework is composed of a multi-scale hollow encoder, an internal and external feature decoupler and a tracking network, and adopts a JDE model based on the combination of MDE and IEFD modules.

[0044] An echo classification framework for carotid plaque is established based on feature recombination and a double-channel 3D-Attention framework, and the feature recombination includes selecting the highest-scored contour of each type of plaque as a local feature from the ultrasound sections selected by the tracking framework, and selecting all unselected ultrasound sections as global features.

[0045] The se-block is added to the global channel of the double-channel 3D-Attention framework for 3D layer expansion, the weight of each frame feature in each channel is learned, and feature selection is performed.

[0046] The beneficial effects of the present application are:

[0047] 1. In the traditional carotid plaque classification framework, the research based on ultrasound video expansion is relatively lacking, the method in the image research ignores the time sequence features between the ultrasound sections, and is contrary to the prior knowledge in the clinic; the technology in the video research ignores the difference between the carotid plaque image and the natural image, and lacks targeted optimization. These problems lead to the performance bottleneck of the current method.

[0048] 2. The newly proposed classification framework based on tracking, through the tracking module based on multi-scale hollow encoder and feature decoupling, combined with feature hierarchical fusion, multi-scale context temporal feature extraction, three-dimensional attention mechanism and other deep learning technologies, not only solves the optimization problem caused by the change of the state between the ultrasound sections in the carotid ultrasound video, but also fully combines the context features and the vascular environment features of the plaque, so that the features corresponding to each type of state reflecting the instability of the plaque are fully reflected.

[0049] 3. The classification framework based on tracking combines the prior knowledge of the clinic, optimizes the extraction of features, and the feature recombination strategy refines the features, improves the accuracy and stability of the classification. DETAILED DESCRIPTION

[0050] Figure 1 The difficulties faced by the current work are described from the perspective of ultrasound image data, the red arrow in (a) points to the interference tissue similar to the plaque, and the highlighted part of the heat map in (b) represents the area of interest of the deep learning model, and the similar tissue interferes with the learning of the model.

[0051] Figure 2 Different ultrasound sections in the carotid ultrasound video are intercepted, and different sections in the same ultrasound video may reflect different plaque states, such as (a) and (b).

[0052] Figure 3 is the overall architecture of the ultrasound video based carotid plaque tracking and echogenicity classification framework of the present invention.

[0053] Figure 4 is the overall flowchart of the ultrasound video based carotid plaque tracking and echogenicity classification framework of the present invention.

[0054] Figure 5 is Figure 3 is the detailed configuration of the multi-scale hollow encoding module (MDE).

[0055] Figure 6 is the selection method of the tracking framework to complete the lesion frame selection.

[0056] Figure 7 is the feature reorganization flowchart based on the tracking framework.

[0057] Figure 8 is Figure 3 is the detailed configuration of the classification network.

[0058] Figure 9 lists the performance comparison of the control experiments in the multi-scale hollow encoding module with the selected feature extraction strategy as the variable.

[0059] Figure 10 lists the influence of the selected feature extraction strategy in the multi-scale hollow encoding module on different echogenicity type plaques.

[0060] Figure 11 lists the performance comparison of our carotid plaque ultrasound video tracking module and the mainstream plaque tracking method.

[0061] Figure 12 shows the carotid plaque ultrasound video tracking results. These cases include plaques of various echogenicity states, different locations, and various morphological changes.

[0062] Figure 13 shows the carotid plaque ultrasound video tracking classic cases. Compared with other methods, our method has very stable and accurate tracking results for plaques with changing echogenicity states in different ultrasound sections in the ultrasound video.

[0063] Figure 14 lists the performance comparison of the control experiments in the carotid plaque ultrasound video classification module with the selected different feature reorganization strategies as the variable.

[0064] Figure 15 lists the performance comparison of our carotid plaque ultrasound video classification module and other plaque classification methods.

[0065] Figure 16 The performance comparison of different classification modules with and without tracking algorithm is shown. DETAILED DESCRIPTION

[0066] Embodiment 1:

[0067] First, the data set is preprocessed, including data enhancement and standardization, and then applied to the natural image detection and tracking framework, incorporating prior knowledge in the carotid plaque clinical observation and diagnosis process and improving it, accurately locating the plaque and excluding interference features. Due to the environmental features in the blood vessels containing auxiliary plaque classification, the context information between the plaque and the blood vessel environment multi-scale ultrasound section is finally combined, and the key timing features are extracted to reduce feature redundancy, establishing a precise and robust carotid ultrasound video plaque tracking and echo classification framework.

[0068] The carotid plaque tracking and echo classification framework based on ultrasound video includes a multi-scale hollow encoder, an internal and external feature decoupler, and a tracking network to form a carotid plaque ultrasound video tracking framework, and a carotid plaque echo classification framework based on feature reorganization and a dual-channel 3D-Attention framework; the ultrasound video image input tracking framework locates the plaque position, and the feature reorganization selects the plaque contour as the local feature input of the echo classification framework, and the original ultrasound video image as the global feature input of the echo classification framework.

[0069] The multi-scale hollow encoder includes:

[0070] A1. receives the current frame (t), the previous frame (t-1) and the next frame (t+1) as input, C2, C3 and C4 represent the output features of ResNet50;

[0071] A2. C2 and C4 are followed by three residual-like blocks, where a 1x1 convolutional layer is applied to reduce the channel dimension, and a 3x3 dilated convolution with different dilation rates is added to refine the semantic context, generating output features with multiple receptive fields;

[0072] A3. C3 skip connection is added after the residual-like block after C2 and C4, the subnet after the encoder layer adopts the design paradigm of JDE, and is independently trained at each scale.

[0073] The internal and external feature decoupler includes:

[0074] B1. Given input Where i represents the number of channels, H and W represent the length and width of the feature map, and the calculation of the global vector can be represented as:

[0075]

[0076] Where Xg = W g X, W g represents a learnable linear projection;

[0077] B2. Compute the internal weight vector v interior and the external weight vector

[0078] v interior = σ(W i1 X, W i2 X, v) (2)

[0079] v exterior = σ(W e1 X, W e2 X, v) (3)

[0080] where W i1 , W i2 , W e1 and W e2 represent four learnable weight matrices, σ(·) and δ(·) are sigmoid and relu activation functions, respectively;

[0081] B3. Finally, the original features are re-weighted on each channel:

[0082]

[0083]

[0084] where X exterior is used to distinguish the state of the blob to assist detection, and X interior is used to distinguish the identity of the blob to assist tracking.

[0085] The tracking network includes:

[0086] C1. The tracking network consists of two parts, the detection subnet and the linking subnet. The detection subnet predicts the location and echo state of the blob, and the linking subnet completes the tracking trajectory according to the correlation between the detection results in each frame.

[0087] C2. The detection subnet consists of two branches, where the contour classification branch contains four convolutional layers, each of which consists of a 1x1 convolution and a 3x3 convolution; the contour regression branch has the same structure as the first three layers of the classification branch.

[0088] C3. The first two layers of the linking subnet consist of 3x3 convolutions, and the last layer is a fully connected layer.

[0089] C4. The identity-based Embedding features and bounding boxes are used to calculate the correlation matrix by cosine similarity. Then, the Hungarian algorithm and Kalman filter are used to match the patches between different frames and complete the establishment of the tracking trajectory.

[0090] Feature reorganization includes:

[0091] D1. For each patch, select the highest-scoring profile of each type of patch from the ultrasound sections selected by the tracking framework as the local features of the dual-channel time-series patch echo classification model.

[0092] D2. Select all unselected ultrasound sections as global information, focusing on changes in blood vessel profiles and echo attenuation, as global features of the dual-channel time-series patch echo classification model.

[0093] The dual-channel 3D-Attention framework includes:

[0094] E1. The global channel and the local channel each contain three convolution blocks, each of which consists of two 3D-Conv layers and a global pooling layer;

[0095] E2. Add se-block to the global channel for 3D-level expansion to learn the weight of each frame feature in each channel and select features;

[0096] E3. A connection is added between the two channels to integrate the information learned together;

[0097] E4. The information learned together is followed by a fully connected layer for classification, consisting of two fully connected layers and a layer for patch echo classification.

[0098] In this embodiment, the server running the experiment is equipped with an Intel(R) Xeon(R) Gold 6226R CPU@2.90GHz and 128.0GB RAM, and two GeForce GTX 3090Ti. In terms of modeling, the implementation of the model is based on the open source TensorFlow public Keras library, namely TensorFlow-gpu 2.2.0 and Keras GPU 2.3.0, we use five-fold cross-validation to evaluate our model, study the influence of single components for each task, and compare our method with the most advanced network model. In terms of experimental settings, the experiment selects Adam as the optimization algorithm, the batch size is set to 8, the initial learning rate is set to 0.0003, the weight decay factor is 0.2, and the number of iterations is set to 800, and the learning rate is reduced to 1 / 2 of the previous one every 100 iterations. In terms of loss function, the tracking module still uses the loss function of the traditional JDE model, and the classification module uses the cross-entropy loss function with L2 regularization penalty. In terms of performance evaluation, the experiment evaluates the tracking performance of the method through performance parameters such as MOTA (multiple-objects tracking accuracy), MOTP (multiple-object tracking precision), IDF1, MT (mostly tracked), ML (mostly lost), and evaluates the classification performance of the method through performance parameters such as Accuracy, Precision, Recall, F1, etc. The larger the performance indicators such as MOTA, MOTP, IDF1, MT, Accuracy, Precision, Recall, F1, etc., the better the performance of the method, and vice versa for ML.

[0099] The data set used by the application is collected from cooperative hospitals, and 377 plaque ultrasound videos from 216 patients are used. Each video is cropped to 120 frames by a doctor, and the label of the video type includes weak echo, mixed echo and strong echo, in addition, each plaque in the video is identity labeled, and the echo state of the plaque in each ultrasound section is labeled, and the label of the section type includes weak echo state, mixed echo state and strong echo state. The collected data includes 130 high echo plaques, 101 low echo plaques and 146 mixed echo plaques, covering different types of plaques. The labels of the data set are manually drawn by three professional doctors who have been uniformly trained, and mutual evaluation is carried out after the preliminary labeling is completed to reduce the difference.

[0100] The main framework structure of the application is as shown in Figure 3 The main flow chart is as shown in Figure 4 ​

[0101] A. Lesion annotation and label establishment: The labels of the dataset are manually drawn by three professional doctors who have been uniformly trained, including identity annotation of each plaque in the video and annotation of the echo state of each plaque in the ultrasound section. The labels of the section type include weak echo state, mixed echo state and strong echo state.

[0102] B. Data preprocessing: The carotid ultrasound video is subjected to size standardization and pixel normalization. The specific steps are as follows:

[0103] The resolution of the original ultrasound video changes after the doctor cuts off the useless ultrasound machine parameter information. In order to realize the unified standard input of the model, the present application carries out three-channel mean normalization with mean value (0.435, 0.416, 0.446) and variance (0.239, 0.221, 0.223) on the ultrasound video, and uniformly adjusts the resolution to 740x540, completing the normalization processing between different ultrasound videos.

[0104] C. Establish a tracking framework: Based on the carotid plaque echo classification research of the ultrasound video, there are two challenges: on the one hand, how to distinguish the tissues similar to the plaque, such as Figure 1 As shown, the tissues similar to the plaque interfere with the learning of the model. On the other hand, how to select the key ultrasound section reflecting the nature of the plaque from the video, such as Figure 2 As shown, different ultrasound sections in the same video reflect different plaque states, and the traditional method is not suitable for the tracking scene of the target semantic real-time change. Therefore, we establish a carotid plaque ultrasound video tracking framework combining multi-scale hollow coding, internal and external feature decoupling technology and tracking network, including the following specific contents:

[0105] C1, Multi-scale hollow coding module (MDE): The multi-scale hollow coding module (MDE) is improved on the basis of the traditional FPN (Feature Pyramid Networks), Figure 3 (a) is its basic structure, Figure 5 is the detailed configuration of the module. The specific process is as follows:

[0106] C11, receiving the current frame (t), the previous frame (t-1) and the next frame (t+1) as input, wherein C2, C3 and C4 represent the output features of ResNet50, and the sampling rates are {2, 4, 8} respectively.

[0107] C12, C2 and C4 are followed by three similar residual blocks, in which a 1x1 convolution layer is applied to reduce the channel dimension, and a 3x3 dilated convolution with different dilation rates is added to refine the semantic context, generating output features with multiple receptive fields. The purpose of using dilated convolution is to maintain the spatial resolution of the features and preserve more details in the output of the encoder.

[0108] C13, C3 skip connection is added after the similar residual block after C2 and C4 to enhance the characteristics of small targets. For the subnetwork after the encoder layer, the design paradigm of JDE is adopted, but it is trained independently at each scale.

[0109] C2, internal-external feature decoupling technology (IEFD): in the tracking scene where the target semantic changes in real time, the traditional tracking network often ignores the part of the target area class that changes, and always focuses on the original semantic part of the target, or regards the new semantic part as a new target. This is mainly because in the traditional detection-based tracking method, the purpose of the model is to strengthen the connection between the same target. In contrast, the detection part aims to maximize the difference between different classes, even if these classes belong to the same target. The internal-external feature decoupling technology (IEFD) tracks the contour according to the internal similarity between target identities, and detects the state according to the external difference between target semantics, and its operation process mainly consists of three parts: global receptive field feature vector generation, channel weight calculation and weighting of input features, Figure 3 (b) is its basic structure, and its specific process is as follows:

[0110] C21, given input Where i represents the number of channels, H and W represent the length and width of the feature map, and the calculation of the global vector can be represented as:

[0111]

[0112] Where X g =W g ⊙X, W g represents a learnable linear projection, and in the present application, we use 1x1 convolution.

[0113] C22, the internal weight vector v interior and the external weight vector

[0114] v interior = σ (W i1 ⊙ (δ (W i2 ⊙v) ) ) (2)

[0115] v exterior = σ (W e1 ⊙ (δ (We2 ⊙v))) (3)

[0116] where W i1 , W i2 , W e1 and W e2 represent four learnable weight matrices, and σ(·) and δ(·) are sigmoid and relu activation functions, respectively.

[0117] C23, Finally, the original features are re-weighted on each channel:

[0118]

[0119]

[0120] where X exterior is used to distinguish the state of the patch to assist detection, and X interior is used to distinguish the identity of the patch to assist tracking.

[0121] C3, Tracking network: The tracking network consists of two parts, the detection subnet (box subnet and class subnet) and the linking subnet (Embedding). The detection subnet predicts the location and echo state of the patch, and the linking subnet completes the tracking trajectory according to the correlation between the detection results in each frame.

[0122] C31, The detection subnet consists of two branches, where the class subnet contains four convolutional layers, each consisting of a 1x1 convolution and a 3x3 convolution; the box regression subnet has the same structure as the first three layers of the class subnet.

[0123] C32, The first two layers of the linking subnet consist of 3x3 convolutional layers, and the last layer uses a fully connected layer.

[0124] C33, The identity-based Embedding feature and the detection box are related by cosine similarity, and then the Hungarian algorithm and Kalman filtering are used to match the patches between different frames and complete the establishment of the tracking trajectory.

[0125] The detection subnet and the linking subnet still follow the structure of JDE.

[0126] D, Establish a classification framework: Although the tracking algorithm we propose can accurately locate the patch position and exclude the interference of similar tissues, the same patch in different ultrasound sections in the same ultrasound video may reflect different echo semantics. Therefore, we propose a dual-channel temporal patch echo classification model based on ultrasound video, which includes the following specific contents:

[0127] D1, Feature recombination: In the clinical carotid plaque echo evaluation, doctors often combine factors such as lesion properties, vascular profiles, and echo attenuation for comprehensive evaluation. Based on the above clinical knowledge, the application recombines the tracking framework as the input of the classification framework. The specific method is as follows:

[0128] D11, For each plaque, 30 plaque profiles are selected from the 120 ultrasound sections completed by the tracking framework for lesion framing. The selection method is: from every 12 ultrasound sections of the ultrasound video, the highest score profile corresponding to the plaque in three states is selected, a total of 30. In addition, the feature maps of the second-to-last layer in the profile classification subnet and the link subnet are selected as supplementary features, and the number of channels is controlled to be 24. Finally, the tracking profile and the supplementary features are spliced as the local features of the dual-channel time sequence plaque echo classification model, which can highlight the risky ultrasound performance while reducing feature redundancy.

[0129] D12, All 120 unframed ultrasound sections are selected as global information, focusing on changes in vascular profiles and echo attenuation, as global features of the dual-channel time sequence plaque echo classification model.

[0130] D2, Dual-channel 3D-Attention time sequence model: The dual-channel time sequence model based on recombined features proposed by the application is mainly different from the common multi-channel model. Our application is more in line with the prior knowledge in clinical practice. The local path (Local-Path) simulates the observation of the doctor in the clinical practice on the lesion area, captures the semantic features of the plaque lesion, and finds the echo discrimination basis from the semantic changes of the plaque. The global path (Global-Path) simulates the doctor's search for judgment basis related to plaque echo from the environment around the lesion, and captures features related to plaque echo from global information. Figure 6 is the selection method of the tracking framework for lesion framing, Figure 7 is the main process of feature recombination, Figure 3 (d) is its basic structure, Figure 8 is the detailed configuration of the module, which is specifically configured as follows:

[0131] D21, Each green convolution block is composed of two 3D-Conv layers and a global pooling layer.

[0132] D22, se-block is an extension of the traditional feature attention mechanism on the 3D level. It is used to learn the weight of each frame feature in each channel for feature selection

[0133] D23, A connection is added between the two channels to integrate the information learned together.

[0134] D24, Classification after joint learning information with full connection layer, FC-block consists of two full connection layers and SoftMax layer for plaque echo classification.

[0135] Referring to Figure 9 , the performance comparison of the control experiment with the selected feature extraction strategy as a variable in the multi-scale hollow coding module is listed.

[0136] The purpose of this experiment is to verify the effectiveness of the multi-scale hollow coding module and the internal and external feature decoupling module in the tracking framework, therefore, the comparison strategy of the experiment is as follows: we use the JDE model as the backbone network, and use the JDE based on FPN as the baseline, and compare the JDE based on MDE, the JDE combined with FPN and IEFD module, and the JDE combined with MDE and IEFD module. The experimental results show that the JDE model based on the combination of MDE and IEFD module achieves the optimal performance.

[0137] Referring to Figure 10 , the influence on different echo type plaques when the selected feature extraction strategy is used as a variable in the multi-scale hollow coding module is shown. After comparison, it can be seen that our invention has obvious improvement on mixed echo and low echo plaques, in fact, these two plaques are irregular in shape and have large differences in different ultrasound sections in the same video, so it is often difficult to evaluate their properties in clinical practice. This result shows that our model optimizes the optimization contradiction mainly existing in mixed echo plaques and low echo plaques.

[0138] Referring to Figure 11 , the performance comparison of our carotid plaque ultrasound video tracking module with mainstream plaque tracking methods is listed. The purpose of this experiment is to test the tracking performance of the proposed carotid plaque ultrasound video tracking architecture, therefore, in addition to the method of the invention, the experiment also compares five other mainstream methods, including JDE, CTracker, DeepSortv2, MAT and CSTrack. The experimental results show that the tracking method proposed in this paper is better than the mainstream tracking method in each performance index.

[0139] Reference Figure 12 , the carotid plaque ultrasound video tracking result case is shown. The orange box is the tracking result of the model, the green box is the expert annotation comparison, and the number in the upper left corner represents the plaque with different identity. The tracking results show that the tracking method of the invention has accurate and stable tracking performance on plaques with various echo states, plaques in different positions, and plaques with various morphological changes.

[0140] Reference Figure 13, which shows a classic case of carotid plaque ultrasound video tracking. In this case, the mixed echo plaque video reflects different echo states in different ultrasound sections, and only the strong echo state of the plaque is reflected in frame 35, and the mixed echo state of the plaque is reflected in frames 48 and 56, while the shape of the plaque also changes significantly. The tracking results of JDE and CSTrack failed to associate with the correct trajectory. Specifically, JDE ignored the low echo area of the plaque in frame 48 and incorrectly labeled the area as a new plaque in frame 56. Similarly, CSTrack produced incorrect results in two frames. In contrast, our tracking module successfully identified the target. The case shows that our method has very stable and accurate tracking results for plaques with changing echo states in different ultrasound sections in ultrasound videos.

[0141] Reference Figure 14 , which lists the performance comparison of the control experiments in the carotid plaque ultrasound video classification module with different feature reorganization strategies selected as variables. The purpose of this experiment is to verify the effectiveness of the feature reorganization method and the dual-channel time series classification method in the classification framework, and to explore the influence of different path sampling parameters on the classification performance. The subscript s in the figure represents the ultrasound section (global feature), and the subscript t represents the tracking contour (local feature). The comparison strategies of the experiment are as follows: whether based on multi-scale features, single path or multi-path, whether based on 3D attention mechanism. The experimental results show that the dual-channel model based on 3D attention mechanism combined with global features and local features achieves the best performance, and the optimal sampling parameters are 100% ultrasound sections for global features and contour selection in every 12 ultrasound sections for local features.

[0142] Reference Figure 15 , which lists the performance comparison of our carotid plaque ultrasound video classification module with other plaque classification methods. The purpose of this experiment is to test the classification performance of the proposed carotid plaque ultrasound video classification architecture, therefore, in addition to the method of the invention, the experiment also compares seven other mainstream methods, including C3D, LRCN, TSN, ECO, I3D, STM, and SlowFast. The experimental results show that the classification method proposed in this paper is better than the mainstream classification methods in each performance indicator.

[0143] Reference Figure 16 , which shows the performance comparison of different classification modules with and without tracking algorithms. The purpose of this experiment is to test the necessity of the tracking framework in the classification framework in the invention. Among the models participating in the test, the model with the worst classification performance based on the tracking framework is better than the model with the best classification performance in the single classification framework. This result shows that the tracking framework in the invention can effectively focus on plaque features

[0144] The interference of similar tissues is excluded, and the accuracy and stability of the classification method are improved.

[0145] In summary, the present application proposes a carotid plaque tracking and echo classification framework based on ultrasound video, which is used for tracking and echo classification of plaques in ultrasound video. In the tracking task, the present application locates plaques and excludes similar tissues based on a tracking framework optimized by a multiscale hollow encoder and an internal and external feature decoupler. In the echo classification task, the present application proposes a dual-channel 3D-Attention CNN which uses a feature reorganization method and 3D-Attention to comprehensively evaluate the real-time state changes of plaques in the entire video. The framework can relieve the interference of noise in ultrasound images and the variable shape of plaques on the model, and can refine the semantic context and reuse high-level features, and is still applicable in the tracking scene of real-time changes of target semantics. It can be widely used for carotid plaque tracking and classification of various positions and types.

[0146] The present application is described in conjunction with the drawings and specific embodiments in the specification, which are only used to help understand the method and core idea of the present application. The method described in the present application is not limited to the examples described in the specific embodiments, and other embodiments obtained by those skilled in the art according to the method and idea of the present application also belong to the technical innovation range of the present application. The content of the specification should not be understood as a limitation of the present application.

Claims

1. An ultrasound video-based carotid plaque tracking and echogenicity classification method comprising a multiscale hollow encoder, characterized in that: The carotid plaque tracking framework is composed of a multi-scale hollow encoder, an internal and external feature decoupler, and a tracking network, and the carotid plaque echo classification framework is established based on feature recombination and a dual-channel 3D-Attention framework; the ultrasound video image input tracking framework locates the plaque position, and based on feature recombination, the plaque contour is selected as the local feature input of the echo classification framework, and the original ultrasound video image is selected as the global feature input of the echo classification framework; The multi-scale hollow encoder comprises: A1. receiving a current frame, a previous frame and a next frame as input, C2, C3 and C4 represent the output features of ResNet50; A2. C2 and C4 are followed by three residual-like blocks, wherein a 1x1 convolutional layer is applied to reduce the channel dimension, and a 3x3 hollow convolution with different expansion rates is added to refine the semantic context, thereby generating output features with multiple receptive fields; A3. A C3 skip connection is added after the residual-like blocks behind C2 and C4, the subnetwork behind the encoder layer adopts the design paradigm of JDE, and is independently trained at each scale; The internal and external feature decoupler comprises: B1. Given input , where i represents the number of channels, H and W represent the length and width of the feature map, the calculation of the global vector can be represented as: wherein represents a learnable isometric projection; B2. Compute the internal weight vector and the external weight vector separately by feature decoupling :​ where , , and represent four learnable weight matrices, and are sigmoid and relu activation functions, respectively; B3. Finally, the original features are reweighted on each channel: wherein to distinguish plaque status to assist in detection, to distinguish plaque identity to assist in tracking; The tracking network comprises: C1. The tracking network is composed of two parts: a detection subnetwork and a linking subnetwork, the detection subnetwork predicts the position and echo state of the plaque, and the linking subnetwork completes the tracking trajectory according to the correlation between the detection results in each frame; C2. The detection subnetwork is composed of two branches, wherein the contour classification branch class subnet comprises four convolutional layers, each convolutional layer is composed of a 1x1 convolution and a 3x3 convolution; the contour regression branch box subnet has the same structure as the first three layers of the classification branch; C3. The first two layers of the linking subnetwork are composed of 3x3 convolutional layers, and the last layer uses a fully connected layer; C4. The identity-based Embedding feature and the detection box are used to calculate the correlation matrix through cosine similarity, and then the Hungarian algorithm and Kalman filtering are used to match the plaques between different frames and complete the establishment of the tracking trajectory; The feature recombination comprises: D1. For each plaque, the ultrasound section selected by the lesion frame through the tracking framework is selected, the highest-scored plaque contour of each type is selected, in addition, the feature map of the second-to-last layer in the contour classification subnetwork and the linking subnetwork is selected as a supplementary feature, and finally the tracking contour and the supplementary feature are spliced as the local feature of the dual-channel time-series plaque echo classification model; D2. All unselected ultrasound sections are selected as global information, focusing on the changes in the blood vessel contour and the echo attenuation, as the global feature of the dual-channel time-series plaque echo classification model.

2. The method of claim 1, wherein The dual-channel 3D-Attention framework comprises: E1. The global channel and the local channel each contain three convolutional blocks, each of which is composed of two 3D-Conv layers and a global pooling layer; E2. An se-block is added in the global channel to expand in the 3D layer, learn the weight of each frame feature in each channel, and select the feature; E3. A connection is added between the two channels to integrate the information learned together. E4. The information co-learned is followed by a fully connected layer for classification, consisting of two fully connected layers and a layer for plaque echo classification.

Images