Carotid Artery Plaque Echo Classification Method Based on Key Point Detection
By applying key point detection technology and de-redundancy technology in ultrasound imaging, the global and local characteristics of carotid plaques were extracted, and the problem of difficulty in extracting features was solved, and high-precision plaque echo classification was achieved.
Patent Information
- Application Number
- CN202210723775.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-24
- Publication Date
- 2025-05-30
- Estimated Expiration
- 2042-06-24
AI Technical Summary
Small plaque size in ultrasound images leads to difficulty in extracting features, and it is difficult for the prior art to effectively classify the echo types of carotid plaques.
Using a method based on key point detection, the global and local characteristics of plaques are extracted through data preprocessing and key point positioning technology, and combined with de-redundancy technology, a plaque classification subnet is established to realize the classification of carotid plaque echoes.
The accuracy of carotid ultrasound plaque echo classification is improved, and heavy data segmentation and detection frame labeling is avoided, so as to effectively utilize the information of the plaque area and the entire ultrasound map.
Smart Images
Figure CN114943727B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computers, and particularly relates to a method for detecting and classifying the echoes of carotid plaques in medical images. Background Art
[0002] Cardiovascular diseases have become one of the diseases with the highest mortality rates in the world. Carotid atherosclerotic plaque, referred to as carotid plaque for short, is an important factor leading to cardiovascular events. Carotid plaque is produced by the interaction of lipids, extracellular matrix, macrophages, and smooth muscle cells accumulated on the arterial wall. When the plaque ruptures, atherosclerotic thromboembolism composed of platelet aggregates or plaque fragments may enter the brain, block smaller arteries, and cause ischemic attacks or strokes. Plaques that are prone to rupture are called vulnerable plaques or unstable plaques. The stability of the plaque is related to its composition and tissue structure. Plaques rich in calcium and with a smooth surface tend to be stable plaques. On the contrary, plaques rich in lipids and with a rough surface tend to be unstable plaques. Currently, the main means of diagnosing carotid plaques include carotid ultrasound, computed tomography, magnetic resonance imaging, digital subtraction angiography, etc. Ultrasound has become the most widely used carotid examination method due to its characteristics such as convenience, low cost, radiation-free, and non-invasive.
[0003] Doctors can observe the vascular morphology and plaque morphology of patients in real time through carotid ultrasound examination. By simple annotation, doctors can obtain more technical indicators of the plaque, such as the stenosis rate and size of the plaque, and then make a diagnosis. Usually, doctors classify plaques into hyperechoic plaques, mixed echo plaques, and hypoechoic plaques according to the echo type of the plaque. Among them, hyperechoic plaques are generally more stable, hypoechoic plaques are unstable, and the stability of mixed echo plaques is between the two. However, since the proportion of carotid plaques in the carotid ultrasound image relative to the whole image is relatively low, about two percent, and there are difficulties such as irregularly distributed artifacts, speckle noise, low contrast, and local gray level changes, it is still a challenge to use computer technology to classify the echo types of plaques in ultrasound. Summary of the Invention
[0004] The technical problem to be solved by the present invention is the difficulty in extracting plaque features caused by the small size of the plaque in the ultrasound image. The purpose of the present invention is to propose a method for classifying the echoes of carotid plaques based on key point detection by means of key point detection technology, so as to combine the global and local features of the plaque. This model can alleviate the difficulty in extracting plaque features caused by the small size of the plaque in the ultrasound image, and can effectively utilize the effective information of the plaque area and the whole ultrasound image, and can be used for the echo classification of single-plaque carotid ultrasound images.
[0005] To achieve the above object, the concept of the present invention is as follows: First, preprocess the ultrasonic image to remove irrelevant information. Then, use the key point localization technology to determine the approximate position of the plaque. Finally, extract the global and local features of the ultrasonic image regarding the plaque based on the localization and classify the plaque.
[0006] The technical solution of the present invention:
[0007] A carotid plaque echo classification method based on key point detection, which is characterized by including:
[0008] A. Data preprocessing: Crop the non-imaging area of the ultrasonic image and normalize the ultrasonic image;
[0009] B. Generation of localization labels: According to the annotation of the center point of the carotid artery ultrasonic plaque, use a non-standard two-dimensional normal distribution to generate a heat map label for localization;
[0010] C. Establishment of a plaque localization sub-network: Treat a plaque as a point, use the key point localization technology to establish a plaque localization sub-network, and complete the localization of the center point of the carotid plaque;
[0011] D. Establishment of a plaque classification sub-network: Extract the global and local features of the plaque based on the output result of the plaque localization sub-network, and combine with the redundancy removal technology to establish a plaque classification sub-network to complete the carotid plaque echo classification.
[0012] In step B, let D xy represent the value of the localization label at the position (x, y), and the generated localization label can be defined as follows:
[0013]
[0014] where (x 0 , y 0 ) represents the center position of the plaque, and radius represents the penalty radius.
[0015] In step C: The plaque localization sub-network first uses convolutional and pooling layers to convert the ultrasonic image into a feature layer with 128 channels, then downsamples the 128-channel feature layer group three times through residual blocks and pooling layers, and then upsamples and restores the resolution through residual blocks and nearest neighbor interpolation. The features of the downsampling and upsampling layers with the same resolution are connected through residual blocks; finally, the feature channels are reduced to 1 through a convolutional layer, and a carotid plaque localization heat map is output.
[0016] In step D: The plaque classification sub-network is divided into three modules: a multi-branch feature extraction module, an orthogonal fusion module, and a weighted multi-loss module. First, the high-definition detail features of the plaque region and the overall features of the entire ultrasound image are respectively extracted through the multi-branch feature extraction module; then, the orthogonal fusion module eliminates redundant information and performs feature fusion on the detail features and the overall features in an orthogonal manner; finally, the weighted multi-loss module constrains the classification results of the global branch, the local branch, and the fusion branch by means of weighted loss for discriminative training.
[0017] The multi-branch feature extraction module adopts a multi-branch design of a global branch and a local branch. The global branch takes the four-way image after splicing the resized carotid plaque ultrasound image and the positioning heat map of the carotid plaque as input, and the local branch takes the cropped image of the plaque region of the original carotid plaque ultrasound image as input. The two branches have the same structure, both being a four-layer double convolution + pooling structure.
[0018] The orthogonal fusion module takes the output after flattening the last pooling layer of the multi-branch feature extraction module as input, and performs fusion using the splicing method after orthogonal constraint. The orthogonal constraint is implemented by an orthogonal loss (L Orth ), and the specific definition is as follows:
[0019]
[0020] where F 1 = [α 1 , α 1 ,..., α n , F 2 = [β 1 , β 1 ,..., β n are the features after flattening operations before the classification layers of the global and local branches respectively, and θ is the included angle between F 1 and F 2 .
[0021] The weighted multi-loss module consists of a weighted global branch loss, a local branch loss, a fusion branch loss, and an orthogonal loss. The global, local, and fusion branch losses are all implemented using the cross-entropy loss function. The definition of the cross-entropy loss function is as follows:
[0022]
[0023] where i represents the plaque category, y = [y 0 , y 1 , y 2 represents the plaque category label, and p = [p 0 , p 1 , p 2represents the predicted probability of the model, and the total loss of model classification is defined as follows:
[0024] L classification = w 1 L G + w 2 L L + w 3 L F + w 4 L orth
[0025] where w 1 、w 2 、w 3 、w 4 are weights.
[0026] The length of the largest patch in the dataset is L, and the radius of the Gaussian distribution in the localization label is taken as L / 2; w 1 、w 2 、w 3 、w 4 are taken as 0.1, 1, 0.1, and 0.4 respectively.
[0027] The beneficial effects of the present invention are:
[0028] 1. For traditional carotid artery ultrasound plaque echo classification methods, either directly using existing classification models fails to take into account the actual situation that the proportion of the plaque foreground is relatively small, resulting in low classification accuracy, or it is necessary to first segment or detect the plaque, which requires heavy data annotation for model training. The newly proposed carotid artery plaque echo classification method based on key point detection uses key point localization technology to locate the central position of the plaque and combines techniques such as redundancy removal, avoiding the heavy annotation of data segmentation or detection frames, and improving the accuracy of carotid artery ultrasound plaque echo classification
[0029] 2. This method uses multiple branches and multiple losses to fully extract the local features and global features of the plaque.
[0030] 3. This method uses an orthogonal redundancy removal method to effectively reduce the redundancy of local features and global features. Description of the Drawings
[0031] Figure 1 are typical examples of ultrasound images of plaques with three different echo types. a is the ultrasound image of a strongly echogenic plaque, b is the ultrasound image of a mixed echogenic plaque, and c is the ultrasound image of a hypoechoic plaque.
[0032] Figure 2 is the overall framework diagram of the carotid artery plaque echo classification method based on key point detection of the present invention.
[0033] Figure 3It is the single-branch configuration structure diagram of the classification sub-network in the carotid plaque echo classification method based on key point detection of the present invention.
[0034] Figure 4 Lists the selection of w 1 , w 2 , w 3 , w 4 The intermediate experimental results of the optimal values.
[0035] Figure 5 Lists the comparison of the localization performance of the localization sub-network under different Gaussian distribution radii of the localization labels.
[0036] Figure 6 Lists the comparison of the localization performance of the localization sub-network when different hourglass modules are stacked.
[0037] Figure 7 Lists the comparison of the localization performance of the localization sub-network using different numbers of downsampling layers in a single hourglass network.
[0038] Figure 8 Lists the comparison of the experimental performance of different chunk sizes of the input of the local branch in the classification sub-network.
[0039] Figure 9 Lists the ablation experiment results of the multi-branch feature extraction module.
[0040] Figure 10 Lists the ablation experiment results of the orthogonal fusion module.
[0041] Figure 11 Lists the ablation experiment results of the weighted multi-loss module.
[0042] Figure 12 Lists the comparison of the model performance with and without using the localization network.
[0043] Figure 13 Lists the comparison of the performance of the carotid plaque echo classification method based on key point detection of the present invention with seven other popular classification methods. Detailed implementation manners
[0044] Example 1:
[0045] The carotid plaque echo classification method based on key point detection locates the approximate position of the plaque through a key point detection convolutional neural network, and combines the global and local features of the plaque to establish a deep learning model for carotid plaque echo classification with high accuracy. This model can alleviate the difficulty of plaque feature extraction caused by the small size of the plaque in the ultrasound image, and can differentially utilize the effective information of the plaque area and the entire ultrasound image, and can be used for the echo classification of carotid ultrasound images of single plaques. The specific steps are as follows:
[0046] A. Data preprocessing: Crop the non-imaging area of the ultrasound image and normalize the ultrasound image phenomenon;
[0047] B. Localization label generation: According to the doctor's annotation of the center point of the carotid artery ultrasound plaque, use a non-standard two-dimensional normal distribution to generate a heat map label for localization;
[0048] C. Establish a plaque localization sub-network: Treat a plaque as a point, and use key point localization technology to establish a plaque localization sub-network to complete the localization of the center point of the carotid artery plaque;
[0049] D. Establish a plaque classification sub-network: Extract the global and local features of the plaque based on the output results of the plaque localization sub-network, and combine the redundancy removal technology to establish a plaque classification sub-network to complete the echo classification of the carotid artery plaque;
[0050] The step B of generating a heat map label for localization by using a non-standard two-dimensional normal distribution according to the doctor's annotation of the center point of the carotid artery ultrasound plaque is as follows:
[0051] B1. The doctor's annotation of the plaque center is a point coordinate, but the plaque is an irregular area. During training, instead of directly imposing a penalty on the center point of the plaque, the penalty gradually decreases outward within a certain penalty radius centered on the center point of the plaque. The amount of penalty attenuation is given by a non-standard two-dimensional Gaussian distribution. Let D xy represent the value of the localization label at the position (x, y), and the generated localization label can be defined as follows:
[0052]
[0053] where (x 0 , y 0 ) represents the center position of the plaque, and radius represents the penalty radius.
[0054] The step C of treating a plaque as a point and using key point localization technology to establish a plaque localization sub-network is as follows:
[0055] C1. As the first part of the carotid plaque echo classification method based on key point detection, the input of the plaque localization sub-network is the carotid plaque ultrasound image, and the output is the localization heat map of the carotid plaque. The specific implementation of this part of the model refers to the implementation method of the Stacked Hourglass model. However, considering the differences between carotid ultrasound images and natural images, we abandoned the stacking method in the Stacked Hourglass and adopted a single Hourglass module, and reduced the number of downsamplings in the Hourglass module. Specifically, first use convolutional and pooling layers to convert the ultrasound image into a feature layer with 128 channels, then downsample the 128-channel feature layer group three times through residual blocks and pooling layers, and then upsample and restore the resolution through residual blocks and nearest neighbor interpolation. The features of the downsampling and upsampling layers with the same resolution are connected through residual blocks. Finally, the feature channels are reduced to 1 through a convolutional layer, and the carotid plaque localization heat map (H) is output.
[0056] On the basis of the results produced by the plaque localization sub-network in step D, extract the global and local features of the plaque, and establish a plaque classification sub-network. The specific steps are as follows:
[0057] D1. As the second part of the carotid plaque echo classification method based on key point detection, the input of the plaque classification sub-network is the cropped image (Ic) of the plaque area of the original carotid plaque ultrasound image, the resized carotid plaque ultrasound image (Ir), and the localization heat map (H) of the carotid plaque, and the output is the carotid plaque echo category. This architecture divides the model into three modules, namely the multi-branch feature extraction module, the orthogonal fusion module, and the weighted multi-loss module. First, the framework extracts the high-definition detailed features of the plaque area and the overall features of the entire ultrasound image through the multi-branch feature extraction module respectively; then the orthogonal fusion module eliminates redundant information and performs feature fusion on the detailed features and the overall features in an orthogonal manner; finally, the weighted multi-loss module constrains the classification results of the global branch, the local branch, and the fusion branch in a weighted loss manner for discriminative training. Among them:
[0058] D11. The multi-branch feature extraction module adopts a multi-branch design of a global branch and a local branch to solve the problem that it is difficult to extract the high-definition details of the lesion area and the macroscopic information of the entire carotid plaque image in the same network at the same time. The global branch takes the four-way image after splicing the resized carotid plaque ultrasound image and the localization heat map of the carotid plaque as the input, and the local branch takes the cropped image of the plaque area of the original carotid plaque ultrasound image as the input. The two branches have the same structure, both of which are four-layer double convolutional + pooling structures.
[0059] D12. The orthogonal fusion module uses the idea of orthogonality to reduce the redundancy between the global features and the local features. The inputs of both the global branch and the local branch include the patch region images, and the main information for forming the diagnostic judgment comes from the patch region images. Therefore, the generated global features and local features have a high degree of redundancy. To reduce this redundancy, the orthogonal fusion module takes the flattened output of the last pooling layer of the multi-branch feature extraction module as the input, and after orthogonal constraint, it uses the concatenation method for fusion. The orthogonal constraint is implemented by an orthogonal loss (L orth ) and is specifically defined as follows:
[0060]
[0061] where F 1 = [α 1 , α 1 ,..., α n , F 2 = [β 1 , β 1 ,..., β n are the features after flattening operations before the classification layers of the global and local branches respectively, and θ is the included angle between F 1 and F 2 .
[0062] D13. The weighted multi-loss module consists of a weighted global branch loss, a local branch loss, a fusion branch loss, and an orthogonal loss. Due to different input data, the quality of the features generated by the local branch and the global branch is different. Among them, the patch region images should be the main basis for discrimination, and the non-patch region images should be the auxiliary basis for discrimination. When performing constraints, the penalty for local errors should be increased, the penalty for global branch errors should be reduced, and the orthogonal loss should select an appropriate ratio to avoid overshadowing the main features and affecting the extraction of effective features. The global, local, and fusion branch losses are all implemented using the cross-entropy loss function. The cross-entropy loss function is defined as follows:
[0063]
[0064] where i represents the patch category, y = [y 0 , y 1 , y 2 represents the patch category label, and p = [p 0 , p 1 , p 2 represents the prediction probability of the model. The total classification loss of the model is defined as follows:
[0065] L classification = w 1 L G + w 2 L L+w 3 L F +w 4 L orth
[0066] According to the experiment, w 1 、w 2 、w 3 、w 4 take 0.1, 1, 0.1, and 0.4 respectively.
[0067] Example 2: In this example, the specific configuration of the server for running the experiment is as follows: The CPU is an Intel Xeon Gold 6226, the graphics card is two Nvidia RTX 3090 GPUs, and the memory is 128GB. In terms of model establishment, both this multi-modal fusion segmentation framework and the comparison model are implemented based on the open-source deep learning tool PyTorch 1.7.0. In terms of experimental settings, the experiment selects Adam as the optimization algorithm, and the batch size is set to 8. During model training, first freeze the classification sub-network and train the localization sub-network for 100 epochs at a learning rate of 0.001; then freeze the localization sub-network and train the classification sub-network for 50 epochs at a learning rate of 0.0001; finally, fine-tune the entire network for 150 epochs at a learning rate of 0.0001. In terms of localization performance evaluation, in the experiment, the Euclidean distance (distance) between the predicted plaque center point and the plaque center point in the label, the accuracy rate when the Euclidean distance is less than 30, and the accuracy rate when the Euclidean distance is less than 100 are calculated. In terms of classification performance evaluation, in the experiment, through performance indicators such as Accuracy, Precision, Recall, and F1-Score, the larger these indicators are, the better the classification effect of the method.
[0068] The dataset used in this example contains 1,898 longitudinal carotid ultrasound images from 204 patients. The resolution of each image is 540*740, and each image contains only one plaque. Among them, the labels of the image echo types include strong echo, mixed echo, and low echo, and the corresponding numbers of images are 539, 597, and 762 respectively. Typical examples of carotid ultrasound images of each echo type are as Figure 1 shown. The central position of the plaque in each image has been marked by point annotation. The dataset echo type labels and plaque center point labels are all made by experienced professional doctors.
[0069] The method for classifying carotid plaque echoes based on key point detection of the present invention, as Figure 2 shown, includes the following steps:
[0070] A. Data preprocessing: Crop the non-imaging area of the ultrasound image and normalize the ultrasound image phenomenon;
[0071] B. According to the doctor's annotation of the center point of the carotid artery ultrasound plaque, a heatmap label for positioning is generated using a non-standard two-dimensional normal distribution. The specific steps are as follows:
[0072] B1. The doctor's annotation of the plaque center is a point coordinate, but the plaque is an irregular area. During training, instead of directly imposing a penalty on the center point of the plaque, the penalty gradually decreases outward within a certain penalty radius centered on the plaque center point. The amount of penalty attenuation is given by a non-standard two-dimensional Gaussian distribution. Let D xy represent the value of the positioning label at the (x, y) position. The generated positioning label can be defined as follows:
[0073]
[0074] where (x 0 , y 0 ) represents the center position of the plaque, and radius represents the penalty radius.
[0075] C. Establish a plaque localization sub-network: Treat a plaque as a point and use key-point localization technology to establish a plaque localization sub-network. The specific steps are as follows:
[0076] C1. As the first part of the carotid artery plaque echo classification method based on key-point detection, the input of the plaque localization sub-network is the carotid artery plaque ultrasound image, and the output is the localization heatmap of the carotid artery plaque. The specific implementation of this part of the model refers to the implementation method of the Stacked Hourglass model. However, considering the differences between carotid artery ultrasound images and natural images, we discarded the stacking method in the Stacked Hourglass and adopted a single Hourglass module, and reduced the number of downsamplings in the Hourglass module. Specifically, first use convolutional and pooling layers to convert the ultrasound image into a 128-channel feature layer, then downsample this 128-channel feature layer group three times through residual blocks and pooling layers, and then upsample to restore the resolution through residual blocks and nearest neighbor interpolation. The features of the downsampling and upsampling layers with the same resolution are connected through residual blocks. Finally, the feature channels are reduced to 1 through a convolutional layer, and the carotid artery plaque localization heatmap is output.
[0077] D. Establish a plaque classification sub-network: Extract the global and local features of the plaque based on the output results of the plaque localization sub-network, and combine redundancy removal technology to establish a plaque classification sub-network. The single-branch configuration of the classification sub-network is as Figure 3 shown; the specific steps include:
[0078] D1. As the second part of the carotid plaque echo classification method based on key point detection, the input of the plaque classification sub-network is the cropped image of the carotid plaque ultrasound original image plaque area, the resized carotid plaque ultrasound image, and the localization heat map of the carotid plaque, and the output is the carotid plaque echo category. This architecture divides the model into three modules, namely the multi-branch feature extraction module, the orthogonal fusion module, and the weighted multi-loss module. First, the framework extracts the high-definition detailed features of the plaque area and the overall features of the entire ultrasound image through the multi-branch feature extraction module respectively; then the orthogonal fusion module uses the orthogonal method to eliminate redundant information and fuse the detailed features and the overall features; finally, the weighted multi-loss module constrains the classification results of the global branch, the local branch, and the fusion branch by means of weighted loss for discriminative training. Among them:
[0079] D11. The multi-branch feature extraction module adopts a multi-branch design of global branch and local branch to solve the problem that it is difficult to extract the high-definition details of the lesion area and the macroscopic information of the entire carotid plaque image in the same network at the same time. The global branch takes the four-way image after splicing the resized carotid plaque ultrasound image and the localization heat map of the carotid plaque as the input, and the local branch takes the cropped image of the carotid plaque ultrasound original image plaque area as the input. The two branches have the same structure, both of which are four-layer double convolution + pooling structures.
[0080] D12. The orthogonal fusion module uses the idea of orthogonality to reduce the redundancy between the global features and the local features. The inputs of both the global branch and the local branch include the plaque area image, and the main information for forming the diagnostic judgment comes from the plaque area image. Therefore, the generated global features and local features have a high degree of redundancy. To reduce this redundancy, the orthogonal fusion module takes the output after flattening the last pooling layer of the multi-branch feature extraction module as the input, and performs fusion using the splicing method after orthogonal constraint. The orthogonal constraint is implemented by an orthogonal loss (L Orth ), and the specific definition is as follows:
[0081]
[0082] Among them, F 1 = [α 1 , α 1 ,..., α n , F 2 = [β 1 , β 1 ,..., β n are the features after flattening operations before the classification layers of the global and local branches respectively, and θ is the included angle between F 1 , F 2 .
[0083] D13. The weighted multi-loss module consists of a weighted global branch loss (L G ), a local branch loss (L L ), a fusion branch loss (L F ), and an orthogonal loss. Since the input data for the local branch and the global branch are different, the quality of the generated features is different. Among them, the patch region image should be the main basis for discrimination, and the non-patch region image should be the auxiliary basis for discrimination. When performing constraints, more punishment should be imposed on local errors, less punishment should be imposed on global branch errors, and an appropriate ratio should be selected for the orthogonal loss to avoid overshadowing the main features and affecting effective feature extraction. The global, local, and fusion branch losses are all implemented using the cross-entropy loss function. The cross-entropy loss function is defined as follows:
[0084]
[0085] where i represents the patch category, y = [y 0 , y 1 , y 2 represents the patch category label, and p = [p 0 , p 1 , p 2 represents the predicted probability of the model. The total loss of model classification is defined as follows:
[0086] L classification = w 1 L G + w 2 L L + w 3 L F + w 4 L orth
[0087] Using the enumeration method to determine the values of w 1 , w 2 , w 3 , w 4 will bring unbearable experimental overhead, so the method of gradually determining the values of these four weights is adopted. The experimental results are as shown in Figure 4 . First, determine the values of w 1 , w 2 , then determine the value of w 3 , and finally determine the value of w 4 . First, set the values of w 1 , w 2 , w 3 , w 4 to 1 for experiments, and the results of this experiment are used as the comparison benchmark. According to Figure 7From the experimental results, it can be seen that the feature quality of local features is higher than that of global features. Therefore, a larger weight should be given to the loss of the local branch, and a smaller weight should be given to the loss of the global branch. Set w 1 = 0.1, w 2 = 1. And, set w 1 = 1, w 2 = 0.1 for the comparative experiment. The experiment proves the rationality of this setting. To explore the appropriate value of the loss weight of the fusion branch, set w 3 to 1 and 0.1 respectively. When w 3 takes 0.1, the classification accuracy is higher. Fix the values of w 1 , w 2 , w 3 , and set the values of w 4 to 0.2, 0.4, 0.6, and 0.8 respectively for the experiment. The experimental results show that when w 1 = 0.1, w 2 = 1, w 3 = 0.1, w 4 = 0.4, the classification performance of the model is optimal.
[0088] Referring to Figure 5 , it shows the comparison of the localization performance of the localization sub-network under different Gaussian distribution radii of the localization labels. The purpose of this experiment is to explore the optimal value of the Gaussian distribution radius in the optimal localization heatmap label suitable for carotid plaque localization. The radius of the Gaussian distribution in the localization label determines the range of the image that the model focuses on with the plaque as the center. A larger radius means that the model focuses on a larger range in the ultrasound image. If this large range far exceeds the actual size of the plaque, it means that the model focuses on more noise, which will affect the accuracy of the model. And too small a radius will make it difficult for the model to locate the plaque position. We conduct experiments at intervals of L / 4 with the length (L) of the largest plaque in the dataset as the benchmark. The experimental results show that taking L / 2 as the Gaussian radius of the localization label is the best.
[0089] Referring to Figure 6 - Figure 7 , it shows the comparison of the localization performance of the localization sub-network using the hourglass network under different complexities. The purpose of this experiment is to verify that the low-complexity localization sub-network is sufficient to meet the needs of plaque center localization. Therefore, this experiment compares the localization performance of the network when 1 - 3 hourglass modules are stacked. The experimental results are as shown in Figure 6 , and the localization performance of 1 - 4 downsampling layers under the single hourglass network. The experimental results are as shown in Figure 7 . The experimental results show that taking the hourglass module containing three small sampling layers as the localization sub-network model has low complexity and the best effect.
[0090] Referring to Figure 8, which shows the experimental performance comparison of different cropping sizes of the local branch input in the classification sub-network. The purpose of this experiment is to find the best cropping size of the local branch input in the classification sub-network. Based on the length of the longest patch in the dataset, experiments are carried out at intervals of one-fourth of the longest patch length. The experimental results show that the classification performance is the best when the cropping size is 286*286.
[0091] Refer to Figure 9 , which shows the ablation experiment results of the multi-branch feature extraction module. The purpose of this experiment is to prove the rationality of the global branch and local branch settings. This experiment sets up a comparison between a network containing both a global branch and a local branch, a network containing only the global branch, and a network containing only the local branch. The network containing only the global branch is trained in the same way as the network containing both the global branch and the local branch. In the network containing only the local branch, the localization sub-network and the classification sub-network are split into two sub-networks, and the localization sub-network and the classification sub-network are trained separately in the same way as the training method of the proposed model, but in the step of fine-tuning the entire network, it is changed to fine-tuning the classification sub-network. The experimental results show that the network containing both the local part and the global branch is better than the network containing only one of the branches, which proves the necessity of the dual-branch setting.
[0092] Refer to Figure 10 , which shows the ablation experiment results of the orthogonal fusion module. The purpose of this experiment is to prove the necessity of the orthogonal fusion module setting. This experiment compares the performance of models with and without the orthogonal loss. The experimental results show that the orthogonal fusion module improves the carotid artery ultrasound echo classification accuracy by 2.38 percentage points.
[0093] Refer to Figure 11 , which shows the ablation experiment results of the weighted multi-loss module. Since the ablation experiment of the orthogonal module has proved the necessity of the orthogonal loss, the purpose of this experiment is to prove the necessity of setting loss functions for the global branch, local branch, and fusion branch respectively. The output of the fusion branch is used as the final output of the echo classification, so the loss function of the fusion branch cannot be omitted. This experiment compares the performance of the model when not using the global branch loss, not using the local branch loss, and not using the global branch loss. The experimental results show that setting loss functions for the global branch, local branch, and fusion branch respectively is beneficial to plaque echo classification, especially the loss of the local branch plays a crucial role in improving the model performance.
[0094] Refer to Figure 12 , which shows the comparison of the model performance with and without using the localization network. The purpose of this experiment is to prove the necessity of the localization sub-network setting. The experimental results show that using the localization sub-network greatly improves the echo classification performance of the model.
[0095] Refer toFigure 13 , which shows the performance comparison between the carotid plaque echo classification method based on key point detection of the present invention and seven other popular classification methods. The purpose of this experiment is to test the classification performance of the proposed carotid plaque echo classification method based on key point detection. In addition to the method of the present invention, the other popular classification methods compared in the experiment include Alexnet, Resnet50, ResNext50, Densenet169, MobileNet-V2, EfficientNet-b7, and Comformer-S. Among them, Comformer-S is also a specific implementation of the global and local design concept. All the comparison methods are pre-trained on the ImageNet dataset and then fine-tuned on the plaque dataset in this paper. The model of the present invention is trained from scratch on the plaque dataset. The experimental results show that the proposed carotid echo classification method in this paper is superior to the popular classification methods in all performance indicators.
[0096] In summary, the carotid plaque echo classification method based on key point detection of the present invention uses the key point detection technology to combine the global and local features of the plaque, and proposes a carotid plaque echo classification method based on key point detection. This model can alleviate the difficulty of plaque feature extraction caused by the small size of the plaque in the ultrasound image, and can differentially utilize the effective information of the plaque area and the whole ultrasound image, and can be used for the echo classification of the carotid ultrasound image of a single plaque.
[0097] This paper combines the description of the accompanying drawings of the specification and specific embodiments only to help understand the method and core idea of the present invention. The method described in the present invention is not limited to the embodiments described in the specific implementation manner. Other implementation manners obtained by those skilled in the art based on the method and idea of the present invention also belong to the scope of the technical innovation of the present invention. The content of this specification should not be construed as a limitation to the present invention.
Claims
1. A method for classifying carotid plaque echoes based on key point detection, characterized in that it includes: A. Data preprocessing: Crop the non-imaging area of the ultrasound image and normalize the ultrasound image phenomenon; B. Generation of localization labels: According to the annotation of the center point of the carotid artery ultrasound plaque, use a non-standard two-dimensional normal distribution to generate heat map labels for localization; C. Establish a plaque localization sub-network: Regard a plaque as a point, use key point localization technology to establish a plaque localization sub-network to complete the localization of the center point of the carotid plaque; The plaque localization sub-network first uses convolutional and pooling layers to convert the ultrasound image into a 128-channel feature layer, and then downsample this 128-channel feature layer group three times through residual blocks and pooling layers, and then upsample and restore the resolution through residual blocks and nearest neighbor interpolation. The features of the downsampling and upsampling layers with the same resolution are connected through residual blocks; Finally, the feature channels are reduced to 1 through a convolutional layer, and the carotid plaque localization heat map is output; D. Establish a plaque classification sub-network: Extract the global and local features of the plaque based on the output of the plaque localization sub-network, and establish a plaque classification sub-network by combining redundancy reduction techniques to complete the classification of carotid plaque echoes; The plaque classification sub-network is divided into three modules: a multi-branch feature extraction module, an orthogonal fusion module, and a weighted multi-loss module. First, the high-definition detail features of the plaque region and the overall features of the entire ultrasound image are extracted through the multi-branch feature extraction module respectively; Then, the orthogonal fusion module eliminates redundant information and performs feature fusion on the detail features and the overall features in an orthogonal manner; Finally, the weighted multi-loss module constrains the classification results of the global branch, the local branch, and the fusion branch by means of weighted loss for discriminative training. The orthogonal fusion module takes the flattened output of the last pooling layer of the multi-branch feature extraction module as the input, and performs fusion in a splicing manner after orthogonal constraint. The orthogonal constraint is implemented by an orthogonal loss which is defined as follows: wherein are the features after the flattening operation before the global and local branch classification layers respectively, is the included angle of.
2. The method for classifying carotid plaque echoes based on key point detection according to claim 1, characterized in that In step B, using to represent the value of the positioning tag at the (x, y) position, the generated positioning tag can be defined as follows: wherein ( represents the central position of the plaque, represents the penalty radius.
3. The method for classifying carotid plaque echoes based on key point detection according to claim 1, characterized in that: The multi-branch feature extraction module adopts a multi-branch design of a global branch and a local branch. The global branch takes the four-channel image after splicing the resized carotid plaque ultrasound image and the localization heat map of the carotid plaque as input, and the local branch takes the cropped image of the original image plaque area of the carotid plaque as input. The two branches have the same structure, both of which are four-layer double convolution + pooling structures.
4. The method for classifying carotid plaque echoes based on key point detection according to claim 1, characterized in that: The weighted multi-loss module consists of a weighted global branch loss, a local branch loss, a fusion branch loss, and an orthogonal loss. The global, local, and fusion branch losses are all implemented using the cross-entropy loss function. The cross-entropy loss function is defined as follows: where \(i\) represents the plaque category, and \(y = \) represents the plaque category label, represents the predicted probability of the model, and the total loss of model classification is defined as follows: wherein is the weight.
5. The method for classifying carotid plaque echoes based on key point detection according to claim 2, characterized in that: The length of the largest plaque in the dataset is L, and the radius of the Gaussian distribution in the localization label is taken as L / 2.
6. The method for classifying carotid plaque echoes based on key point detection according to claim 5, characterized in that: Take 0.1, 1, 0.1, and 0.4 respectively.