A system and method for automatically detecting plaques in carotid artery ultrasound images

Abstract

The present invention discloses a system and method for automatically detecting plaques in carotid ultrasound images. The system comprises a feature extraction module for extracting high-level visual features from carotid ultrasound images; a target region generation module for finding a target region within the overall carotid artery region or the carotid blood flow region based on the high-level visual features and generating a target frame; a target frame position correction module for correcting positional deviations of the target frame within the high-level visual feature map; and a contour fitting module for performing contour fitting to obtain the overall carotid artery region and the carotid blood flow region. This system overcomes the problems of traditional carotid plaque detection, such as the high number of medical personnel required, low examination efficiency, and significant subjective influence.

Description

Technical Field

[0001] The present invention relates to the field of medical image segmentation, and in particular to a system and method for automatically detecting plaques in carotid artery ultrasound images. Background Art

[0002] Plaque at the carotid bifurcation is a major source of thrombosis and cerebral embolism. Clinically, carotid plaque is assessed by measuring the degree of carotid stenosis and plaque morphology to determine its harmfulness. Noninvasive and localized quantification of carotid plaque is crucial for monitoring disease progression and response to treatment in high-risk patients. Ultrasound imaging of carotid plaque plays an important role due to its noninvasive nature, low cost, short examination time, and widespread availability, as well as the valuable information it provides on plaque composition and stability based on ultrasound image features.

[0003] Currently, in clinical practice, whether there is a plaque in the carotid artery is mainly determined by experienced observers or medical experts through non-invasive B-mode ultrasound examinations. If a plaque is present, they are required to manually outline the plaque boundary. After the boundary is marked, the shape and texture of the plaque are measured and evaluated, and the severity and harmfulness of the carotid plaque are measured, and subsequent corresponding treatment plans are formulated. However, it is very time-consuming for observers to learn and practice manually outlining the boundaries of carotid plaques, and the variability of measurements depends on the experience of the observer. As a result, this method has some problems and cannot meet current clinical needs: (1) The number of patients with carotid artery disease is large and the distribution is relatively dense, which requires a huge cost of medical staff; (2) The examination time for each patient is long and the efficiency is low; (3) It is subject to significant subjective influence, resulting in misjudgment during the high-intensity examination tasks in a day.

[0004] Ding Mingyue et al. pointed out the urgent need for rapid and automatic segmentation of ultrasonic carotid plaques in clinical practice in their invention patent CN102800088 B, titled "Ultrasonic Carotid Plaque Automatic Segmentation Method." However, the invention uses traditional digital image processing methods, which are relatively backward in the current era of artificial intelligence and require certain empirical interventions.

[0005] With the gradual maturity of artificial intelligence technology, computer-assisted carotid plaque segmentation methods can reduce the subjectivity and segmentation time of manual methods without the need to train observers, attracting a large number of researchers to use artificial intelligence technology to detect carotid plaques in ultrasound images. For example, Groves LA et al., Automatic segmentation of the carotid artery and internal jugular vein from 2D ultrasound images for 3D vascular reconstruction. International journal of computer assisted radiology and surgery, 2020, 15(11): 1835-1846. used the Mask RCNN algorithm and the U-Net algorithm to detect and segment the carotid artery and internal jugular vein, respectively, but did not detect and segment the plaque area within the cross-section of the carotid artery. Moreover, since the algorithm is based on the principle of pixel classification, the large number of pixels in a single ultrasound image causes the algorithm to consume a long time; Zhou R et al., Deep learning-based carotid media-adventitia and lumen-intima boundary segmentation from three-dimensionalultrasound images.Medical Physics, 2019, 46(7): 3180-3193. In this study, a dynamic convolutional neural network was used to segment carotid artery cross-sectional ultrasound images and detect the lumen-intima boundary and media-adventitia boundary. However, the algorithm required the observer to manually set 4 to 8 anchor points to initialize the contour and limit the search area of the sliding window algorithm. This algorithm is a semi-automatic detection method because it requires manual marking. In addition, the detection of a single carotid artery ultrasound image is also based on the principle of pixel classification, which takes an average of 34.4±9.8s and is also time-consuming. In summary, previous methods for detecting carotid plaques based on ultrasound images are all based on the principle of pixel classification, which is time-consuming and requires manual marking to determine the initial contour, and are mostly semi-automatic.

[0006] However, recently, Peng S et al. (Deep snake for real-time instance segmentation, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020: 8533-8542) first proposed the Deep Snake algorithm. Based on the principle of contour fitting, this algorithm introduces circular convolution for effective feature learning of contours. Tested on datasets such as Cityscapes, Kins, Sbd, and COCO, it achieved excellent results in real-time segmentation. However, to date, no one has applied the Deep Snake algorithm to plaque detection in carotid ultrasound images. Summary of the Invention

[0007] In order to overcome the problems of traditional carotid artery plaque detection requiring a large number of medical personnel, low inspection efficiency, and significant subjective influence, the present invention provides a system and method for automatic detection of carotid artery plaques using ultrasound images.

[0008] The purpose of the present invention is achieved by the following technical solutions:

[0009] An automatic detection system for carotid artery ultrasound plaques, comprising:

[0010] A feature extraction module for extracting high-level visual features of carotid artery ultrasound images;

[0011] a target region generating module, configured to search for a target region of the overall carotid artery region or the carotid artery blood flow region according to the high-level visual features, and generate a target frame;

[0012] The target frame position correction module is used to correct the position deviation of the target frame on the high-level visual feature map;

[0013] The contour fitting module is used to perform contour fitting to obtain the overall carotid artery area and the carotid artery blood flow area; the difference between the overall carotid artery area and the carotid artery blood flow area is used to obtain the carotid artery plaque area.

[0014] As a further improvement of the present invention, in the feature extraction, ResNet is used to extract a high-level visual feature map of the carotid artery ultrasound image, including:

[0015] ResNet is used to batch normalize the data, including:

[0016] 1.1) Find the mean of each training batch of data:

[0017]

[0018] Among them, x i represents the i-th sample in a batch, and n represents the number of samples in a batch;

[0019] 1.2) Find the variance of each training batch of data:

[0020]

[0021] 1.3) Data standardization:

[0022]

[0023] Where x′ i Represents x i As a result of standardization, ε is a small positive number;

[0024] 1.4) Training parameters γ, β:

[0025] Among them, the scale factor γ and the translation factor β are learned by back propagation during training;

[0026] 1.5) The output y is converted to its original value through a linear transformation of γ and β. The linear transformation method is:

[0027] y i =γ*x′ i +β

[0028] Construct a residual network block structure, including:

[0029] Linear activation is performed as follows:

[0030] z[l+1]=W[l+1]*a[l]+*b[l+1]

[0031] Among them, a[l] is the input, W[l+1] is the weight matrix, and b[l+1] is the bias factor;

[0032] ReLU nonlinear activation is performed according to this formula:

[0033] a[l+1]=g(z[l+1])

[0034] Where g() is the ReLU function;

[0035] After activation, the residual network block structure is obtained;

[0036] Extracting high-level visual features of carotid artery ultrasound images via ResNet.

[0037] As a further improvement of the present invention, in the target region generation module, the target frame is generated based on the DeepSnake carotid artery plaque detection algorithm, and the region generation network is used to perform a target frame search on the high-level visual feature map obtained by ResNet to generate the optimal target frame where the overall carotid artery region and blood flow region are located; the module comprises the following steps:

[0038] Use sliding windows of different sizes, lengths and widths to sequentially scan the high-level visual feature maps extracted by ResNet;

[0039] The region generation network uses the softmax activation function to classify each sliding window to determine whether the sliding window maximizes the high-level visual features of the overall carotid artery area or blood flow area. If so, the offset of the sliding window is calculated;

[0040] The Proposals layer of the region generation network obtains the sliding window and offset containing the overall carotid artery area or blood flow area features, and removes the sliding window that does not meet the feature map boundary;

[0041] On the extracted high-level visual feature map, the optimal target box containing the overall area of the carotid artery or the blood flow area features is obtained.

[0042] As a further improvement of the present invention, in the target frame position correction module, the target frame position correction adopts the ROIAlign method, and bilinear interpolation is performed according to the length and width two-dimensional coordinates of the image to correct the position deviation of the target frame on the original carotid artery ultrasound image; the ROI Align method includes:

[0043] Traverse each candidate region and keep the floating point boundaries unquantized;

[0044] The candidate region is divided into k*k units, and the boundary of each unit is not quantized;

[0045] Multiple fixed coordinate positions are calculated in each unit, and the values of multiple positions are calculated using the bilinear interpolation method. Then, a maximum pooling operation is performed to correct the position deviation of the target box on the original carotid artery ultrasound image.

[0046] As a further improvement of the present invention, in the contour fitting module, a circular convolution method is used to extract high-level visual features of the target area contour, and the circular convolution method includes:

[0047]

[0048] Among them, k is the convolution kernel function and * is the convolution operator symbol.

[0049] As a further improvement of the present invention, in the contour fitting module, contour fitting is performed through a Deep Snake network model, and the contour fitting includes:

[0050] The optimal target frame obtained in the target frame position correction module is used as the initial outline of the target area;

[0051] Take the midpoints of each side of the target box and connect them to get a diamond outline;

[0052] The four vertices of the diamond outline are displaced to the four extreme points of the target area to obtain the extreme points of the target area outline;

[0053] For the left and right extremes, extend the distance of one-quarter of the edge of the previous rectangular target box upward and downward respectively; if it extends to the edge of the target box, it is truncated; for the upper and lower extremes, extend the distance of one-quarter of the edge to the left and right respectively, and truncate it if it encounters the edge of the target box; finally, the octagonal outline is obtained;

[0054] Select N nodes in the octagonal contour to represent the target area contour, and calculate the sum of the offset losses of all nodes during each contour fitting;

[0055] Continuously iterate and update to obtain the final fitting contour of the target area.

[0056] As a further improvement of the present invention, the data preprocessing part of the Deep Snake network model preliminarily transforms the square target box into an octagonal outline, and uses the octagonal outline as the input of the outline fitting network model; the DeepSnake network model includes:

[0057] an extraction module, configured to extract high-level visual features of the octagonal outline through circular convolution;

[0058] A fusion module, configured to fuse the target regions if multiple target regions of the same category, close to each other and disconnected are detected;

[0059] The prediction module is used to fit the true contour by predicting the offset of each node in the contour.

[0060] As a further improvement of the present invention, during the contour fitting process, the following loss function is used to calculate the offset loss of all nodes in the contour:

[0061]

[0062] in, is a node in the predicted target area contour line, It is a node in the actual target area contour line.

[0063] As a further improvement of the present invention, in the process of moving to the four extreme points of the target area, the following loss function is used to calculate the target box offset loss:

[0064]

[0065] in, are the four extreme points of the predicted target area, are the four extreme points of the real target area.

[0066] A method for automatically detecting plaques in carotid artery ultrasound images, comprising:

[0067] Extracting high-level visual features from carotid artery ultrasound images;

[0068] Searching for a target area of the overall carotid artery area or the carotid artery blood flow area according to the high-level visual features, and generating a target frame;

[0069] Correct the position deviation of the target box on the high-level visual feature map;

[0070] Contour fitting is performed to obtain the overall carotid artery area and the carotid artery blood flow area; the carotid artery plaque area is obtained by subtracting the overall carotid artery area from the carotid artery blood flow area.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] The technical solution of the present invention has the following beneficial effects:

[0073] This invention utilizes computers to assist doctors in diagnosing carotid artery plaques, significantly reducing their workload. With the help of computer-assisted diagnosis, doctors can interpret the computer's results, significantly reducing their workload. Furthermore, if sufficient ultrasound image data of carotid artery plaques from a sufficient number of patients at different time periods is available, the algorithm can be used to approximate the growth trend of each patient's carotid artery plaque, further assisting doctors in their diagnosis. This invention utilizes a computer to output automated carotid artery plaque detection results, which are unaffected by subjective human input and can effectively reduce the likelihood of misjudgments made by doctors under intense workloads.

[0074] Furthermore, the present invention applies the Deep Snake algorithm, a field of artificial intelligence, to carotid artery ultrasound image plaque detection, significantly improving the efficiency of carotid plaque detection. This method employs a contour-fitting approach, selecting a few nodes (40 nodes in this case) to represent contour lines, rather than relying on the classification of thousands of pixels to detect carotid plaques. This significantly improves the efficiency of carotid plaque detection.

[0075] Furthermore, the present invention tested and compared this method with mainstream deep learning algorithms (such as Mask RCNN and U-Net) under the same hardware environment and the same data set. The method requires very little computing time to detect carotid artery plaques and can basically achieve real-time detection. If applied in clinical practice, it can greatly improve the doctor's diagnostic experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0077] Figure 1 This is a diagram illustrating the basic concept of the method of the present invention, wherein 101 is the overall area of the carotid artery, 102 is the carotid artery blood flow area, and 103 is the carotid artery plaque area. It is particularly noted that 102 and 103 can be represented by different colors;

[0078] Figure 2 is a flow chart of the method of the present invention;

[0079] Figure 3 301 is a feature extraction module, 302 is a target region generation module, 303 is a target frame position correction module, and 304 is a contour fitting module.

[0080] Figure 4 Schematic diagram of the circular convolution operation process;

[0081] Figure 5 Schematic diagram of the specific process of contour fitting for the overall area of the carotid artery;

[0082] Figure 6 This is a schematic diagram of the specific process of contour fitting of the carotid artery blood flow area;

[0083] Figure 7 The Deep Snake network architecture diagram of the present invention, wherein 701 is the feature extraction module, 702 is the target fusion module, and 703 is the prediction module;

[0084] Figure 8 Schematic diagram of the data annotation process in a specific example of the present invention;

[0085] Figure 9 This is a diagram showing the carotid artery plaque detection results of the present invention;

[0086] Figure 10Schematic diagram of the concepts of true positive, true negative, false positive, and false negative, where TP is true positive, TN is true negative, FP is false positive, and FN is false negative. DETAILED DESCRIPTION

[0087] In order to make the purpose and technical solution of the present invention clearer and easier to understand, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0088] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0089] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0090] The technical solutions of the present invention will be described clearly and completely below with reference to the accompanying drawings and specific embodiments. It is obvious that the embodiments described are only a portion of the embodiments of the present invention, rather than all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are considered to fall within the scope of the present invention.

[0091] This paper attempts to apply the Deep Snake algorithm to carotid artery plaque detection and finds that it can detect a single carotid artery ultrasound image in just 0.05 seconds, a remarkably fast speed that essentially achieves real-time detection in ultrasound equipment while maintaining a reasonable level of accuracy. Furthermore, this method eliminates the need for manual marking and is a fully automated method for carotid artery plaque detection.

[0092] Given the irregular shape of carotid plaques, direct detection of carotid plaques requires greater computational effort and increased time to achieve high accuracy. Conversely, the overall cross-sectional area of the carotid artery is a relatively simple, circular region, making it easier to detect. Furthermore, the carotid blood flow region displays a lower pixel value on ultrasound images. Furthermore, color Doppler ultrasound can be used to detect carotid blood flow, making it equally easy to detect.

[0093] The present invention provides a method for automatic detection of plaques in carotid artery ultrasound images. The specific concept is as follows: first, the overall area of the carotid artery cross-section is detected using a Deep Snake-based algorithm model; then, the carotid artery blood flow area is detected inward using a Deep Snake-based algorithm model; finally, the carotid artery cross-section plaque area is indirectly obtained by subtracting the blood flow area from the detected overall carotid artery area.

[0094] The Deep Snake algorithm model is used to detect the overall cross-sectional area or blood flow area of the carotid artery, which includes the following steps:

[0095] (1) Extract high-level visual features of carotid artery ultrasound images;

[0096] (2) Find the most likely area of the carotid artery or the carotid artery blood flow area based on the extracted high-level visual features, and frame the optimal target area with a rectangular frame, which is referred to as the target frame;

[0097] (3) Correct the position deviation of the target box on the high-level visual feature map;

[0098] (4) Perform contour fitting to obtain the overall area of the carotid artery and the carotid artery blood flow area.

[0099] The present invention provides a method for automatically detecting plaques in carotid artery ultrasound images, wherein the functional modules thereof include a feature extraction module, a target area generation module, a target frame position correction module, and a contour fitting module.

[0100] The feature extraction module includes data batch normalization and residual network block structure, which aims to speed up network training, improve network generalization ability, and solve the problem of gradient explosion or gradient disappearance during the back propagation process of deep convolutional neural network.

[0101] The target region generation module includes sliding windows of different proportions, classifiers, offset calculation and output target region, with the aim of improving the overall computational efficiency of the network and obtaining a target frame containing the overall area of the carotid artery or the blood flow area characteristics.

[0102] The target frame position correction module adopts the ROI Align method to correct the inevitable position deviation of the generated target frame on the high-level visual feature map.

[0103] The contour fitting module includes contour feature extraction, target area fusion, and contour prediction, with the aim of improving computational efficiency and obtaining a true contour that is closer to the target.

[0104] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0105] The present invention provides a method for automatically detecting plaques in carotid artery ultrasound images. Figure 1 ,include:

[0106] First, the entire cross-sectional area of the carotid artery in the ultrasound image is roughly detected and framed with a rectangular frame;

[0107] Then, the contour is fitted inward according to the rectangular frame to obtain a finer division of the entire carotid artery cross section area;

[0108] Secondly, the contour is fitted inward to obtain a more finely divided carotid artery cross-sectional blood flow area; the difference between the two areas is the carotid artery cross-sectional plaque area.

[0109] To detect the entire cross-sectional area or blood flow area of the carotid artery, it is necessary to use the DeepSnake-based carotid artery ultrasound image plaque automatic detection system provided by the present invention, and its various functional modules are as shown in the attached figure. Figure 3 ,include:

[0110] Feature extraction module 301 , target region generation module 302 , target frame position correction module 303 , and contour fitting module 304 .

[0111] In the block diagram, "Input" is the input original carotid artery cross-sectional ultrasound image, "ResNet 101" is a 101-layer residual deep neural network, "Conv" is the convolution operation, "softmax" is the activation function, "Proposals" is used to obtain the target region box, "ROI Align" is used to correct the position deviation of the target box, and "Deep Snake" is an algorithm for contour fitting.

[0112] The functional modules provided by the present invention are as follows:

[0113] Feature extraction module 301

[0114] For the feature extraction part, the high-level visual feature map of the carotid artery ultrasound image is extracted by using ResNet 101, which includes the following steps:

[0115] 1) Batch Normalization:

[0116] ResNet performs batch normalization on data, normalizing it not only before input but also after each transformation (such as convolution). Batch normalization is equivalent to zero-meaning, allowing each round of network training to learn relatively uniform feature information without data drift by fine-tuning the batch data only, improving network training efficiency. Furthermore, batch normalization can restore a relatively uniform distribution of data features, improving network generalization.

[0117] The specific steps of batch normalization are as follows:

[0118] (1.1) Find the mean of each training batch of data.

[0119]

[0120] Among them, x i represents the i-th sample in a batch, and n represents the number of samples in a batch.

[0121] (1.2) Find the variance of each training batch of data.

[0122]

[0123] (1.3) The data is standardized.

[0124]

[0125] Where x′ i Represents x i As a result of normalization, ε is a small positive number.

[0126] (1.4) Training parameters γ, β.

[0127] Among them, γ is the scale factor and β is the translation factor, both of which can be learned by back propagation during training.

[0128] (1.5) The output y is converted to its original value through linear transformation of γ and β.

[0129] y i =γ*x′ i +

[0130] 2) Residual network block structure:

[0131] ResNet mainly uses a residual network block structure. By adding two high-level visual feature map matrices with the same shape structure and a specific number of layers apart, it solves the problem of gradual degradation of deep convolutional neural networks as the number of training layers increases. In other words, it solves the problem of gradient explosion or gradient vanishing during the back propagation process of deep convolutional neural networks. It includes the following steps:

[0132] 1) Linear activation

[0133] Linear activation is performed according to this formula:

[0134] z[l+1]=W[l+1]*a[l]+*b[l+1]

[0135] Among them, a[l] is the input, W[l+1] is the weight matrix, and b[l+1] is the bias factor.

[0136] 2) ReLU nonlinear activation

[0137] ReLU nonlinear activation is performed according to this formula:

[0138] a[l+1]=g(z[l+1])

[0139] Where g() is the ReLU function.

[0140] Target area generation module 302

[0141] For target frame generation, the Deep Snake-based carotid plaque detection algorithm uses a region generation network to perform a target frame search on the high-level visual feature map obtained by ResNet, thereby generating the optimal target frame for the overall carotid artery area and blood flow area. The following steps are included:

[0142] 1) Nine sliding windows of different sizes (0.5, 1, and 2 times) and different lengths and widths are used to sequentially scan the high-level visual feature maps extracted by ResNet;

[0143] 2) The region generation network uses the softmax activation function to classify each sliding window and determine whether the sliding window maximizes the high-level visual features of the overall carotid artery area or blood flow area. If so, the offset of the sliding window is calculated;

[0144] 3) The Proposals layer at the end of the region generation network structure integrates the sliding windows and offsets containing the overall carotid artery area or blood flow area features, and removes sliding windows with small windows or those that exceed the boundary of the feature map.

[0145] In this way, the optimal target box containing the overall area of the carotid artery or the blood flow area features is obtained on the extracted high-level visual feature map.

[0146] Target frame position correction module 303

[0147] Since there is a certain degree of image compression in the process of obtaining a high-level visual feature map through feature extraction from the original carotid artery ultrasound image, if the length and width of the high-level visual feature map obtained after feature extraction are not integers, and the target area generation module only retains the integer part of the length and width as the target box position coordinates, then when the obtained high-level visual feature results are mapped back to the original carotid artery ultrasound image, the target box will have a non-negligible position deviation.

[0148] For the target frame position correction part, the present invention adopts the ROI Align method to perform bilinear interpolation based on the length and width two-dimensional coordinates of the image, thereby correcting the position deviation of the target frame on the original carotid artery ultrasound image and further improving the accuracy of the detection model. The specific steps are as follows:

[0149] (1) Traverse each candidate region and keep the floating point boundary without quantization;

[0150] (2) Divide the candidate region into k*k units, and do not quantize the boundaries of each unit;

[0151] (3) Calculate four fixed coordinate positions in each unit, use bilinear interpolation to calculate the values of these four positions, and then perform maximum pooling operation.

[0152] Contour fitting module 304

[0153] After obtaining the optimal target frame area, based on the idea of contour fitting, it is necessary to extract the features of the contour. However, ordinary two-dimensional convolution operations cannot accurately extract the high-level visual features of the contour. Considering that the boundary contours of the carotid artery overall area and the carotid artery blood flow area are both closed loops, the present invention uses the circular convolution method to extract the high-level visual features of the target area contour. Figure 4 , where the bottom-level nodes are the input high-level visual feature maps, that is, the high-level visual feature maps of the optimal target frame area obtained by the above-mentioned feature extraction and target frame generation network parts; the middle-level nodes are the convolution kernels of the circular convolution; and the top-level nodes are the outputs obtained after the convolution of the bottom-level nodes and the middle-level nodes.

[0154] The entire process of circular convolution can be summarized as:

[0155]

[0156] Among them, k is the convolution kernel function and * is the convolution operator symbol.

[0157] For the contour fitting part, the present invention uses the Deep Snake network model to perform contour fitting, and the specific process of contour fitting of the overall area of the carotid artery is as shown in the attached figure. Figure 5 The specific process of contour fitting of the carotid artery blood flow area is as shown in the attached Figure 6 The specific steps of the contour fitting provided by the present invention are as follows:

[0158] (1) The optimal target frame obtained in the target frame position correction module is used as the initial outline of the target area;

[0159] (2) Take the midpoints of each side of the target box and connect them to obtain a diamond outline;

[0160] (3) The four vertices of the diamond outline are displaced to the four extreme points of the target area, namely the top point, the bottom point, the leftmost point and the rightmost point of the target area, which are collectively referred to as the extreme points of the target area outline;

[0161] In the process of moving to the four extreme points of the target area, the present invention uses the following loss function to calculate the target box offset loss:

[0162]

[0163] in, are the four extreme points of the predicted target area, are the four extreme points of the real target area.

[0164] (4) For the left and right extreme points, start from the point and extend upward and downward by a quarter of the distance of the previous rectangular target box edge. If it reaches the edge of the target box, it will be truncated. Similarly, for the upper and lower extreme points, extend to the left and right by a quarter of the distance of the edge. If it reaches the edge of the target box, it will be truncated. Finally, the octagonal outline is obtained;

[0165] Steps (1)-(4) are the data preprocessing part of the Deep Snake network model. The square target box is initially converted into an octagonal outline, and this octagonal outline is used as the input of the contour fitting network model. This can greatly reduce the calculation amount of position offset loss in the subsequent contour fitting process of the network model and improve the computational efficiency.

[0166] The Deep Snake network model consists of three modules. Figure 7 , where "Contour" is the octagonal outline mentioned above, "Feature Extraction Module" is the backbone module of this network, "CirConv-Bn-ReLU" is "Circular Convolution-Batch Normalization-Activation Function", "Add" is the addition of feature maps, and "Concat" is the increase in the number of channels. The specific steps are as follows:

[0167] First, the feature extraction part 701 aims to extract high-level visual features of the contour through circular convolution;

[0168] Then, target region fusion 702 is performed. If multiple target regions of the same category are detected that are close to each other but not connected, these target regions are fused together. For example, if a target is occluded by another object, causing the target region of the image to be divided into multiple disconnected parts, target region fusion will merge them together to form a single target.

[0169] Finally, the prediction part 703 makes the prediction result closer to the actual contour of the target area by predicting the offset of each node in the contour.

[0170] (5) Select N nodes from the octagonal contour obtained in step (4) to represent the contour of the target area, and calculate the sum of the offset losses of all nodes during each contour fitting;

[0171] During the contour fitting process, the present invention uses the following loss function to calculate the offset loss of all nodes in the contour:

[0172]

[0173] in, is a node in the predicted target area contour line, is a node in the real target area contour line. In the present invention, the target area contour line is set to be replaced by 40 nodes, that is, N=40.

[0174] This is because if there are too many nodes representing the contour, the algorithm becomes computationally complex and may result in overfitting. On the other hand, if there are too few nodes representing the contour, the contour cannot be well represented, the fitting effect is poor, and underfitting may occur. In actual experiments, the present invention found that 40 nodes can well represent the contour of the overall carotid artery area or the contour of the carotid artery blood flow area. Therefore, in the present invention, N is set to 40.

[0175] (6) Continuously iterate and update to obtain the final fitting contour of the target area.

[0176] Specific examples are as follows:

[0177] According to the method for automatic detection of plaques in carotid ultrasound images provided by the present invention, a carotid plaque detection experiment was conducted on 26 subjects, all aged between 20 and 60, including 16 healthy subjects and 10 patients with various carotid plaque conditions. Ten ultrasound physicians collected carotid ultrasound video data from these subjects, and data augmentation was performed to generate 7,060 carotid ultrasound images. During data preprocessing, this carotid ultrasound image dataset was shuffled and randomly divided into training, validation, and test sets in an 8:1:1 ratio.

[0178] For the labeling data, the present invention uses Labelme software for labeling. The labeling process is shown in the attached Figure 8 To ensure the accuracy and objectivity of the annotations, each carotid ultrasound image was reviewed by two other annotators after being annotated. Finally, ultrasound doctors reviewed the data to ensure that it was correctly annotated.

[0179] Among them, according to the present invention, a specific example is tested to obtain the carotid artery plaque detection result diagram, see the attached Figure 9 .

[0180] The evaluation criteria used in this invention include accuracy, precision, recall, intersection of overlap (IoU), Dice similarity coefficient (DSC), detection time, etc. The calculation rules are as follows:

[0181]

[0182]

[0183]

[0184]

[0185]

[0186] Among them, true positive (TP) is the number of pixels in the target area predicted by the algorithm model as positive; true negative (TN) is the number of pixels in the background area predicted as negative; false positive (FP) is the number of pixels in the background area predicted as positive; false negative (FN) is the number of pixels in the target area predicted as negative. Figure 10 .

[0187] In order to evaluate the advantages and disadvantages of the automatic detection method for carotid artery ultrasound image plaques provided by the present invention, the carotid artery ultrasound image detection results obtained by the present invention are compared with the carotid artery ultrasound image detection results obtained by the current mainstream deep learning algorithms (such as Mask RCNN and U-Net) under the same hardware environment (CPU: Intel Core i7 / 10700; GPU: NVIDIA RTX3060 (12G video memory)) tested on the same data set. The comparison method adopts the above-mentioned evaluation criteria. The comparison results are detailed in Table 1.

[0188] Table 1 Comparison results between the present invention and mainstream algorithms

[0189]

[0190] The following results are obtained:

[0191] (1) When the present invention detects carotid artery plaques, the DSC index is slightly lower than that of the carotid artery plaque detection algorithm based on Mask RCNN or U-Net.

[0192] Because when the data set was annotated, the overall carotid artery area and the blood flow area were annotated separately by different people, which caused errors in the annotation of the regional boundaries. This led to slight errors between the true values annotated in the data set and the actual results. That is, the results detected by the present invention were compared with the true values annotated in the data set, not with the actual results, resulting in slight errors, which in turn led to a slightly lower DSC of the results output by the present invention. However, the present invention is closer to the detection results of the actual plaque area and has a certain ability to avoid the influence of this error.

[0193] (2) When detecting carotid artery plaques, the present invention is significantly faster than the carotid artery plaque detection algorithm based on Mask RCNN or U-Net.

[0194] The present invention is relatively efficient based on the basic idea of contour fitting. It only takes an average of 0.05 seconds to process a carotid artery ultrasound image, and the calculation speed is relatively fast. Considering that the number of video frames of ultrasound video data collected on the ultrasound data acquisition equipment produced by different mainstream manufacturers is about 24 to 30 frames per second, the present invention has basically achieved real-time detection. Moreover, the automatic detection method for carotid artery ultrasound image plaques provided by the present invention can achieve real-time detection by certain algorithm optimization or simple hardware upgrade, which greatly improves the doctor's user experience.

[0195] The present invention provides a system for automatically detecting plaques in carotid artery ultrasound images, comprising:

[0196] A feature extraction module for extracting high-level visual features of carotid artery ultrasound images;

[0197] a target region generating module, configured to search for a target region of the overall carotid artery region or the carotid artery blood flow region according to the high-level visual features, and generate a target frame;

[0198] The target frame position correction module is used to correct the position deviation of the target frame on the high-level visual feature map;

[0199] The contour fitting module is used to perform contour fitting to obtain the overall area of the carotid artery and the carotid artery blood flow area.

[0200] In the feature extraction, ResNet is used to extract high-level visual feature maps of the carotid artery ultrasound image, including:

[0201] ResNet is used to batch normalize the data, including:

[0202] 1.1) Find the mean of each training batch of data:

[0203]

[0204] Among them, x i represents the i-th sample in a batch, and n represents the number of samples in a batch;

[0205] 1.2) Find the variance of each training batch of data:

[0206]

[0207] 1.3) Data standardization:

[0208]

[0209] Where x′ i Represents x i As a result of standardization, ε is a small positive number;

[0210] 1.4) Training parameters γ, β:

[0211] Among them, the scale factor γ and the translation factor β are learned by back propagation during training;

[0212] 1.5) The output y is converted to its original value through a linear transformation of γ and β. The linear transformation method is:

[0213] y i =γ*x′ i +β

[0214] Construct a residual network block structure, including:

[0215] Linear activation is performed as follows:

[0216] z[l+1]=W[l+1]*a[l]+*b[l+1]

[0217] Among them, a[l] is the input, W[l+1] is the weight matrix, and b[l+1] is the bias factor;

[0218] ReLU nonlinear activation is performed according to this formula:

[0219] a[l+1]=g(z[l+1])

[0220] Where g() is the ReLU function;

[0221] After activation, the residual network block structure is obtained;

[0222] Extracting high-level visual features of carotid artery ultrasound images via ResNet.

[0223] In the target region generation module, the target frame is generated based on the Deep Snake carotid artery plaque detection algorithm, and the region generation network is used to perform a target frame search on the high-level visual feature map obtained by ResNet to generate the optimal target frame where the overall carotid artery region and blood flow region are located. The module comprises the following steps:

[0224] Use sliding windows of different sizes, lengths and widths to sequentially scan the high-level visual feature maps extracted by ResNet;

[0225] The region generation network uses the softmax activation function to classify each sliding window to determine whether the sliding window maximizes the high-level visual features of the overall carotid artery area or blood flow area. If so, the offset of the sliding window is calculated;

[0226] The Proposals layer at the end of the region generation network structure obtains the sliding window and offset containing the overall carotid artery region or blood flow region features, and removes the sliding window that does not meet the feature map boundary.

[0227] On the extracted high-level visual feature map, the optimal target box containing the overall area of the carotid artery or the blood flow area features is obtained.

[0228] In the target frame position correction module, the target frame position correction adopts the ROI Align method, and bilinear interpolation is performed according to the length and width two-dimensional coordinates of the image to correct the position deviation of the target frame on the original carotid artery ultrasound image; the ROI Align method includes:

[0229] Traverse each candidate region and keep the floating point boundaries unquantized;

[0230] The candidate region is divided into k*k units, and the boundary of each unit is not quantized;

[0231] Multiple fixed coordinate positions are calculated in each unit, and the values of multiple positions are calculated using the bilinear interpolation method. Then, a maximum pooling operation is performed to correct the position deviation of the target box on the original carotid artery ultrasound image.

[0232] In the contour fitting module, a circular convolution method is used to extract high-level visual features of the target area contour. The circular convolution method includes:

[0233]

[0234] Among them, k is the convolution kernel function and * is the convolution operator symbol.

[0235] In the contour fitting module, contour fitting is performed through the Deep Snake network model, and contour fitting includes:

[0236] The optimal target frame obtained in the target frame position correction module is used as the initial outline of the target area;

[0237] Take the midpoints of each side of the target box and connect them to get a diamond outline;

[0238] The four vertices of the diamond outline are shifted to the four extreme points of the target area, namely the top point, the bottom point, the leftmost point and the rightmost point of the target area, and the extreme points of the target area outline are obtained;

[0239] For the left and right extremes, extend the distance of one-quarter of the edge of the previous rectangular target box upward and downward respectively; if it extends to the edge of the target box, it is truncated; for the upper and lower extremes, extend the distance of one-quarter of the edge to the left and right respectively, and truncate it if it encounters the edge of the target box; finally, the octagonal outline is obtained;

[0240] Select N nodes in the octagonal contour to represent the target area contour, and calculate the sum of the offset losses of all nodes during each contour fitting;

[0241] Continuously iterate and update to obtain the final fitting contour of the target area.

[0242] The data preprocessing part of the Deep Snake network model initially transforms the square target box into an octagonal outline, and uses this octagonal outline as the input of the outline fitting network model; the Deep Snake network model includes:

[0243] an extraction module, configured to extract high-level visual features of the octagonal outline through circular convolution;

[0244] A fusion module, configured to fuse the target regions if multiple target regions of the same category, close to each other and disconnected are detected;

[0245] The prediction module is used to fit the true contour by predicting the offset of each node in the contour.

[0246] The present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps of the method for automatically detecting plaques in carotid artery ultrasound images are implemented.

[0247] The method for automatically detecting plaques in carotid artery ultrasound images comprises the following steps:

[0248] Extracting high-level visual features from carotid artery ultrasound images;

[0249] Searching for a target area of the overall carotid artery area or the carotid artery blood flow area according to the high-level visual features, and generating a target frame;

[0250] Correct the position deviation of the target box on the high-level visual feature map;

[0251] Perform contour fitting to obtain the overall carotid artery area and carotid artery blood flow area;

[0252] The carotid artery blood flow area was subtracted from the total carotid artery area to obtain the carotid artery plaque area.

[0253] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method for automatic detection of carotid artery ultrasound image plaques.

[0254] The method for automatically detecting plaques in carotid artery ultrasound images comprises the following steps:

[0255] Extracting high-level visual features from carotid artery ultrasound images;

[0256] Searching for a target area of the overall carotid artery area or the carotid artery blood flow area according to the high-level visual features, and generating a target frame;

[0257] Correct the position deviation of the target box on the high-level visual feature map;

[0258] Perform contour fitting to obtain the overall carotid artery area and carotid artery blood flow area;

[0259] The carotid artery blood flow area was subtracted from the total carotid artery area to obtain the carotid artery plaque area.

[0260] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0261] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0262] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0263] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0264] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. An automatic detection system for carotid artery ultrasound plaques, characterized in that: include: Feature extraction module, which extracts high-level visual feature maps of carotid artery ultrasound images based on ResNet; a target region generating module, configured to search for a target region of the overall carotid artery region or the carotid artery blood flow region according to the high-level visual feature map, and generate a target frame; The target frame generation algorithm for carotid artery plaque detection based on the DeepSnake network model uses the target region generation network to perform target frame search on the high-level visual feature map obtained by ResNet, generating the optimal target frame for the overall carotid artery area and blood flow area. The target frame position correction module is used to correct the position deviation of the target frame on the high-level visual feature map. The target frame position correction adopts the ROIAlign method and performs bilinear interpolation based on the length and width two-dimensional coordinates of the image to correct the position deviation of the target frame on the original carotid artery ultrasound image. The contour fitting module is used to perform contour fitting. It uses the circular convolution method to extract the high-level visual features of the target area contour, and performs contour fitting through the DeepSnake network model to obtain the overall carotid artery area and the carotid artery blood flow area. The difference between the overall carotid artery area and the carotid artery blood flow area is used to obtain the carotid artery plaque area.

2. The automatic detection system for carotid artery ultrasound plaques according to claim 1, characterized in that: The feature extraction is specifically used for: ResNet is used to batch normalize the data, including: 1.1) Find the mean of each training batch of data: in, Indicates the first samples, Indicates the number of samples in a batch; 1.2) Find the variance of each training batch of data: 1.3) Data standardization: in, express As a result of standardization, ε is a small positive number; 1.4) Training parameters γ, β: Among them, the scale factor γ and the translation factor β are learned by back propagation during training; 1.5) The output y is converted to its original value through a linear transformation of γ and β. The linear transformation method is: Construct a residual network block structure, including: Linear activation is performed as follows: in, For input, is the weight matrix, is the deviation factor; ReLU nonlinear activation is performed according to this formula: in is the ReLU function; After activation, the residual network block structure is obtained; Extracting high-level visual features of carotid artery ultrasound images via ResNet.

3. The automatic detection system for carotid artery ultrasound plaques according to claim 1, characterized in that: The target area generation module is specifically used for: Use sliding windows of different sizes, lengths, and widths to sequentially scan the high-level visual feature maps extracted by ResNet; The region generation network uses the softmax activation function to classify each sliding window to determine whether the sliding window maximizes the high-level visual features of the overall carotid artery area or blood flow area. If so, the offset of the sliding window is calculated; The Proposals layer of the region generation network obtains sliding windows and offsets that contain the overall carotid artery region or blood flow region features, and removes sliding windows that do not meet the feature map boundaries. On the extracted high-level visual feature map, the optimal target box containing the overall area of the carotid artery or the blood flow area features is obtained.

4. The automatic detection system for carotid artery ultrasound plaques according to claim 1, characterized in that: The target frame position correction module is specifically used for: Traverse each candidate region and keep the floating point boundaries unquantized; The candidate region is divided into m*m units, and the boundary of each unit is not quantized; Calculate multiple fixed coordinate positions in each unit, use bilinear interpolation to calculate the values of multiple positions, and then perform maximum pooling operation; To correct the position deviation of the target box on the original carotid artery ultrasound image.

5. The automatic detection system for carotid artery ultrasound plaques according to claim 1, characterized in that: In the contour fitting module, the circular convolution method includes: in, k is the convolution kernel function, and * is the convolution operator symbol.

6. The automatic detection system for carotid artery ultrasound plaques according to claim 1, characterized in that: In the contour fitting module, contour fitting includes: The optimal target frame obtained in the target frame position correction module is used as the initial outline of the target area; Take the midpoints of each side of the target box and connect them to get a diamond outline; The four vertices of the diamond outline are displaced to the four extreme points of the target area to obtain the extreme points of the target area outline; For the left and right extremes, extend the distance of one-quarter of the edge of the previous rectangular target box upward and downward respectively; if it extends to the edge of the target box, it is truncated; for the upper and lower extremes, extend the distance of one-quarter of the edge to the left and right respectively, and truncate it if it encounters the edge of the target box; finally, the octagonal outline is obtained; Select N nodes in the octagonal contour to represent the target area contour, and calculate the sum of the offset losses of all nodes during each contour fitting; Continuously iterate and update to obtain the final fitting contour of the target area.

7. The automatic detection system for carotid artery ultrasound plaques according to claim 6, characterized in that: The data preprocessing part of the Deep Snake network model initially transforms the square target box into an octagonal outline, and uses this octagonal outline as the input of the outline fitting network model; The Deep Snake network model includes: an extraction module, configured to extract high-level visual features of the octagonal outline through circular convolution; A fusion module, configured to fuse the target regions if multiple target regions of the same category, close to each other and disconnected are detected; The prediction module is used to fit the true contour by predicting the offset of each node in the contour.

8. The automatic detection system for carotid artery ultrasound plaques according to claim 6, characterized in that: During the contour fitting process, the following loss function is used to calculate the offset loss of all nodes in the contour: in, is a node in the predicted target area contour line, It is a node in the actual target area contour line.

9. The automatic detection system for carotid artery ultrasound plaques according to claim 6, characterized in that: In the process of moving to the four extreme points of the target area, the following loss function is used to calculate the target box offset loss: in, are the four extreme points of the predicted target area, are the four extreme points of the real target area.

10. A method for automatically detecting plaques in carotid artery ultrasound images, characterized in that: include: Extract high-level visual feature maps of carotid artery ultrasound images based on ResNet; Searching for a target area of the carotid artery overall area or the carotid artery blood flow area according to the high-level visual feature map, and generating a target frame; The target frame generation algorithm for carotid artery plaque detection based on the DeepSnake network model uses the target region generation network to perform target frame search on the high-level visual feature map obtained by ResNet, generating the optimal target frame for the overall carotid artery area and blood flow area. Correct the position deviation of the target box on the high-level visual feature map; the target box position correction adopts the ROIAlign method, and performs bilinear interpolation based on the length and width two-dimensional coordinates of the image to correct the position deviation of the target box on the original carotid artery ultrasound image; Perform contour fitting; use the circular convolution method to extract high-level visual features of the target area contour, and perform contour fitting through the DeepSnake network model to obtain the overall carotid artery area and the carotid artery blood flow area. The difference between the overall carotid artery area and the carotid artery blood flow area is used to obtain the carotid artery plaque area.

Images