A portable three-dimensional carotid artery ultrasound automatic diagnosis system and method

Abstract

One technical solution of the present invention is to provide a portable three-dimensional carotid artery ultrasound automatic diagnosis system, which is characterized by including a data acquisition module, an automatic segmentation network, an automatic diagnosis network, and a three-dimensional reconstruction and visualization module. Another technical solution of the present invention is to provide a portable three-dimensional carotid artery ultrasound automatic diagnosis method implemented based on the foregoing system. In terms of data acquisition, the present invention stipulates a standard scanning process to reduce artifacts in the reconstruction process. At the same time, the present invention makes the three-dimensional reconstruction result smoother by filtering the position information and other methods. In addition, in terms of post-processing of data, the present invention realizes the automation, intelligence, and visualization of three-dimensional carotid artery ultrasound image segmentation and analysis by using deep learning technology.

Description

Technical Field

[0001] The present invention relates to medical imaging and ultrasonic carotid artery vascular imaging technology. Background Art

[0002] In recent years, stroke caused by carotid atherosclerosis has been one of the main causes of death. The pathological manifestations of carotid atherosclerosis are the increase in the thickness of the carotid intima-media and the appearance of plaques. As the plaques continue to develop, on the one hand, the lumen of the carotid artery becomes narrowed or even completely blocked, hindering the blood flow in the blood vessel and thus affecting the oxygen supply to the human brain. On the other hand, the plaques may fall off, ulcerate, or cause local damage, resulting in a hypercoagulable state of the blood, causing the aggregation of red blood cells and platelets, and forming thrombi, leading to stroke. According to the 2018 China Cardiovascular Disease Report, about 290 million people in China suffer from cardiovascular diseases, among which 13 million are stroke patients, and the incidence rate of carotid atherosclerosis in people over 40 years old is about 36.2%.

[0003] Most of the ultrasounds used in clinics are two-dimensional ultrasounds. Although they have advantages such as fast speed, their image quality is poor, the information dimension provided is limited, and they are easily affected by the operator's experience. On the other hand, since the examination of carotid atherosclerosis is mainly carried out by experienced ultrasound doctors in hospitals, it is difficult for patients in underdeveloped or remote areas to obtain timely diagnosis and treatment. At the same time, due to the large population base in China and the serious aging problem, the number of people undergoing carotid ultrasound examination is large, the appointment time is long, the medical treatment procedure is cumbersome, and a large number of carotid patients bring a huge burden to the medical system.

[0004] Three-dimensional ultrasound imaging can provide more abundant dimensional information, reduce the dependence on the operator's experience, and at the same time can quantitatively give the volume size of the region of interest (ROI). In the diagnosis of carotid atherosclerosis, three-dimensional ultrasound can directly provide information such as the volume size, morphological characteristics, and echo intensity of the plaques, which helps ultrasound doctors make more accurate diagnoses. At the same time, portable devices have a wider range of application scenarios, making it possible to screen for carotid atherosclerosis in large-scale communities and remote areas.

[0005] Current three-dimensional carotid ultrasound often uses robotic arm scanning imaging. Although it has advantages such as stable imaging and simple reconstruction algorithms, the degree of freedom of robotic arm scanning is limited, resulting in poor two-dimensional imaging quality. Portable three-dimensional unconstrained ultrasound uses a magnetic positioning three-dimensional positioning method, which can make the scanning method more free. The current related methods of portable unconstrained three-dimensional ultrasound imaging mainly include unconstrained scanning and voxel-based three-dimensional real-time bone imaging methods. For example, the invention patent application with the application number CN201911132940.5 discloses an unconstrained scanning and voxel-based three-dimensional real-time bone imaging method, and the invention patent application with the application number CN202010165914.9 discloses a handheld unconstrained scanning wireless three-dimensional ultrasound real-time voxel imaging system. However, the related methods or systems have the following problems in three-dimensional carotid ultrasound reconstruction and automatic recognition:

[0006] 1) Since the three-dimensional carotid ultrasound imaging is relatively delicate, the probe used needs to be replaced with a linear array probe with higher imaging resolution.

[0007] 2) If the three-dimensional imaging steps in the related methods are used for carotid ultrasound imaging, due to factors such as involuntary breathing of the subject and blood vessel pulsation during the scanning process of the carotid artery, the carotid ultrasound images at the same position at different times will be inconsistent, resulting in artifacts in the reconstruction.

[0008] 3) Since the three-dimensional carotid ultrasound has high requirements for the resolution of the reconstruction, the involuntary shaking of the scanner's hand during the scanning process will affect the quality of the reconstructed image.

[0009] 4) The related systems can only provide the results of three-dimensional ultrasound imaging, and there is no specific algorithm and process for automatic segmentation and diagnosis of the carotid artery, and the automatic segmentation, recognition, and visualization of carotid plaques cannot be achieved. Summary of the Invention

[0010] The object of the present invention is to provide a three-dimensional ultrasound carotid atherosclerosis automatic diagnosis technology combining a portable handheld three-dimensional ultrasound system and deep learning technology.

[0011] To achieve the above object, a technical solution of the present invention is to provide a portable three-dimensional carotid ultrasound automatic diagnosis system, which is characterized in that it includes a data acquisition module, an automatic segmentation network, an automatic diagnosis network, and a three-dimensional reconstruction and visualization module, wherein:

[0012] The data acquisition module is used to acquire a series of two-dimensional B-mode images of the carotid artery blood vessels of the subject and their corresponding position information, and further define this series of two-dimensional B-mode images of the carotid artery blood vessels as a two-dimensional ultrasound B-mode image sequence;

[0013] An automatic segmentation network is used to infer the LIB region and MAB region in each two-dimensional ultrasound B-mode image in a sequence of two-dimensional ultrasound B-mode images and generate a corresponding mask. Among them, the MAB region is the region between the outer membrane boundaries of blood vessels, and the LIB region is the region between the inner membrane of the blood vessel and the lumen boundary.

[0014] The LIB region, MAB region inferred by the automatic segmentation network and the corresponding mask are used as the input of the automatic diagnosis network. Among them, the LIB region and MAB region are used as image inputs, and the mask is used as a label input. The automatic diagnosis network consists of two symmetric feature extraction networks and a feature fusion network. The image input and label input are respectively input into the two feature extraction networks to obtain image features and label features of the same dimension. The image features and label features are concatenated in the channel dimension as the input of the feature fusion network, and the classification result, that is, whether there is a plaque, is output by the feature fusion network.

[0015] The three-dimensional reconstruction and visualization module first filters, smooths or regularizes the three-dimensional position information and then performs three-dimensional reconstruction, including the following steps:

[0016] Perform low-pass filtering on the collected three-dimensional position information to filter out the high-frequency components in the three-dimensional position information, that is, the hand jitter of the operator, to obtain smooth three-dimensional position information.

[0017] For the three-dimensional position information with trajectory backtracking, key frame analysis is adopted, and the three-dimensional position information with backtracking and the corresponding two-dimensional images are rearranged according to the front and back information of the positions to avoid different two-dimensional images at the same or similar positions, resulting in three-dimensional reconstruction artifacts.

[0018] After obtaining the two-dimensional MAB region and LIB region, combined with the smoothed three-dimensional position information, the true three-dimensional model of the carotid artery is reconstructed using the voxel-based inverse mapping method.

[0019] Performing volume rendering on the reconstructed three-dimensional model can obtain the three-dimensional visualization model of the carotid artery.

[0020] Preferably, the automatic segmentation network adopts a U-Net structure, and a batch normalization layer is connected behind each convolutional layer of the U-Net structure.

[0021] Preferably, the automatic segmentation network includes a first convolutional unit, a first pooling layer, a second convolutional unit, a second pooling layer, a third convolutional unit, a third pooling layer, a fourth convolutional unit, a fourth pooling layer, a fifth convolutional unit, a first upsampling layer, a sixth convolutional unit, a second upsampling layer, a seventh convolutional unit, a third upsampling layer, an eighth convolutional unit, a fourth upsampling layer, a ninth convolutional unit, and a fully connected layer. The outputs of the first convolutional unit, the second convolutional unit, the third convolutional unit, and the fourth convolutional unit are directly concatenated with the outputs of the first upsampling layer, the second upsampling layer, the third upsampling layer, and the fourth upsampling layer along the channel dimension and then input into the sixth convolutional unit, the seventh convolutional unit, the eighth convolutional unit, and the ninth convolutional unit. The outputs of the sixth convolutional unit, the seventh convolutional unit, the eighth convolutional unit, and the ninth convolutional unit are gradually upsampled through the upsampling layers to enlarge the size of the image, where:

[0022] The image is input into the first convolutional unit; the output of the first convolutional unit: on the one hand, it passes through the first pooling layer and outputs to the second convolutional unit; on the other hand, it outputs to the ninth convolutional unit, and is directly concatenated with the output of the fourth upsampling layer along the channel dimension and then input into the ninth convolutional unit;

[0023] The output of the second convolutional unit: on the one hand, it passes through the second pooling layer and outputs to the third convolutional unit; on the other hand, it outputs to the eighth convolutional unit, and is directly concatenated with the output of the third upsampling layer along the channel dimension and then input into the eighth convolutional unit;

[0024] The output of the third convolutional unit: on the one hand, it passes through the third pooling layer and outputs to the fourth convolutional unit; on the other hand, it outputs to the seventh convolutional unit, and is directly concatenated with the output of the second upsampling layer along the channel dimension and then input into the seventh convolutional unit;

[0025] The output of the fourth convolutional unit: on the one hand, it passes through the fourth pooling layer and outputs to the fifth convolutional unit; on the other hand, it outputs to the sixth convolutional unit, and is directly concatenated with the output of the first upsampling layer along the channel dimension and then input into the sixth convolutional unit;

[0026] The output of the fifth convolutional unit is input into the first upsampling layer, the output of the sixth convolutional unit is input into the second upsampling layer, the output of the seventh convolutional unit is input into the third upsampling layer, and the output of the eighth convolutional unit is input into the fourth upsampling layer;

[0027] The output of the ninth convolutional unit passes through a fully connected layer and then outputs the segmentation map.

[0028] Preferably, the training of the automatic segmentation network includes the following steps:

[0029] After obtaining the two-dimensional ultrasound B-mode image sequence and the corresponding position information, manually label the MAB region and the LIB region of each two-dimensional ultrasound B-mode image;

[0030] After the marking is completed, preprocessing is performed on the two-dimensional ultrasound B-mode image. The preprocessing includes image size normalization, gray-scale stretching, and data augmentation, where:

[0031] Image size normalization processing: Reset the two-dimensional ultrasound B-mode image through nearest neighbor interpolation;

[0032] Gray-scale stretching: Change the image intensity to between 0 and 1;

[0033] Data augmentation: Perform online data augmentation.

[0034] All preprocessed two-dimensional ultrasound B-mode images form a two-dimensional ultrasound image sequence. The two-dimensional ultrasound image sequence and the label information of the manually annotated MAB region and LIB region are input into the automatic segmentation network for training. The loss function of the automatic segmentation network consists of the Dice loss function and the cross-entropy loss function.

[0035] Preferably, the LIB region and MAB region obtained by inferring the automatic segmentation network and the corresponding masks are respectively cropped and adjusted to the same size and then used as the image input and label input to be input into the two feature extraction networks respectively.

[0036] Preferably, the training of the automatic diagnosis network includes the following steps:

[0037] Obtain two-dimensional ultrasound B-mode images through the data acquisition module;

[0038] After preprocessing the two-dimensional ultrasound B-mode image, all preprocessed two-dimensional ultrasound B-mode images form a two-dimensional ultrasound image sequence. Among them, the preprocessing of the two-dimensional ultrasound B-mode image specifically includes the following steps:

[0039] Crop the MAB region and LIB region in each two-dimensional ultrasound B-mode image, set the image intensity in the LIB region to 0, and set the image intensity in the MAB region to the original blood vessel wall region image intensity; Crop and adjust the masks of the MAB region and LIB region to the same size as the images of the MAB region and LIB region to obtain the label input;

[0040] Perform online training data augmentation on the image input and label input of the automatic diagnosis network;

[0041] Manually annotate whether there is a plaque in each two-dimensional ultrasound B-mode image, generate a label corresponding to the two-dimensional ultrasound image sequence, and input the two-dimensional ultrasound image sequence and the corresponding label into the automatic diagnosis network for training. The loss function of the automatic diagnosis network uses the cross-entropy loss function.

[0042] Another technical solution of the present invention is to provide a portable three-dimensional carotid artery ultrasound automatic diagnosis method implemented based on the foregoing system, which is characterized by including the following steps:

[0043] Step 1: The subject lies in a supine position, rotates the head in the direction of the side to be scanned, and exposes the neck skin on one side. Subsequently, the researcher holds a portable ultrasound probe and scans uniformly and linearly along the carotid artery from the distal common carotid artery at the bifurcation to the bifurcation, so as to collect a series of two-dimensional B-mode images of the carotid artery and their corresponding position information. This series of two-dimensional B-mode images of the carotid artery is further defined as a two-dimensional ultrasound B-mode image sequence;

[0044] To obtain a clearer three-dimensional reconstruction image quality, the following scanning method is followed during carotid artery scanning:

[0045] Keep the scanning speed uniformly unchanged. Control the time of a single carotid artery scan to be between 5 seconds and 10 seconds;

[0046] Keep the scanning direction unchanged and avoid the scanning trajectory from retreating;

[0047] Keep the scanning trajectory smoothly changing and avoid large fluctuations during the scanning process;

[0048] Step 2: Input the obtained two-dimensional ultrasound B-mode image sequence into the trained automatic segmentation network. The automatic segmentation network infers the LIB region and MAB region in each two-dimensional ultrasound B-mode image in the two-dimensional ultrasound B-mode image sequence, and generates a corresponding mask;

[0049] Step 3: Based on the LIB region and MAB region of the two-dimensional ultrasound B-mode image sequence obtained in Step 2, use a three-dimensional reconstruction and visualization module to generate a three-dimensional visualization model of the carotid artery;

[0050] Step 4: Each cross-sectional image of the three-dimensional visualization model of the carotid artery is a slice image. Input the slice image and the mask obtained in Step 2 into the trained automatic diagnosis network, and infer the diagnosis result of each slice. If N consecutive slices are judged by the automatic diagnosis network to have plaques, it is judged that the subject has carotid atherosclerosis; otherwise, it is judged that the subject does not have carotid atherosclerosis, where N is an empirical threshold.

[0051] In terms of data acquisition, the present invention reduces artifacts in the reconstruction process by stipulating a standard scanning process. At the same time, the present invention makes the three-dimensional reconstruction result smoother by filtering the position information and other methods. In addition, in terms of post-processing of data, the present invention realizes the automation, intelligence, and visualization of three-dimensional carotid artery ultrasound image segmentation and analysis by using deep learning technology. Description of the Drawings

[0052] Figure 1 is the overall flow chart for system use;

[0053] Figure 2 is the network structure diagram for automatic segmentation;

[0054] Figure 3 is the network structure diagram for automatic diagnosis;

[0055] Figure 4 Schematically shows the comparison between the automatic segmentation result and the manual marking result. In the figure, the brighter lines are the results automatically recognized by the algorithm, and the darker lines are the results manually marked by humans. It can be seen that the automatic segmentation by the algorithm is basically consistent with the manual marking segmentation;

[0056] Figure 5 Schematically shows the comparison of the position information before and after filtering. Among them, the curve with larger jitter is the position information before filtering, and the smoother curve is the relative position information after filtering;

[0057] Figure 6A and Figure 6B is the longitudinal sectional view comparison of the three-dimensional reconstruction of blood vessels before and after filtering, Figure 6A is the reconstruction after filtering, Figure 6B is the direct reconstruction. It can be observed that the reconstructed image after filtering is smoother;

[0058] Figure 7A and Figure 7B is the three-dimensional visualization comparison between the automatic segmentation result and the manual marking result, Figure 7A is the automatic segmentation result, Figure 7B is the manual marking segmentation result. Detailed Embodiments

[0059] The following further elaborates the present invention in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

[0060] A portable three-dimensional carotid artery ultrasound automatic diagnosis system disclosed in this embodiment includes:

[0061] i) A data acquisition module

[0062] The data acquisition module is used to perform carotid artery scanning through a portable ultrasound probe, so as to collect a series of two-dimensional B-mode images of carotid artery vessels about 4 cm from the common carotid artery to the bifurcation of the subject and their corresponding position information. This series of two-dimensional B-mode images of carotid artery vessels is further defined as a two-dimensional ultrasound B-mode image sequence.

[0063] When performing carotid artery scanning, the subject lies in a supine position and rotates the head towards the side to be scanned, exposing the neck skin on one side. Subsequently, the researcher holds the portable ultrasound probe and scans uniformly and linearly along the carotid artery from the distal common carotid artery at the bifurcation to the bifurcation, so as to collect a series of two-dimensional B-mode images of carotid artery vessels about 4 cm from the common carotid artery to the bifurcation and their corresponding position information.

[0064] To obtain a clearer three-dimensional reconstruction image quality, the following scanning method should be followed when performing carotid artery scanning:

[0065] Keep the scanning speed uniform and unchanged. Control the time of a single carotid artery scan between 5 seconds and 10 seconds.

[0066] Keep the scanning direction unchanged and avoid the scanning trajectory from going back.

[0067] Keep the scanning trajectory changing smoothly and avoid large-amplitude jitters during the scanning process.

[0068] II) Automatic segmentation network

[0069] The automatic segmentation network adopts an improved U-Net structure. Compared with the original U-Net network, the automatic segmentation network in the present invention connects a batch normalization (Batchnorm) layer behind each convolutional layer to accelerate network convergence and improve the segmentation accuracy. Specifically, as Figure 2As shown in the figure, the automatic segmentation network consists of 9 convolutional units, 4 max-pooling layers, 4 upsampling layers, and 1 fully-connected layer (a convolutional layer with a size of 1*1). Among them, convolutional unit 1, pooling layer 1, convolutional unit 2, pooling layer 2, convolutional unit 3, pooling layer 3, convolutional unit 4, pooling layer 4, convolutional unit 5, upsampling layer 1, convolutional unit 6, upsampling layer 2, convolutional unit 7, upsampling layer 3, convolutional unit 8, upsampling layer 4, convolutional unit 9, and the fully-connected layer are connected in sequence. The output dimensions of convolutional unit 1, convolutional unit 2, convolutional unit 3, and convolutional unit 4 are 224*224*64, 112*112*128, 56*56*256, and 28*28*512 respectively. The outputs of convolutional unit 1, convolutional unit 2, convolutional unit 3, and convolutional unit 4 are directly concatenated with the outputs of upsampling layer 1, upsampling layer 2, upsampling layer 3, and upsampling layer 4 along the channel dimension and then input into convolutional unit 6, convolutional unit 7, convolutional unit 8, and convolutional unit 9. After being processed by convolutional unit 6, convolutional unit 7, convolutional unit 8, and convolutional unit 9, their output dimensions are 28*28*512, 56*56*256, 112*112*128, and 224*224*64 respectively. The outputs of convolutional unit 6, convolutional unit 7, convolutional unit 8, and convolutional unit 9 are gradually upsampled by the upsampling layer to enlarge the size of the image. The image is input into convolutional unit 1. The output of convolutional unit 1: on the one hand, it passes through pooling layer 1 and outputs to convolutional unit 2; on the other hand, it outputs to convolutional unit 9, is directly concatenated with the output of upsampling layer 4 along the channel dimension, and then input into convolutional unit 9. The output of convolutional unit 2: on the one hand, it passes through pooling layer 2 and outputs to convolutional unit 3; on the other hand, it outputs to convolutional unit 8, is directly concatenated with the output of upsampling layer 3 along the channel dimension, and then input into convolutional unit 8. The output of convolutional unit 3: on the one hand, it passes through pooling layer 3 and outputs to convolutional unit 4; on the other hand, it outputs to convolutional unit 7, is directly concatenated with the output of upsampling layer 2 along the channel dimension, and then input into convolutional unit 7. The output of convolutional unit 4: on the one hand, it passes through pooling layer 4 and outputs to convolutional unit 5; on the other hand, it outputs to convolutional unit 6, is directly concatenated with the output of upsampling layer 1 along the channel dimension, and then input into convolutional unit 6. The output of convolutional unit 5 is input into upsampling layer 1, the output of convolutional unit 6 is input into upsampling layer 2, the output of convolutional unit 7 is input into upsampling layer 3, and the output of convolutional unit 8 is input into upsampling layer 4. The output of convolutional unit 9 passes through a fully-connected layer and then outputs the segmentation map.

[0070] In this embodiment, each convolutional unit is composed of two basic convolutional layers. Each basic convolutional layer includes a convolutional layer, followed by a batch normalization layer and a linear activation unit (Linear rectification function, ReLU).

[0071] In the automatic segmentation network: the kernel size of all convolutional layers is 3*3, the stride is 1, and the padding is 1; the kernel size of all pooling layers is 2*2, and the stride is 2.

[0072] The training of the automatic segmentation network includes the following steps:

[0073] After obtaining the two-dimensional ultrasound B-mode images of the carotid artery and the corresponding position information using the data acquisition module, the region between the outer and middle membranes of the blood vessel (media-adventitia boundary, hereinafter referred to as the MAB region) and the region between the inner membrane of the blood vessel and the lumen boundary (lumen-intima boundary, hereinafter referred to as the LIB region) of each two-dimensional ultrasound B-mode image are manually marked using annotation software.

[0074] After the marking is completed, the two-dimensional ultrasound B-mode images are preprocessed. The preprocessing includes image size normalization, gray scale stretching, and data augmentation.

[0075] Image size normalization processing: The two-dimensional ultrasound B-mode images are reset by nearest neighbor interpolation.

[0076] Gray scale stretching: The image intensity is changed to between 0 and 1 using the following formula:

[0077]

[0078] In the formula, I represents the image intensity.

[0079] Data augmentation: Online data augmentation is performed using random image scaling, inversion, rotation, gamma gray scale transformation, etc.

[0080] All preprocessed two-dimensional ultrasound B-mode images form a two-dimensional ultrasound image sequence. The two-dimensional ultrasound image sequence and the label information of the manually marked MAB and LIB are input into the aforementioned automatic segmentation network for training. The loss function of the automatic segmentation network consists of the Dice loss function and the cross-entropy loss function. After the training is completed, the model parameters of the automatic segmentation network are saved for subsequent steps.

[0081] (III) Automatic diagnosis network

[0082] The automatic diagnosis network consists of two symmetric feature extraction networks and a feature fusion network, as Figure 3As shown. Specifically, each feature extraction network is composed of basic convolutional unit one, basic convolutional unit two, max pooling layer one, basic convolutional unit three, basic convolutional unit four, max pooling layer two, basic convolutional unit five, basic convolutional unit six, and max pooling layer three connected in sequence. The input of the automatic diagnosis network includes an image input and a label input. Among them, the image input is the segmentation map output by the automatic segmentation network, and the label input is obtained by cropping and resizing the labels of the MAB region and LIB region output by the automatic segmentation network to 128*128. After the image input and the label input are respectively input into two feature extraction networks, image features and label features are obtained, where the dimensions of the image features and label features are 16*16*96. The image features and label features are concatenated in the channel dimension as the input of the feature fusion network.

[0083] The feature fusion network is composed of basic convolutional unit thirteen, basic convolutional unit fourteen, max pooling layer seven, global average pooling layer, and fully connected layer connected in sequence. The output dimension of max pooling layer seven is 96*16*16. After passing through the global average pooling layer, the output dimension becomes 96*1. After passing through the fully connected layer, the classification result, that is, whether there is a plaque or not, is output.

[0084] The basic convolutional unit in the automatic diagnosis network is composed of two basic convolutional layers. Each basic convolutional layer includes a convolutional layer followed by a batch normalization (Batchnorm) layer and a linear rectification unit (ReLU).

[0085] In the automatic diagnosis network: the kernel size of all convolutional layers is 3*3, the stride is 1, and the padding is 1; the kernel size of all max pooling layers is 2*2, and the stride is 2.

[0086] The training of the automatic diagnosis network includes the following steps:

[0087] The two-dimensional ultrasound B-mode images obtained by the data acquisition module;

[0088] After preprocessing the two-dimensional ultrasound B-mode images, all the preprocessed two-dimensional ultrasound B-mode images form a two-dimensional ultrasound image sequence;

[0089] Whether there is a plaque in the ultrasound image has nothing to do with the image information outside the adventitia boundary and inside the intima of the blood vessel. Removing the useless image areas can accelerate the training of the network and improve the accuracy. Therefore, in this embodiment, the preprocessing of the two-dimensional ultrasound B-mode images specifically includes the following steps:

[0090] Crop the MAB region and LIB region in each two-dimensional ultrasound B-mode image, set the image intensity in the LIB region to 0, and set the image intensity in the MAB region to the original image intensity of the vascular wall region. Then resize it to 128*128 to obtain the image input for the automatic diagnosis network; crop and resize the labels of the MAB region and LIB region corresponding to the current two-dimensional ultrasound B-mode image to 128*128 to obtain the label input;

[0091] Perform online enhancement of training data on the image input and label input of the automatic diagnosis network by using random image inversion, image rotation transformation, etc.;

[0092] Manually annotate whether there are plaques in each two-dimensional ultrasound B-mode image, generate labels corresponding to the two-dimensional ultrasound image sequence, and input the two-dimensional ultrasound image sequence and the corresponding labels into the automatic diagnosis network for training. The loss function of the automatic diagnosis network uses the cross-entropy loss function. After training is completed, save the model parameters of the automatic diagnosis network for subsequent steps.

[0093] (IV) Three-dimensional reconstruction and visualization module

[0094] Based on the MAB region and LIB region of the two-dimensional ultrasound B-mode image sequence, perform three-dimensional reconstruction of the carotid artery region. Since during the scanning process, the operator may have involuntary hand tremors, directly using the three-dimensional position information for reconstruction may cause artifacts in the reconstructed image. Therefore, the three-dimensional reconstruction and visualization module first filters and smooths or regularizes the three-dimensional position information and then performs three-dimensional reconstruction. Specifically, the three-dimensional reconstruction and visualization module realizes three-dimensional reconstruction including the following steps:

[0095] Perform low-pass filtering on the collected two-dimensional position information to filter out the high-frequency components in the two-dimensional position information, that is, the operator's hand tremors, to obtain smooth three-dimensional position information;

[0096] For the three-dimensional position information with trajectory backtracking, use key-frame analysis to rearrange the data of the three-dimensional position information with backtracking and the corresponding two-dimensional images according to the front-back information of the positions to avoid different two-dimensional images at the same or similar positions, which may cause artifacts in three-dimensional reconstruction. Specifically, the Z-axis values of the three-dimensional position information points obtained by scanning can be sorted first, and at the same time, the two-dimensional images corresponding to the three-dimensional position points can be sorted, so as to obtain a clearer three-dimensional reconstructed vascular image.

[0097] The comparison diagram of the three-dimensional trajectory before and after filtering is as shown by the light and dark curves in Figure 5 The comparison diagram of the three-dimensional reconstruction of the blood vessel before and after filtering is as shown in Figure 6A and Figure 6B shown.

[0098] After obtaining the two-dimensional MAB region and LIB region, combined with the smoothed three-dimensional position information, the true three-dimensional model of the carotid artery is reconstructed using the voxel-based inverse mapping method.

[0099] Performing volume rendering on the reconstructed three-dimensional model can obtain the three-dimensional visualization model of the carotid artery.

[0100] A portable three-dimensional carotid artery ultrasound automatic diagnosis method implemented based on the above system, characterized by including the following steps:

[0101] Step 1: The subject lies in a supine position and rotates the head in the direction of the side to be scanned, exposing the neck skin on one side. Subsequently, the researcher holds a portable ultrasound probe and scans uniformly and linearly along the carotid artery from the distal common carotid artery at the bifurcation to the bifurcation, thereby collecting a series of two-dimensional B-mode images of the carotid artery and their corresponding position information from the common carotid artery to about 4 cm at the bifurcation. This series of two-dimensional B-mode images of the carotid artery is further defined as a two-dimensional ultrasound B-mode image sequence.

[0102] To obtain a clearer three-dimensional reconstruction image quality, the following scanning method should be followed during carotid artery scanning:

[0103] Keep the scanning speed uniform and unchanged. Control the time of one carotid artery scan between 5 seconds and 10 seconds.

[0104] Keep the scanning direction unchanged and avoid the scanning trajectory from going back.

[0105] Keep the scanning trajectory smoothly changing and avoid large fluctuations during the scanning process.

[0106] Step 2: Input the obtained two-dimensional ultrasound B-mode image sequence into the trained automatic segmentation network. The automatic segmentation network infers the LIB region and MAB region in each two-dimensional ultrasound B-mode image in the two-dimensional ultrasound B-mode image sequence and generates a corresponding mask (Mask).

[0107] Step 3: Based on the LIB region and MAB region of the two-dimensional ultrasound B-mode image sequence obtained in Step 2, use the three-dimensional reconstruction and visualization module to generate the three-dimensional visualization model of the carotid artery.

[0108] Step 4: Crop and adjust the size of each cross-sectional image of the three-dimensional visualization model of the carotid artery, input it into the trained automatic diagnosis network, and infer the diagnosis result of each slice, that is, whether there is a plaque. In this embodiment, if 5 consecutive slices are judged by the automatic diagnosis network to have a plaque, it is judged that the subject has carotid atherosclerosis; otherwise, it is judged that the subject does not have carotid atherosclerosis.

[0109] Applying the portable handheld three-dimensional ultrasound carotid atherosclerosis automatic diagnosis technology can achieve rapid three-dimensional imaging of the carotid artery vascular region by inexperienced operators. At the same time, combining artificial intelligence technology can achieve auxiliary diagnosis and three-dimensional visualization of carotid atherosclerosis. Here, 15 carotid artery data in clinical practice are used to apply this system to verify the effectiveness of the system.

[0110] Step 1: Data acquisition

[0111] The ultrasound image data are all obtained by a linear array ultrasound probe with a frequency of 8M (Clarius, L738-K, Canada). At the same time, the position information corresponding to each two-dimensional image is obtained by an electromagnetic positioning system (Polhemus, G4 unit, U.S.A). The final obtained two-dimensional image size is 640*480, with 256 gray levels.

[0112] Step 2: Automatic segmentation network training

[0113] After obtaining 40 carotid artery ultrasound scan data, use the annotation software to annotate the obtained two-dimensional ultrasound image sequence. Randomly select 25 ultrasound scan data as the training set and 15 ultrasound scan data as the validation set. Use the carotid artery two-dimensional images and their labels in the training set as input data to train the deep learning model of the automatic segmentation network.

[0114] Step 3: Automatic diagnosis network training

[0115] Mark the obtained two-dimensional ultrasound image sequence as having or not having plaques. Preprocess the corresponding images of the 25 ultrasound scan data selected in Step 2, and use the preprocessed ultrasound images and their corresponding labels as input to train the deep learning model of the automatic diagnosis network.

[0116] Step 4: Image automatic segmentation inference

[0117] Input the ultrasound image sequences of the randomly selected 15 validation sets into the trained automatic segmentation deep learning model to obtain the corresponding output labels. Use expressions (2)(3) to evaluate the results of the automatic segmentation.

[0118]

[0119] HD(A,B) = max(hd(A,B),hd(B,A))#(3)

[0120] Where

[0121] hd(A,B) = max a∈A (min b∈B ||a - b||)#(4)

[0122] hd(B, A) = max b∈B (min a∈A ||b - a||) #(5)

[0123] Where P and L are the results predicted and labeled by the automatic segmentation network respectively. DSC is a performance metric for evaluating the segmentation algorithm. A represents the true subset of the label, B represents the true subset of the segmentation network, HD(A, B) is an evaluation metric for the distance between the true subsets A and B in the metric space, hd(A, B) and hd(B, A) respectively represent the one-way Hausdorff distances from set A to set B and from set B to set A, and ||a - b|| represents the distance norm between point set a and point set b.

[0124] The result of automatic segmentation is as Figure 4 shown. The numerical results of automatic segmentation are shown in Table 1 below.

[0125]

[0126] Table 1 Numerical comparison between the results of automatic segmentation and manual labeling

[0127] Step Five: 3D Reconstruction and Visualization

[0128] Perform 3D reconstruction on the labels of MAB and LIB obtained by automatic segmentation in Step Four to obtain the 3D structure of the blood vessels, and then use volume rendering to visualize the 3D structure of the blood vessels. The visualization result diagrams of the 3D structure of the manually labeled blood vessels and the 3D structure of the automatic segmentation algorithm are as Figure 6A and Figure 6B shown.

[0129] Step Six: Inference of the Automatic Diagnosis Network

[0130] After completing the automatic segmentation inference of the images in Step Four, the MAB and LIB labels of the blood vessels obtained by predicting 15 ultrasound image sequences in the validation set are obtained. Input the cross-sectional slices of the preprocessed 3D blood vessel images into the automatic diagnosis deep learning model to obtain the prediction results of the automatic diagnosis of each 2D ultrasound image. The comparison between the automatic diagnosis results and the labeled results is shown in Table 2 below.

[0131]

[0132] Table 2 Contingency table of the automatic diagnosis results at the image level and the manually labeled diagnosis results

[0133] The sensitivity, specificity, and accuracy of the prediction results are 0.73, 0.97, and 0.91, respectively. For the automatic diagnosis results of individual patients, first, the three-dimensional vascular structure obtained in Step 5 is sliced into cross-sections. For each slice, the automatic diagnosis network is used for inference. If five consecutive slices are predicted by the automatic diagnosis network to have plaques, then the data is judged to have plaques. The comparison between the data-level automatic diagnosis results and the labeled results is shown in Table 3 below. The sensitivity, specificity, and accuracy of the prediction results are 0.81, 0.75, and 0.80, respectively.

[0134]

[0135] Table 3 Contingency table of data-level automatic diagnosis results and manually labeled diagnosis results.

Claims

1. A portable three-dimensional carotid artery ultrasound automatic diagnosis system, characterized in that, it includes a data acquisition module, an automatic segmentation network, an automatic diagnosis network, and a three-dimensional reconstruction and visualization module, wherein: The data acquisition module is used to acquire a series of two-dimensional B-mode images of the carotid artery vessels of the subject and their corresponding position information, and further define this series of two-dimensional B-mode images of the carotid artery vessels as a two-dimensional ultrasound B-mode image sequence; The automatic segmentation network is used to infer the LIB region and the MAB region in each two-dimensional ultrasound B-mode image in the two-dimensional ultrasound B-mode image sequence and generate a corresponding mask, wherein the MAB region is the region between the outer membrane boundaries of the blood vessel, and the LIB region is the region between the inner membrane of the blood vessel and the lumen boundary; The LIB region and the MAB region inferred by the automatic segmentation network and the corresponding mask are used as the input of the automatic diagnosis network, wherein the LIB region and the MAB region are used as image inputs, and the mask is used as a label input; the automatic diagnosis network is composed of two symmetric feature extraction networks and a feature fusion network. The image input and the label input are respectively input into the two feature extraction networks to obtain image features and label features of the same dimension. The image features and the label features are concatenated in the channel dimension as the input of the feature fusion network, and the classification result, that is, whether there is a plaque, is output by the feature fusion network; The three-dimensional reconstruction and visualization module first filters and smooths or regularizes the three-dimensional position information, and then performs three-dimensional reconstruction, including the following steps: Perform low-pass filtering on the collected three-dimensional position information to filter out the high-frequency components in the three-dimensional position information, that is, the hand jitter of the operator, to obtain smooth three-dimensional position information; For the three-dimensional position information with trajectory backtracking, key frame analysis is adopted, and the three-dimensional position information with backtracking and the corresponding two-dimensional images are rearranged according to the front and back information of the positions to avoid different two-dimensional images at the same or similar positions, resulting in three-dimensional reconstruction artifacts; After obtaining the two-dimensional MAB region and LIB region, combined with the smoothed three-dimensional position information, a real three-dimensional model of the carotid artery vessel is reconstructed by using the voxel-based inverse mapping method; Performing volume rendering on the reconstructed three-dimensional model can obtain a three-dimensional visualization model of the carotid artery vessel.

2. The portable three-dimensional carotid artery ultrasound automatic diagnosis system according to claim 1, characterized in that, The automatic segmentation network adopts a U-Net structure, and a batch normalization layer is connected behind each convolutional layer of the U-Net structure.

3. The portable three-dimensional carotid artery ultrasound automatic diagnosis system according to claim 1, characterized in that, The automatic segmentation network includes a first convolutional unit, a first pooling layer, a second convolutional unit, a second pooling layer, a third convolutional unit, a third pooling layer, a fourth convolutional unit, a fourth pooling layer, a fifth convolutional unit, a first upsampling layer, a sixth convolutional unit, a second upsampling layer, a seventh convolutional unit, a third upsampling layer, an eighth convolutional unit, a fourth upsampling layer, a ninth convolutional unit, and a fully connected layer. The outputs of the first convolutional unit, the second convolutional unit, the third convolutional unit, and the fourth convolutional unit are directly concatenated with the outputs of the first upsampling layer, the second upsampling layer, the third upsampling layer, and the fourth upsampling layer along the channel dimension and then input into the sixth convolutional unit, the seventh convolutional unit, the eighth convolutional unit, and the ninth convolutional unit. The outputs of the sixth convolutional unit, the seventh convolutional unit, the eighth convolutional unit, and the ninth convolutional unit are gradually upsampled through the upsampling layers to enlarge the size of the image, where: The image is input into the first convolutional unit; the output of the first convolutional unit: on the one hand, it is output to the second convolutional unit through the first pooling layer; on the other hand, it is output to the ninth convolutional unit, directly concatenated with the output of the fourth upsampling layer along the channel dimension and then input into the ninth convolutional unit; The output of the second convolutional unit: on the one hand, it is output to the third convolutional unit through the second pooling layer; on the other hand, it is output to the eighth convolutional unit, directly concatenated with the output of the third upsampling layer along the channel dimension and then input into the eighth convolutional unit; The output of the third convolutional unit: on the one hand, it is output to the fourth convolutional unit through the third pooling layer; on the other hand, it is output to the seventh convolutional unit, directly concatenated with the output of the second upsampling layer along the channel dimension and then input into the seventh convolutional unit; The output of the fourth convolutional unit: on the one hand, it is output to the fifth convolutional unit through the fourth pooling layer; on the other hand, it is output to the sixth convolutional unit, directly concatenated with the output of the first upsampling layer along the channel dimension and then input into the sixth convolutional unit; The output of the fifth convolutional unit is input into the first upsampling layer, the output of the sixth convolutional unit is input into the second upsampling layer, the output of the seventh convolutional unit is input into the third upsampling layer, and the output of the eighth convolutional unit is input into the fourth upsampling layer; The output of the ninth convolutional unit passes through a fully connected layer and then outputs a segmentation map.

4. A portable three-dimensional carotid artery ultrasound automatic diagnosis system according to claim 1, characterized in that the training of the automatic segmentation network includes the following steps: After obtaining a two-dimensional ultrasound B-mode image sequence and corresponding position information, manually mark the MAB region and LIB region of each two-dimensional ultrasound B-mode image; After the marking is completed, preprocess the two-dimensional ultrasound B-mode image. The preprocessing includes image size normalization, gray scale stretching, and data augmentation, where: Image size normalization processing: Reset the two-dimensional ultrasound B-mode image through nearest neighbor interpolation; Gray scale stretching: Change the image intensity to between 0 and 1; Data augmentation: Perform online data augmentation. All preprocessed two-dimensional ultrasound B-mode images form a two-dimensional ultrasound image sequence. The two-dimensional ultrasound image sequence and the label information of the manually marked MAB region and LIB region are input into the automatic segmentation network for training. The loss function of the automatic segmentation network consists of a Dice loss function and a cross-entropy loss function.

5. A portable three-dimensional carotid artery ultrasound automatic diagnosis system according to claim 1, characterized in that, the LIB region and the MAB region obtained by the automatic segmentation network inference and the corresponding masks are respectively cropped and adjusted to the same size and then used as the image input and the label input and respectively input into the two feature extraction networks.

6. A portable three-dimensional carotid artery ultrasound automatic diagnosis system according to claim 1, characterized in that, the training of the automatic diagnosis network includes the following steps: the two-dimensional ultrasound B-mode images obtained by the data acquisition module; after preprocessing the two-dimensional ultrasound B-mode images, all the preprocessed two-dimensional ultrasound B-mode images form a two-dimensional ultrasound image sequence, wherein the preprocessing of the two-dimensional ultrasound B-mode images specifically includes the following steps: cropping the MAB region and the LIB region in each two-dimensional ultrasound B-mode image, and setting the image intensity in the LIB region to 0 and the image intensity of the MAB region to the original blood vessel wall region image intensity; cropping and adjusting the masks of the MAB region and the LIB region to the same size as the images of the MAB region and the LIB region to obtain the label input; performing online enhancement of training data on the image input and the label input of the automatic diagnosis network; manually annotating whether there are plaques in each two-dimensional ultrasound B-mode image to generate labels corresponding to the two-dimensional ultrasound image sequence, inputting the two-dimensional ultrasound image sequence and the corresponding labels into the automatic diagnosis network for training, and the loss function of the automatic diagnosis network adopts the cross-entropy loss function.

Images