Carotid artery vulnerable plaque identification system and method
Through multimodal image fusion and automatic identification of plaque recognition modules and medical staff review, the problem of insufficient accuracy of carotid artery vulnerable plaque recognition in the prior art is solved, and the accurate identification of carotid artery vulnerable plaques is achieved. , reducing the risk of misdiagnosis.
Patent Information
- Application Number
- CN202411250888.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2025-01-03
AI Technical Summary
The prior art lacks accuracy in the identification of carotid artery vulnerable plaques, especially ultrasound imaging is limited in distinguishing the internal components of the plaque, and a single imaging modality is difficult to fully reflect the complex characteristics of the plaque, which increases misdiagnosis or missed diagnosis. risk.
The multimodal image fusion module is used to integrate ultrasound, CT and magnetic resonance imaging results, and the plaque recognition module is used to automatically identify and re-identify the boundaries of vulnerable plaques, and medical staff are introduced to review it, using Bayesian neural network and generating confrontation. Network enhances identification accuracy.
It improves the ability to identify internal components of the plaque, reduces the risk of misdiagnosis, realizes accurate identification of vulnerable plaques in the carotid artery, and enhances the reliability of diagnosis.
Smart Images
Figure CN119251630A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of carotid artery image recognition, and in particular to a carotid artery vulnerable plaque recognition system and method. Background Art
[0002] In the process of preventing and treating cardiovascular and cerebrovascular diseases, accurate identification of vulnerable carotid plaques plays a pivotal role. Due to their inherent instability, vulnerable plaques can easily trigger thrombosis once they rupture, leading to serious consequences such as stroke or myocardial infarction. Therefore, scientifically evaluating the stability and vulnerability of plaques is crucial for developing effective clinical intervention strategies.
[0003] Although ultrasound imaging, as a non-invasive examination method, has been widely used in the medical field, especially for special groups such as pregnant women and children, and can show the arterial structure and blood flow dynamics in real time, providing doctors with immediate diagnosis, its limitations cannot be ignored. Specifically, ultrasound imaging has significant deficiencies in accurately distinguishing the internal components of plaques (such as lipid core, fibrous cap, calcification, etc.), which directly limits its ability to assess plaque vulnerability.
[0004] Furthermore, although ultrasound, CT and magnetic resonance imaging have their own advantages, they also face their own challenges. Although ultrasound imaging is highly real-time, its resolution is limited; although CT imaging has high spatial resolution, it is accompanied by a high radiation dose; although magnetic resonance imaging has excellent soft tissue contrast and is radiation-free, it may be limited by equipment complexity and examination time in some cases. More importantly, a single imaging modality is often difficult to fully reflect the complex characteristics of plaques. In addition, factors such as differences in equipment performance, individual differences in patients, and different levels of operator experience may increase the risk of misdiagnosis or missed diagnosis.
[0005] It is particularly noteworthy that the diagnostic results of traditional ultrasound imaging are deeply affected by the operator's experience and technical level. This subjectivity not only reduces the consistency of diagnosis, but also poses a threat to the accuracy of diagnosis. Therefore, under the existing technical framework, how to overcome these shortcomings and achieve accurate identification of vulnerable carotid plaques has become an important issue that needs to be solved urgently. Summary of the invention
[0006] In view of the problems existing in the prior art, the present invention provides a carotid vulnerable plaque identification system, comprising:
[0007] a multimodal image fusion module, for respectively acquiring a carotid ultrasound image, a carotid CT image and a carotid magnetic resonance image obtained by scanning the same part of the carotid artery of a patient using an ultrasound imaging device, a CT imaging device and a magnetic resonance imaging device, and then performing multimodal image fusion on the carotid ultrasound image, the carotid CT image and the carotid magnetic resonance image to obtain a fused image;
[0008] A plaque recognition module is connected to the plaque recognition module and is used to perform boundary recognition of vulnerable plaques on the fused image, then extract vulnerable plaques based on the boundary recognition results for re-recognition, highlight the uncertain areas in the re-recognition results and notify medical staff for review, then collect the review marks fed back by the medical staff, and associate the review marks with the uncertain areas in the re-recognition results to obtain the final recognition result.
[0009] Preferably, the plaque identification module includes:
[0010] a plaque recognition unit, configured to input the fused image into a pre-built plaque recognition model to recognize the boundary of vulnerable plaques in the fused image, and take the portion of the fused image surrounded by the boundary of vulnerable plaques as the boundary recognition result;
[0011] an image extraction unit connected to the plaque recognition unit, and configured to extend the boundary of the vulnerable plaque toward the edge of the fused image based on the boundary recognition result in the fused image, and use the image surrounded by the expanded vulnerable plaque boundary in the fused image as a re-recognized image;
[0012] An uncertain marking unit is connected to the image extraction unit, and is used to input the re-recognition image into the plaque recognition model for re-recognition to obtain the re-recognition result, and input the re-recognition image into the Bayesian neural network to obtain a recognition result in which the uncertain area in the re-recognition image is highlighted, and then collect the review mark fed back by the medical staff, and associate the review mark with the uncertain area in the re-recognition result to obtain the final recognition result.
[0013] Preferably, it also includes a plaque recognition building module, connected to the plaque recognition module, including:
[0014] A training data collection unit, used for collecting a training data set including a plurality of standard vulnerable plaque samples with different plaque sizes;
[0015] The multi-scale feature extraction unit is used to adjust the scale of the convolution kernel of the standard convolutional neural network based on the plaque size in the standard vulnerable plaque sample input into the pre-constructed standard convolutional neural network during each training to perform multi-scale feature extraction training to obtain the plaque recognition model.
[0016] Preferably, it further comprises a data preprocessing module, connected to the plaque identification module, comprising:
[0017] A model building module, used to collect a real ultrasound image dataset containing vulnerable plaque samples and various other types of plaque samples, then build a generative adversarial network model containing a generator and a discriminator, and input the real ultrasound image dataset into the generative adversarial network model for training to obtain a plaque data enhancement model;
[0018] A data enhancement unit is connected to the model building module and is used to use the generator in the plaque data enhancement model to generate multiple synthetic ultrasound images and add them to the training data set to expand the data volume in the training data set.
[0019] Preferably, the image extraction unit comprises:
[0020] An iterative dilation subunit is used to iteratively expand the vulnerable plaque boundary toward the edge of the fused image multiple times using a morphological dilation operation based on the boundary recognition result in the fused image, until the pixel distance between the expanded vulnerable plaque boundary and the original vulnerable plaque boundary reaches a preset pixel distance, and then use the image surrounded by the expanded vulnerable plaque boundary in the fused image as a re-recognized image.
[0021] The present invention also provides a method for identifying vulnerable carotid plaques, which is applied to the above-mentioned vulnerable carotid plaque identification system, comprising:
[0022] Step S1, the carotid vulnerable plaque identification system obtains a carotid ultrasound image, a carotid CT image and a carotid magnetic resonance image obtained by scanning the same part of the patient's carotid artery with an ultrasound imaging device, a CT imaging device and a magnetic resonance imaging device, respectively, and then performs multi-modal image fusion on the carotid ultrasound image, the carotid CT image and the carotid magnetic resonance image to obtain a fused image;
[0023] Step S2, the carotid vulnerable plaque recognition system performs vulnerable plaque boundary recognition on the fused image, then extracts the vulnerable plaque based on the boundary recognition result for re-recognition, highlights the uncertain area in the re-recognition result and notifies the medical staff to review, then collects the review mark fed back by the medical staff, associates the review mark with the uncertain area in the re-recognition result to obtain the final recognition result.
[0024] Preferably, the step S2 comprises:
[0025] Step S21, the carotid vulnerable plaque recognition system inputs the fused image into a pre-built plaque recognition model to recognize the vulnerable plaque boundary in the fused image, and uses the portion of the fused image surrounded by the vulnerable plaque boundary as the boundary recognition result;
[0026] Step S22, the carotid vulnerable plaque recognition system, in the fused image, uses the boundary recognition result as a basic starting point, extends the vulnerable plaque boundary toward the edge of the fused image, and uses the image surrounded by the expanded vulnerable plaque boundary in the fused image as a re-recognized image;
[0027] In step S23, the carotid vulnerable plaque recognition system inputs the re-recognition image into the plaque recognition model for re-recognition to obtain the re-recognition result, and inputs the re-recognition image into the Bayesian neural network to obtain a recognition result in which the uncertain area in the re-recognition image is highlighted, and then collects the review mark fed back by the medical staff, and associates the review mark with the uncertain area in the re-recognition result to obtain the final recognition result.
[0028] Preferably, the method further includes a plaque recognition construction process, including:
[0029] Step A1, the carotid vulnerable plaque recognition system collects a training data set including a plurality of standard vulnerable plaque samples with different plaque sizes;
[0030] Step A2: During each training, the carotid vulnerable plaque recognition system adjusts the scale of the convolution kernel of the standard convolutional neural network based on the plaque size in the standard vulnerable plaque sample input into the pre-constructed standard convolutional neural network to perform multi-scale feature extraction training to obtain the plaque recognition model.
[0031] Preferably, before executing step A2, a data preprocessing process is also included, including:
[0032] Step B1, the carotid vulnerable plaque recognition system collects a real ultrasound image dataset containing vulnerable plaque samples and multiple other types of plaque samples, then constructs a generative adversarial network model containing a generator and a discriminator, and inputs the real ultrasound image dataset into the generative adversarial network model for training to obtain a plaque data enhancement model;
[0033] Step B2: the carotid vulnerable plaque identification system uses the generator in the plaque data enhancement model to generate multiple synthetic ultrasound images and add them to the training data set to expand the data volume in the training data set.
[0034] Preferably, the image expansion process in step S22 includes:
[0035] The carotid vulnerable plaque identification system, in the fused image, takes the boundary identification result as a basic starting point, and uses morphological dilation operation to iteratively expand the vulnerable plaque boundary toward the edge of the fused image for multiple times until the pixel distance between the expanded vulnerable plaque boundary and the original vulnerable plaque boundary reaches a preset pixel distance, and then uses the image surrounded by the expanded vulnerable plaque boundary in the fused image as a re-identified image.
[0036] The above technical solution has the following advantages or beneficial effects:
[0037] 1) The present invention uses a multimodal image fusion module, which can integrate the scan results of the same part (carotid artery) from ultrasound imaging equipment, CT imaging equipment and magnetic resonance imaging equipment. It fully utilizes the advantages of each imaging technology, namely the real-time nature of ultrasound, the high spatial resolution of CT, and the soft tissue contrast and non-radiation of magnetic resonance imaging. The fused image not only improves the image quality, but also enhances the ability to identify the internal components of the plaque, such as lipid core, fibrous cap and calcification, thereby overcoming the limitations of a single imaging modality in accurately assessing plaque vulnerability.
[0038] 2) The plaque recognition module automatically identifies the boundaries of vulnerable plaques in the fused image, improving the accuracy and efficiency of recognition. This module can also perform re-recognition based on the boundary recognition results, further refine the recognition results, highlight the uncertain areas and notify medical staff for review. This step introduces a manual review link, reducing the risk of misdiagnosis or missed diagnosis due to differences in equipment performance, individual differences in patients and different levels of operator experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 This is a structural schematic diagram of a carotid vulnerable plaque identification system in a preferred embodiment of the present invention;
[0040] Figure 2 A schematic flow chart of a method for identifying vulnerable carotid plaques in a preferred embodiment of the present invention;
[0041] Figure 3 Schematic diagram of a sub-process of step S2 in a preferred embodiment of the present invention;
[0042] Figure 4 The figure is a flow chart of the process of building a plaque recognition model in a preferred embodiment of the present invention;
[0043] Figure 5 The figure is a flow chart of the data preprocessing process in a preferred embodiment of the present invention. DETAILED DESCRIPTION
[0044] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The present invention is not limited to this embodiment, and other embodiments may also fall within the scope of the present invention as long as they conform to the gist of the present invention.
[0045] In a preferred embodiment of the present invention, based on the above problems existing in the prior art, a carotid vulnerable plaque identification system is provided. Figure 1 As shown, including:
[0046] The multimodal image fusion module 1 is used to obtain a carotid ultrasound image, a carotid CT image and a carotid magnetic resonance image obtained by scanning the same part of the patient's carotid artery with an ultrasound imaging device, a CT imaging device and a magnetic resonance imaging device, respectively, and then perform multimodal image fusion on the carotid ultrasound image, the carotid CT image and the carotid magnetic resonance image to obtain a fused image;
[0047] Plaque recognition module 2 is connected to plaque recognition module 1 and is used to identify the boundaries of vulnerable plaques in the fused image, and then extract the vulnerable plaques based on the boundary recognition results for re-identification, highlight the uncertain areas in the re-identification results and notify medical staff to review, and then collect the review marks fed back by the medical staff, and associate the review marks with the uncertain areas in the re-identification results to obtain the final recognition results.
[0048] Specifically, in this embodiment, the present invention effectively solves the technical problems mentioned in the background technology in the following ways:
[0049] Specifically, for the implementation of the multimodal image fusion module, we can combine the common practices and technical processes in the field of medical image processing to explain. The following is a simplified implementation to illustrate how to perform multimodal image fusion of ultrasound imaging, CT imaging and magnetic resonance imaging (MRI):
[0050] Example: Multimodal Image Fusion Process
[0051] 1. Data Acquisition
[0052] Ultrasound imaging: Use high-resolution ultrasound imaging equipment to scan the patient's carotid artery and obtain ultrasound images of the carotid artery. These images can show the arterial structure and blood flow dynamics in real time.
[0053] CT imaging: The patient is scanned using a CT scanner to obtain CT images of the carotid artery. CT images are known for their high spatial resolution and can clearly show the calcified part of the plaque.
[0054] Magnetic resonance imaging (MRI): The patient is scanned using an MRI device to obtain MRI images of the carotid artery. MRI images are superior in soft tissue contrast and can show in detail structures such as the lipid core and fibrous cap inside the plaque.
[0055] 2. Preprocessing
[0056] Noise and Artifact Removal: Preprocess the images acquired by each imaging modality to identify and remove noise and artifacts in the images to improve image quality.
[0057] Image registration: Select an image of one imaging modality as a reference image (such as MRI images, because of their superior soft tissue contrast), and perform geometric transformation on images of other imaging modalities (ultrasound images and CT images) to align them with the reference image in spatial position. This step is the key to image fusion, which ensures that images of different imaging modalities can be fused in the same coordinate system.
[0058] 3. Image Fusion
[0059] Fusion strategy selection: Select an appropriate image fusion strategy based on specific needs. Common fusion strategies include pixel-level fusion, feature-level fusion, and decision-level fusion. In the scenario of carotid plaque recognition in this embodiment, feature-level fusion may be more appropriate because it can reduce information redundancy while maintaining image features.
[0060] Fusion algorithm implementation: Use appropriate fusion algorithms (in this embodiment, a feature fusion network based on deep learning is preferred) to fuse the preprocessed images. These algorithms can automatically extract useful features from images of different imaging modalities and integrate them into the fused image.
[0061] Evaluation of fusion results: Evaluate the fused images to ensure that they can fully reflect the complex characteristics of the plaque and improve the accuracy of the assessment of plaque vulnerability.
[0062] 4. Fusion Image Application
[0063] The fused image is input into the plaque recognition module to automatically identify the boundaries of vulnerable plaques.
[0064] Based on the recognition results, vulnerable plaques are extracted and re-identified, and uncertain areas are highlighted and notified to medical staff for review.
[0065] The review marks provided by medical staff are collected and associated with the uncertain areas in the re-identification results to obtain the final recognition results.
[0066] Multimodal image fusion: The present invention uses a multimodal image fusion module, which can integrate the scan results of the same part (carotid artery) from ultrasound imaging equipment, CT imaging equipment and magnetic resonance imaging equipment. This fusion of multi-source data fully utilizes the advantages of each imaging technology, namely the real-time nature of ultrasound, the high spatial resolution of CT, and the soft tissue contrast and non-radiation of magnetic resonance imaging. The fused image not only improves the image quality, but also enhances the ability to identify the internal components of the plaque, such as lipid core, fibrous cap and calcification, thereby overcoming the limitations of a single imaging modality in accurately assessing plaque vulnerability.
[0067] Accurate plaque identification and review mechanism: The plaque recognition module automatically identifies the boundaries of vulnerable plaques in the fused image, improving the accuracy and efficiency of identification. This module can also perform re-identification based on the boundary recognition results, further refine the recognition results, highlight the uncertain areas and notify medical staff to review. This step introduces a manual review link, reducing the risk of misdiagnosis or missed diagnosis due to differences in equipment performance, individual differences in patients and different levels of operator experience.
[0068] The review marks fed back by medical staff are associated with the uncertain areas in the re-identification results, and ultimately a more accurate and reliable plaque identification result is obtained. This human-machine combination not only takes advantage of the efficiency of computer automated processing, but also retains the experience and judgment of professional medical staff, achieving accurate identification of vulnerable carotid artery plaques.
[0069] In summary, this technical solution effectively solves the limitations of ultrasound imaging in assessing plaque vulnerability mentioned in the background technology and the problem that a single imaging modality is difficult to fully reflect plaque characteristics through multimodal image fusion and precise plaque identification and review mechanism. It achieves accurate identification of vulnerable carotid artery plaques and provides a more reliable basis for the formulation of clinical intervention strategies.
[0070] In a preferred embodiment of the present invention, Figure 1 As shown, the plaque identification module 2 includes:
[0071] The plaque recognition unit 21 is used to input the fused image into a pre-built plaque recognition model to recognize the vulnerable plaque boundary in the fused image, and take the part of the fused image surrounded by the vulnerable plaque boundary as the boundary recognition result;
[0072] The image extraction unit 22 is connected to the plaque recognition unit 21 and is used to extend the boundary of the vulnerable plaque toward the edge of the fused image based on the boundary recognition result in the fused image, and use the image surrounded by the expanded vulnerable plaque boundary in the fused image as the re-recognition image;
[0073] The uncertain marking unit 23 is connected to the image extraction unit 22, and is used to input the re-identified image into the plaque recognition model for re-identification to obtain a re-identification result, and input the re-identified image into the Bayesian neural network to obtain a recognition result in which the uncertain area in the re-identified image is highlighted, and then collect the review mark fed back by the medical staff, and associate the review mark with the uncertain area in the re-identification result to obtain the final recognition result.
[0074] Specifically, in this embodiment, the following is an embodiment to illustrate how to implement the function of the plaque recognition module:
[0075] 1. Plaque identification unit
[0076] Step 1: Build a plaque recognition model
[0077] A large number of multimodal fusion images of carotid arteries with labeled vulnerable plaque boundaries are used as training data.
[0078] Select a suitable deep learning framework (TensorFlow or PyTorch is used in this embodiment) and model architecture (Convolutional Neural Network (CNN) is used in this embodiment) to build a plaque recognition model.
[0079] The model is trained until its performance on the validation set reaches a preset accuracy or loss value.
[0080] Step 2: Identify vulnerable plaque boundaries
[0081] The multimodal fused image is input into the trained plaque recognition model.
[0082] The model automatically outputs the boundaries of vulnerable patches, which are usually represented with pixel-level or sub-pixel accuracy.
[0083] The part of the fused image surrounded by the vulnerable plaque boundary is taken as the boundary recognition result, that is, the initially identified vulnerable plaque area.
[0084] 2. Image extraction unit
[0085] Step 1: Expand the boundaries of vulnerable patches
[0086] Based on the boundary identification results, an algorithm is designed to expand the boundary of vulnerable patches.
[0087] The expanded boundaries can be adjusted according to actual conditions and clinical needs, such as based on the shape, size or characteristics of the plaque or surrounding tissue.
[0088] The expanded boundary should cover possible vulnerable plaque areas more comprehensively while avoiding the misinclusion of non-plaque areas.
[0089] Step 2: Extract the re-identification image
[0090] The image surrounded by the expanded vulnerable plaque boundary in the fused image is used as the re-identification image.
[0091] Make sure the re-ID image contains enough contextual information for more accurate recognition in subsequent steps.
[0092] 3. Uncertain marking unit
[0093] Step 1: Re-identify vulnerable plaques
[0094] The re-identified image is re-input into the plaque recognition model for re-identification.
[0095] This identification aims to further confirm and refine the boundaries and characteristics of vulnerable plaques.
[0096] Step 2: Highlight the uncertain areas
[0097] The re-identification images are fed into a pre-trained Bayesian neural network.
[0098] The Bayesian neural network can output the uncertainty probability that each pixel or region belongs to a vulnerable patch.
[0099] According to the uncertainty probability, the uncertain area in the re-identified image is highlighted. The highlighting method can be changing the color, adding a border or flashing.
[0100] Step 3: Medical staff review and final identification
[0101] Show the highlighted re-identification image to medical staff.
[0102] Medical staff review the highlighted areas based on their professional knowledge and experience and give review marks (such as confirmation, modification or neglect).
[0103] The verification mark is associated with the uncertain area in the re-identification result to obtain the final vulnerable plaque identification result.
[0104] Through the above steps, the plaque recognition module can accurately identify vulnerable plaques in the carotid artery, and improve the accuracy and reliability of recognition through highlighting of uncertain areas and review by medical staff.
[0105] In a preferred embodiment of the present invention, Figure 1 As shown, it also includes a plaque recognition building module 3, connected to the plaque recognition module 2, including:
[0106] A training data collection unit 31 is used to collect a training data set including a plurality of standard vulnerable plaque samples with different plaque sizes;
[0107] The multi-scale feature extraction unit 32 is connected to the training data acquisition unit 31 and is used to adjust the scale of the convolution kernel of the standard convolutional neural network based on the plaque size in the standard vulnerable plaque sample input into the pre-constructed standard convolutional neural network during each training to perform multi-scale feature extraction training to obtain a plaque recognition model.
[0108] Specifically, in this embodiment, the construction of the plaque recognition model is based on a standard convolutional neural network;
[0109] Step 1: Sample collection
[0110] Carotid artery image data containing vulnerable plaques should be collected from multiple medical imaging centers or databases. These images should be annotated by professional doctors to clearly mark the location and boundaries of vulnerable plaques.
[0111] Samples with different patch sizes were screened out to ensure that the training dataset had sufficient diversity and representativeness.
[0112] Step 2: Data preprocessing
[0113] The collected images are preprocessed, including denoising, normalization, contrast enhancement and other operations, to improve image quality and reduce noise interference during training.
[0114] Crop the images to a uniform size as needed to facilitate subsequent processing.
[0115] Step 3: Build a training dataset
[0116] The preprocessed images are divided into training set, validation set and test set according to a certain ratio. Usually, the training set occupies the majority (such as 70%), and the validation set and test set each occupy a small part (such as 15%).
[0117] Generate a label for each sample, which should contain the location information of the vulnerable patch (such as bounding box coordinates) or pixel-level annotation.
[0118] 2. Multi-scale feature extraction unit
[0119] Step 1: Build a basic convolutional neural network
[0120] A standard convolutional neural network suitable for image recognition tasks is selected as a basic model. In this embodiment, one of VGG, ResNet or DenseNet can be selected.
[0121] Determine the basic architecture of the network, including convolutional layers, pooling layers, activation functions, and fully connected layers.
[0122] Step 2: Multi-scale feature extraction strategy
[0123] Design a multi-scale feature extraction strategy to accommodate patches of different sizes. This can be achieved in several ways:
[0124] Dynamic convolution kernel adjustment: During the training process, the scale of the convolution kernel is dynamically adjusted according to the size of the patches in the input samples. The optimal correspondence between the patch size and the convolution kernel scale can be obtained through multiple manual tests, and can also be obtained in the following way:
[0125] In the patch recognition task, the adaptive mechanism can dynamically adjust the scale of the convolution kernel according to the size of the input patch to improve the accuracy and robustness of feature extraction. The following is an example based on the adaptive mechanism to illustrate the correspondence between the patch size and the "perception" scale of the convolution kernel (note that the "perception" scale here does not directly change the physical size of the convolution kernel, but simulates the receptive field of different scales by other means):
[0126] Adaptive multi-scale receptive field strategy
[0127] Multi-scale input preprocessing:
[0128] Before inputting into the network, the patch image is preprocessed at multiple scales. This can be achieved by scaling the image to different sizes (such as original size, 0.5 times, 1.5 times, etc.).
[0129] Each scale image is used as an input branch of the network.
[0130] Shared convolutional layers:
[0131] Use a shared convolutional layer to process input images of all scales. This convolutional layer uses a fixed convolution kernel scale (such as 3x3 or 5x5), but can extract the basic features of input images of different scales.
[0132] Adaptive pooling or attention mechanism:
[0133] After the shared convolutional layer, an adaptive pooling layer or attention mechanism is introduced to simulate receptive fields of different scales.
[0134] The adaptive pooling layer can dynamically adjust the size or step size of the pooling window according to the size of the input feature map, thereby retaining more detail information or ignoring unimportant background information.
[0135] The attention mechanism can focus on important areas in the image by learning weight distribution, which may be related to the size and position of the patches.
[0136] Feature fusion and decision making:
[0137] The feature maps obtained after the input branches of different scales pass through the convolution layer and the adaptive pooling / attention mechanism are fused.
[0138] The fused feature map contains information at different scales, which helps to identify patches more accurately.
[0139] Finally, a decision is made through a fully connected layer or a convolutional layer to output the classification result or location information of the patch.
[0140] Although the physical size of the convolution kernel is not directly changed in this example, the combination of multi-scale input and adaptive pooling / attention mechanism enables effective recognition of patches of different sizes. This method can be seen as indirectly simulating the change of convolution kernel scale by adjusting the receptive field.
[0141] The multi-scale feature extraction strategy can also be implemented as follows:
[0142] Multi-branch network: Construct multiple parallel convolution branches, each branch uses convolution kernels of different scales. The outputs of these branches are merged in subsequent layers to fuse features of different scales.
[0143] Spatial Pyramid Pooling (SPP): The SPP layer is introduced after a certain stage of the network, which can pool input feature maps of any size and output a feature vector of fixed length.
[0144] Step 3: Model training
[0145] The base convolutional neural network is pre-trained using the training dataset to learn basic image features.
[0146] Introduce a multi-scale feature extraction strategy and continue to train the model. During the training process, dynamically adjust the scale of the convolution kernel according to the size of the patch in the input sample or use multi-branch networks, SPP and other methods to extract multi-scale features.
[0147] Monitor the performance on the validation set and adjust hyperparameters (such as learning rate, batch size, number of training epochs, etc.) as needed.
[0148] In a preferred embodiment of the present invention, a data preprocessing module 4 is also included, such as Figure 1 As shown, the plaque recognition module 2 is connected, including:
[0149] A model building module 41 is used to collect a real ultrasound image dataset including vulnerable plaque samples and various other types of plaque samples, and then build a generative adversarial network model including a generator and a discriminator, and input the real ultrasound image dataset into the generative adversarial network model for training to obtain a plaque data enhancement model;
[0150] The data enhancement unit 42 is connected to the model building module 41 and is used to use the generator in the plaque data enhancement model to generate multiple synthetic ultrasound images and add them to the training data set to expand the data volume in the training data set.
[0151] Specifically, the data preprocessing module in this embodiment significantly enhances and expands the training data set in the plaque recognition task by introducing submodules including a model building module and a data enhancement unit. This design brings the following beneficial effects:
[0152] Improve data diversity: By collecting a dataset of real ultrasound images containing vulnerable plaque samples and samples of various other types of plaques, the model building module is able to learn the characteristics of different types of plaques. This helps the generative adversarial network (GAN) model capture a richer data distribution during training, thereby generating more diverse and realistic synthetic ultrasound images.
[0153] Expanding the training dataset: The data augmentation unit uses the generator in the trained GAN model to generate a large number of synthetic ultrasound images. These images are added to the original training dataset, significantly expanding the size of the dataset. This is especially important for deep learning models, because larger datasets can usually reduce the risk of overfitting and improve the generalization ability of the model.
[0154] In a preferred embodiment of the present invention, the image extraction unit 22 includes:
[0155] The iterative dilation subunit 221 is used to iteratively expand the vulnerable plaque boundary toward the edge of the fused image multiple times using the morphological dilation operation based on the boundary recognition result in the fused image, until the pixel distance between the expanded vulnerable plaque boundary and the original vulnerable plaque boundary reaches a preset pixel distance, and then the image surrounded by the expanded vulnerable plaque boundary in the fused image is used as the re-recognition image.
[0156] Specifically, in this embodiment, boundary expansion adopts an iterative expansion method, which has significant beneficial effects in image processing, especially in medical image analysis such as vulnerable plaque identification. Improve the accuracy of boundary identification: Through the morphological expansion operation, the iterative expansion subunit can gradually expand the boundary of the vulnerable plaque outward until it reaches a preset pixel distance. This process helps to smooth the boundary and reduce boundary identification errors caused by image noise or insufficient resolution. Multiple iterations ensure the continuity and consistency of boundary expansion, so that the finally identified vulnerable plaque boundary is closer to the actual situation.
[0157] Enhanced recognition of complex structures: In medical images, vulnerable plaques often have complex morphology and boundaries. The iterative dilation subunit can better capture these complex structures by gradually expanding the boundaries, improving the system's recognition of complex plaques. In particular, for vulnerable plaques with blurred boundaries or closely connected to other tissues, the iterative dilation operation helps separate them from the background and improves recognition accuracy.
[0158] More specifically, in one embodiment, iterative expansion is performed in the following manner:
[0159] Initial boundary identification:
[0160] The fused image is preliminarily processed using a plaque recognition module (a convolutional neural network is used in this embodiment) to identify the preliminary boundaries of vulnerable plaques.
[0161] The identified boundaries are used as input to the iterative dilation subunit.
[0162] Set iterative expansion parameters:
[0163] Determine the initial conditions of the iterative dilation, including the kernel size of the dilation operation (ie, the size of the structure element), the number of iterations, and a preset pixel distance threshold.
[0164] The choice of core size should be appropriately adjusted according to the size and shape of the plaque to ensure the effectiveness of the expansion operation.
[0165] The preset pixel distance threshold is used to control the stopping condition of iterative expansion, that is, when the pixel distance between the expanded patch boundary and the original boundary reaches or exceeds the threshold, the iteration is stopped.
[0166] Perform an iterative dilation operation:
[0167] Taking the initially identified boundary as the starting point, the boundary is expanded using the morphological dilation operation.
[0168] In each iteration, the dilated boundary is used as the input of the next iteration and the dilation operation continues.
[0169] The iterative process is repeated until the pixel distance between the expanded patch boundary and the original boundary reaches a preset pixel distance threshold.
[0170] Generate re-identification image:
[0171] The final expanded patch boundary is used as the new identification boundary.
[0172] The area enclosed by the boundary in the fused image is output as the re-recognition image.
[0173] In another embodiment, the initially identified vulnerable plaque boundary is a circular area with a radius of R. We set the kernel size to be a 3x3 square structure element, and the preset pixel distance threshold is 5 pixels.
[0174] In the first iteration, the original border is dilated using a 3x3 structure element, resulting in a new, slightly larger border.
[0175] This process is repeated, and after each iteration the pixel distance between the new boundary and the original boundary is checked.
[0176] When the distance reaches or exceeds 5 pixels, the iteration is stopped and the boundary at this time is taken as the final recognition boundary.
[0177] Finally, the area surrounded by the final boundary in the fused image is used as the re-identification image for subsequent analysis.
[0178] The present invention also provides a method for identifying vulnerable carotid plaques, which is applied to the above-mentioned vulnerable carotid plaque identification system. Figure 2 As shown, including:
[0179] Step S1, the carotid vulnerable plaque recognition system obtains a carotid ultrasound image, a carotid CT image and a carotid MRI image obtained by scanning the same part of the patient's carotid artery with an ultrasound imaging device, a CT imaging device and a MRI device, respectively, and then performs multi-modal image fusion on the carotid ultrasound image, the carotid CT image and the carotid MRI image to obtain a fused image;
[0180] In step S2, the carotid vulnerable plaque recognition system performs vulnerable plaque boundary recognition on the fused image, then extracts the vulnerable plaque based on the boundary recognition result for re-recognition, highlights the uncertain area in the re-recognition result and notifies the medical staff for review, then collects the review mark fed back by the medical staff, associates the review mark with the uncertain area in the re-recognition result to obtain the final recognition result.
[0181] In a preferred embodiment of the present invention, Figure 3 As shown, step S2 includes:
[0182] Step S21, the carotid vulnerable plaque recognition system inputs the fused image into a pre-built plaque recognition model to recognize the vulnerable plaque boundary in the fused image, and takes the portion of the fused image surrounded by the vulnerable plaque boundary as the boundary recognition result;
[0183] Step S22, the carotid vulnerable plaque recognition system uses the boundary recognition result as a basic starting point in the fused image to extend the vulnerable plaque boundary toward the edge of the fused image, and uses the image surrounded by the expanded vulnerable plaque boundary in the fused image as a re-recognized image;
[0184] In step S23, the carotid vulnerable plaque recognition system inputs the re-recognition image into the plaque recognition model for re-recognition to obtain a re-recognition result, and inputs the re-recognition image into the Bayesian neural network to obtain a recognition result in which the uncertain area in the re-recognition image is highlighted, and then collects the review mark fed back by the medical staff, and associates the review mark with the uncertain area in the re-recognition result to obtain the final recognition result.
[0185] In a preferred embodiment of the present invention, a plaque recognition construction process is also included, such as Figure 4 As shown, including:
[0186] Step A1, the carotid vulnerable plaque recognition system collects a training data set containing a plurality of standard vulnerable plaque samples with different plaque sizes;
[0187] Step A2: During each training, the carotid vulnerable plaque recognition system adjusts the scale of the convolution kernel of the standard convolutional neural network based on the plaque size in the standard vulnerable plaque sample input into the pre-built standard convolutional neural network, performs multi-scale feature extraction training, and obtains a plaque recognition model.
[0188] In a preferred embodiment of the present invention, before executing step A2, as Figure 5 As shown, the data preprocessing process is also included, including:
[0189] Step B1, the carotid vulnerable plaque recognition system collects a real ultrasound image dataset containing vulnerable plaque samples and various other types of plaque samples, then builds a generative adversarial network model containing a generator and a discriminator, and inputs the real ultrasound image dataset into the generative adversarial network model for training to obtain a plaque data enhancement model;
[0190] Step B2: the carotid vulnerable plaque recognition system uses the generator in the plaque data enhancement model to generate multiple synthetic ultrasound images and add them to the training data set to expand the data volume in the training data set.
[0191] In a preferred embodiment of the present invention, the image expansion process in step S22 includes:
[0192] In the fused image, the carotid vulnerable plaque recognition system takes the boundary recognition result as the starting point and uses the morphological dilation operation to iteratively expand the vulnerable plaque boundary toward the edge of the fused image multiple times until the pixel distance between the expanded vulnerable plaque boundary and the original vulnerable plaque boundary reaches the preset pixel distance. The image surrounded by the expanded vulnerable plaque boundary in the fused image is then used as the re-recognition image.
[0193] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the contents of this specification and illustrations should be included in the protection scope of the present invention.
Claims
1. A carotid vulnerable plaque identification system, characterized in that: include: a multimodal image fusion module, for respectively acquiring a carotid ultrasound image, a carotid CT image and a carotid magnetic resonance image obtained by scanning the same part of the carotid artery of a patient using an ultrasound imaging device, a CT imaging device and a magnetic resonance imaging device, and then performing multimodal image fusion on the carotid ultrasound image, the carotid CT image and the carotid magnetic resonance image to obtain a fused image; A plaque recognition module is connected to the plaque recognition module and is used to perform boundary recognition of vulnerable plaques on the fused image, then extract vulnerable plaques based on the boundary recognition results for re-recognition, highlight the uncertain areas in the re-recognition results and notify medical staff for review, then collect the review marks fed back by the medical staff, and associate the review marks with the uncertain areas in the re-recognition results to obtain the final recognition result.
2. The carotid vulnerable plaque identification system according to claim 1, characterized in that: The plaque identification module comprises: a plaque recognition unit, configured to input the fused image into a pre-built plaque recognition model to recognize the boundary of vulnerable plaques in the fused image, and take the portion of the fused image surrounded by the boundary of vulnerable plaques as the boundary recognition result; an image extraction unit connected to the plaque recognition unit, and configured to extend the boundary of the vulnerable plaque toward the edge of the fused image based on the boundary recognition result in the fused image, and use the image surrounded by the expanded vulnerable plaque boundary in the fused image as a re-recognized image; An uncertain marking unit is connected to the image extraction unit, and is used to input the re-recognition image into the plaque recognition model for re-recognition to obtain the re-recognition result, and input the re-recognition image into the Bayesian neural network to obtain a recognition result in which the uncertain area in the re-recognition image is highlighted, and then collect the review mark fed back by the medical staff, and associate the review mark with the uncertain area in the re-recognition result to obtain the final recognition result.
3. The carotid vulnerable plaque identification system according to claim 2, characterized in that: Also included is a plaque identification building module, connected to the plaque identification module, including: A training data collection unit, used for collecting a training data set including a plurality of standard vulnerable plaque samples with different plaque sizes; A multi-scale feature extraction unit is connected to the training data acquisition unit and is used to adjust the scale of the convolution kernel of the standard convolutional neural network based on the plaque size in the standard vulnerable plaque sample input into the pre-constructed standard convolutional neural network during each training to perform multi-scale feature extraction training to obtain the plaque recognition model.
4. The carotid vulnerable plaque identification system according to claim 3, characterized in that: It also includes a data preprocessing module, connected to the plaque identification module, including: A model building module, used to collect a real ultrasound image dataset containing vulnerable plaque samples and various other types of plaque samples, then build a generative adversarial network model containing a generator and a discriminator, and input the real ultrasound image dataset into the generative adversarial network model for training to obtain a plaque data enhancement model; A data enhancement unit is connected to the model building module and is used to use the generator in the plaque data enhancement model to generate multiple synthetic ultrasound images and add them to the training data set to expand the data volume in the training data set.
5. The carotid vulnerable plaque identification system according to claim 3, characterized in that: The image extraction unit comprises: An iterative dilation subunit is used to iteratively expand the vulnerable plaque boundary toward the edge of the fused image multiple times using a morphological dilation operation based on the boundary recognition result in the fused image, until the pixel distance between the expanded vulnerable plaque boundary and the original vulnerable plaque boundary reaches a preset pixel distance, and then use the image surrounded by the expanded vulnerable plaque boundary in the fused image as a re-recognized image.
6. A method for identifying vulnerable carotid artery plaques, characterized in that: A carotid vulnerable plaque identification system as claimed in any one of claims 1 to 5, comprising: Step S1, the carotid vulnerable plaque identification system obtains a carotid ultrasound image, a carotid CT image and a carotid magnetic resonance image obtained by scanning the same part of the patient's carotid artery with an ultrasound imaging device, a CT imaging device and a magnetic resonance imaging device, respectively, and then performs multi-modal image fusion on the carotid ultrasound image, the carotid CT image and the carotid magnetic resonance image to obtain a fused image; Step S2, the carotid vulnerable plaque recognition system performs vulnerable plaque boundary recognition on the fused image, then extracts the vulnerable plaque based on the boundary recognition result for re-recognition, highlights the uncertain area in the re-recognition result and notifies the medical staff to review, then collects the review mark fed back by the medical staff, associates the review mark with the uncertain area in the re-recognition result to obtain the final recognition result.
7. The method for identifying vulnerable carotid plaques according to claim 6, characterized in that: The step S2 comprises: Step S21, the carotid vulnerable plaque recognition system inputs the fused image into a pre-built plaque recognition model to recognize the vulnerable plaque boundary in the fused image, and uses the portion of the fused image surrounded by the vulnerable plaque boundary as the boundary recognition result; Step S22, the carotid vulnerable plaque recognition system, in the fused image, uses the boundary recognition result as a basic starting point, extends the vulnerable plaque boundary toward the edge of the fused image, and uses the image surrounded by the expanded vulnerable plaque boundary in the fused image as a re-recognized image; In step S23, the carotid vulnerable plaque recognition system inputs the re-recognition image into the plaque recognition model for re-recognition to obtain the re-recognition result, and inputs the re-recognition image into the Bayesian neural network to obtain a recognition result in which the uncertain area in the re-recognition image is highlighted, and then collects the review mark fed back by the medical staff, and associates the review mark with the uncertain area in the re-recognition result to obtain the final recognition result.
8. The method for identifying vulnerable carotid plaque according to claim 7, characterized in that: Also included is the plaque identification build process, including: Step A1, the carotid vulnerable plaque recognition system collects a training data set including a plurality of standard vulnerable plaque samples with different plaque sizes; Step A2: During each training, the carotid vulnerable plaque recognition system adjusts the scale of the convolution kernel of the standard convolutional neural network based on the plaque size in the standard vulnerable plaque sample input into the pre-constructed standard convolutional neural network to perform multi-scale feature extraction training to obtain the plaque recognition model.
9. The method for identifying vulnerable carotid plaques according to claim 8, characterized in that: Before executing step A2, a data preprocessing process is also included, including: Step B1, the carotid vulnerable plaque recognition system collects a real ultrasound image dataset containing vulnerable plaque samples and multiple other types of plaque samples, then constructs a generative adversarial network model containing a generator and a discriminator, and inputs the real ultrasound image dataset into the generative adversarial network model for training to obtain a plaque data enhancement model; Step B2: the carotid vulnerable plaque identification system uses the generator in the plaque data enhancement model to generate multiple synthetic ultrasound images and add them to the training data set to expand the data volume in the training data set.
10. The method for identifying vulnerable carotid plaques according to claim 8, characterized in that: The image expansion process in step S22 includes: The carotid vulnerable plaque identification system, in the fused image, takes the boundary identification result as a basic starting point, and uses morphological dilation operation to iteratively expand the vulnerable plaque boundary toward the edge of the fused image for multiple times until the pixel distance between the expanded vulnerable plaque boundary and the original vulnerable plaque boundary reaches a preset pixel distance, and then uses the image surrounded by the expanded vulnerable plaque boundary in the fused image as a re-identified image.