A method and system for carotid artery vulnerability grading based on multimodal radiomics

Abstract

This invention belongs to the field of medical image recognition and provides a method and system for carotid artery vulnerability grading based on multimodal radiomics. The method includes: acquiring at least one of carotid ultrasound images and MRI images; and, based on at least one of the carotid ultrasound images and MRI images, employing a carotid plaque grading model to obtain the vulnerability level of the carotid plaque. The carotid plaque grading model includes a multi-scale feature fusion network and an attention classification network. The multi-scale feature fusion network is used to fuse extracted features from carotid ultrasound image samples and carotid MRI image samples to obtain fused features. The attention classification network is used to obtain the vulnerability level of the carotid plaque based on the fused features and the carotid ultrasound image sample features / carotid MRI image sample features.

Description

Technical Field

[0001] This invention belongs to the field of medical image recognition, and in particular relates to a method and system for classifying carotid artery vulnerability based on multimodal radiomics. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] The formation of carotid artery plaques is a common pathological phenomenon in the process of carotid atherosclerosis and is closely related to the occurrence of ischemic cerebrovascular disease. Autopsy studies have confirmed that, on the basis of carotid atherosclerosis, the unpredictable and sudden rupture of vulnerable plaques, platelet activation, and thrombosis are important pathogenic mechanisms of ischemic cerebrovascular disease. Carotid artery ultrasound is a routine examination method for carotid artery plaques, which can measure the thickness and length of the plaques and make a preliminary judgment on the plaque composition based on the nature of the echoes.

[0004] Pathological classification of vulnerable plaques: ① Vulnerable plaques with a tendency to rupture: large lipid core, thin fibrous cap, and infiltration of numerous inflammatory cells, including macrophages; ② Fibrous rupture, secondary thrombus formation on the surface, resulting in incomplete vascular occlusion, with early thrombus organization; ③ Plaques rich in glycoprotein matrix and smooth muscle cells, with a surface prone to erosion; ④ Plaques with significant surface erosion, with surface platelet thrombosis, resulting in incomplete vascular occlusion; ⑤ IPH causes plaques to enlarge significantly in a short period of time, and the degree of luminal stenosis deteriorates rapidly; ⑥ Calcified nodules protrude into the lumen; ⑦ Chronic stenotic plaques with severe calcification, old thrombus formation, and eccentric lumen.

[0005] Traditional ultrasound is currently the most convenient and widely studied method for identifying vulnerable plaques in clinical practice. It can provide information such as intima-media thickness (IMT), location, number, size, internal echogenicity, diameter, and degree of luminal stenosis of the plaque. It can also detect plaque calcification, hemorrhage, necrosis, lipids, and fibrous tissue. Based on the echogenicity of the plaque's interior, plaques can be classified as homogeneous hypoechoic, mesoechoic, hyperechoic, and heterogeneous. Hypoechoic areas are considered to be predominantly lipid deposits, irregular anechoic areas within the plaque are considered hemorrhage areas, and areas with uneven intimal surfaces, local depressions, and predominantly hypoechoic echogenicity are considered ulceration areas. Vulnerable plaques are predominantly hypoechoic, while asymptomatic plaques are predominantly hyperechoic and heterogeneous.

[0006] Two-dimensional ultrasound has been recognized in clinical applications, but it has certain limitations: ① Subjective judgment of the examinee has a significant impact, resulting in large differences in results among different operators; ② It has poor ability to identify plaque components, and the ultrasound resolution is insufficient to accurately identify lipid-rich plaques (IPH) and neovascularization within plaques; ③ It is difficult to detect vessels above the carotid bifurcation. In patients with deep locations, obesity, or poor lumen transmission, coupled with the presence of various artifacts, two-dimensional ultrasound is prone to missing some hypoechoic plaques. Furthermore, hyperechoic calcified plaques can interfere with sound waves, affecting image quality; ④ Sonographic images show a cross-section of a local area, making it difficult to clearly display the spatial location and configuration of plaques on a single image, thus limiting its ability to identify vulnerable plaques.

[0007] High-resolution carotid magnetic resonance imaging (HRMRI) is gaining increasing attention for evaluating atherosclerotic plaques in the carotid arteries. This examination not only reveals the degree of vascular stenosis, plaque size, and ulceration, but also provides vulnerability indicators such as plaque composition, fibrous cap thickness, and vessel wall characteristics. It has become the most promising auxiliary diagnostic tool for identifying vulnerable plaques in clinical practice. However, MRI also has limitations due to its high cost, inconvenient operation, and radiation exposure. Summary of the Invention

[0008] To address the technical problems mentioned above, this invention provides a carotid artery vulnerability grading method and system based on multimodal radiomics. By constructing a vulnerable plaque grading model and inputting at least one of carotid ultrasound images and MRI images into the model, the vulnerability level of carotid artery plaques is obtained with ultra-high accuracy.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] The first aspect of the present invention provides a method for grading carotid artery vulnerability based on multimodal radiomics.

[0011] A carotid artery vulnerability grading method based on multimodal radiomics includes:

[0012] Obtain at least one of carotid artery ultrasound images and MRI images;

[0013] Based on at least one of carotid ultrasound images and MRI images, a carotid plaque grading model is used to obtain the vulnerability level of carotid plaques;

[0014] The carotid plaque grading model includes a multi-scale feature fusion network and an attention classification network. The multi-scale feature fusion network is used to fuse extracted features from carotid ultrasound image samples and carotid MRI image samples to obtain fused features. The attention classification network is used to obtain the vulnerability level of carotid plaques based on the fused features and the carotid ultrasound image sample features / carotid MRI image sample features.

[0015] A second aspect of the present invention provides a carotid artery vulnerability grading system based on multimodal radiomics.

[0016] A carotid artery vulnerability grading system based on multimodal radiomics includes:

[0017] The data acquisition module is configured to acquire at least one of a carotid ultrasound image and an MRI image.

[0018] The output module is configured to: obtain the vulnerability level of carotid plaques based on at least one of carotid ultrasound images and MRI images using a carotid plaque grading model;

[0019] The model building module is configured as follows: the carotid plaque grading model includes a multi-scale feature fusion network and an attention classification network; wherein, the multi-scale feature fusion network is used to fuse the extracted features of carotid ultrasound image samples and carotid MRI image samples based on carotid ultrasound image samples and carotid MRI image samples to obtain fused features; the attention classification network is used to obtain the vulnerability level of carotid plaques based on the fused features and the carotid ultrasound image sample features / carotid MRI image sample features.

[0020] A third aspect of the present invention provides a computer-readable storage medium.

[0021] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the carotid artery vulnerability grading method based on multimodal radiomics as described in the first aspect above.

[0022] A fourth aspect of the present invention provides a computer device.

[0023] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the carotid artery vulnerability grading method based on multimodal radiomics as described in the first aspect above.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] This invention constructs a vulnerable plaque grading model and obtains a highly accurate carotid artery plaque vulnerability level by inputting at least one of carotid ultrasound images and MRI images into the model.

[0026] This invention uses MRI to identify tags and combines the portability of ultrasound equipment to quantitatively analyze the vulnerability of carotid artery plaques, thereby guiding the next step of the treatment process. Attached Figure Description

[0027] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0028] Figure 1 This is a framework diagram of the carotid artery vulnerability grading method based on multimodal radiomics as shown in this invention;

[0029] Figure 2 This is a structural diagram of the model training module shown in this invention. Detailed Implementation

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0031] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0032] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0033] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of this disclosure. It should be noted that each block in a flowchart or block diagram may represent a module, segment, or portion of code, which may include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0034] Example 1

[0035] like Figure 1 As shown, this embodiment provides a carotid artery vulnerability grading method based on multimodal radiomics. This embodiment uses the application of this method to a server as an example for illustration. It is understood that this method can also be applied to terminals, and can also be applied to systems including terminals, servers, and other components, and implemented through interaction between the terminal and the server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communication, middleware services, domain name services, CDN security services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. In this embodiment, the method includes the following steps:

[0036] Obtain at least one of carotid artery ultrasound images and MRI images;

[0037] Based on at least one of carotid ultrasound images and MRI images, a carotid plaque grading model is used to obtain the vulnerability level of carotid plaques;

[0038] The carotid plaque grading model includes a multi-scale feature fusion network and an attention classification network. The multi-scale feature fusion network is used to fuse extracted features from carotid ultrasound image samples and carotid MRI image samples to obtain fused features. The attention classification network is used to obtain the vulnerability level of carotid plaques based on the fused features and the carotid ultrasound image sample features / carotid MRI image sample features.

[0039] like Figure 1 As shown, this embodiment includes: a data acquisition module 1, a label creation module 2, a feature fusion module 3, and a result output module 4.

[0040] Data acquisition module 1 is used to acquire ultrasound image information and MRI image information of patients.

[0041] The label creation module 2 is used to create vulnerability grading labels based on ultrasound and MRI findings.

[0042] Model training module 3 is used to train classification models based on multimodal and multitask modes.

[0043] Result output module 4 is used to output ultrasound grading results and nuclear magnetic resonance grading results.

[0044] The ultrasound acquisition module 1 includes, but is not limited to, a handheld ultrasound device. Unlike the traditional ultrasound device with a host and probe, the host is reduced to a small circuit board built into the probe, making it just a "probe" that is equivalent to an ultrasound machine. It can be displayed using a mobile phone or tablet with an ultrasound APP installed. The image is transmitted to the mobile phone / tablet via the probe's built-in Wi-Fi.

[0045] The label creation module 2 refers to the process where three professional MRI experts classify the vulnerability of MRI images based on the images. According to the MRI classification standard for carotid atherosclerotic plaques, the plaques are divided into the following grades: Grade 0: arterial wall thickness is close to normal, without calcification; Grade 1: diffuse thickening of the intima or small, non-calcified, eccentric plaques; Grade 2: plaques containing large necrotic lipid cores and covered with fibrous caps, which may be accompanied by a small amount of calcification. On T1W and PDWI, they may appear as uniform high signal, while on T2WI, they appear as non-uniform high signal; Grade 3: plaque surface ulceration or intraplaque hemorrhage, thrombus formation; Grade 4: fibrotic plaques without lipid cores, which may be accompanied by small calcifications. Based on ultrasound images, three ultrasound experts graded the vulnerability of the ultrasound images. The main characteristics of ultrasound images are: uneven echo, uneven inner membrane surface, local depression, and predominantly low echo. When any of the characteristics are present or absent, the vulnerability is grade 0; when one characteristic is present, it is grade 1; when two characteristics are present, it is grade 2; when three characteristics are present, it is grade 3; and when four characteristics are present, it is grade 4.

[0046] The model training module 3, as described in Figure 2 As shown, the carotid MRI image and carotid ultrasound image are first fed into a multi-scale feature fusion network. In the multi-scale feature fusion network, the features of the three convolutional layers in the two images are fused. Then, the fused result and the original high-level convolutional features are fed into their respective attention classification networks. Finally, the vulnerability classification result is output.

[0047] Specifically, the process of obtaining the fusion features includes:

[0048] A multi-scale feature fusion network is used to extract the first-layer features of carotid ultrasound image samples and carotid MRI image samples respectively. Then, the first-layer features of the carotid ultrasound image samples and the first-layer features of the carotid MRI image samples are fused to obtain the first-layer fused features.

[0049] A multi-scale feature fusion network was used to extract the second-layer features of carotid ultrasound image samples and carotid MRI image samples respectively. Then, the second-layer features of carotid ultrasound image samples, the second-layer features of carotid MRI image samples and the first-layer fusion features were fused to obtain the second-layer fusion features.

[0050] A multi-scale feature fusion network is used to extract the third-layer features of carotid ultrasound image samples and carotid MRI image samples respectively. Then, the third-layer features of carotid ultrasound image samples, the third-layer features of carotid MRI image samples and the second-layer fusion features are fused to obtain the third-layer fusion features, i.e., the fusion features.

[0051] As one or more implementations, the feature fusion refers to first connecting two feature layers, then passing them through a downsampling layer and a softmax layer to train a weight network based on common labels, thereby obtaining a new weighted feature layer.

[0052] As one or more implementations, the nuclear magnetic resonance attention classification network is trained by parallelizing the high-level features extracted from the nuclear magnetic resonance image and the fused features, learning weight values ​​according to the label guidance, and then obtaining the output result.

[0053] As one or more implementations, the ultrasound image attention classification network is trained by parallelizing the high-level features extracted from the ultrasound image and the fused features, learning weight values ​​according to the label guidance, and then obtaining the output result.

[0054] As one or more implementations, the result output module 4 outputs the grading results of ultrasound images and magnetic resonance images from level 0 to 4 according to the grading model, where level 0 is a stable plaque and level 4 is an extremely unstable plaque.

[0055] Example 2

[0056] This embodiment provides a carotid artery vulnerability grading system based on multimodal radiomics.

[0057] A carotid artery vulnerability grading system based on multimodal radiomics includes:

[0058] The data acquisition module is configured to acquire at least one of a carotid ultrasound image and an MRI image.

[0059] The output module is configured to: obtain the vulnerability level of carotid plaques based on at least one of carotid ultrasound images and MRI images using a carotid plaque grading model;

[0060] The model building module is configured as follows: the carotid plaque grading model includes a multi-scale feature fusion network and an attention classification network; wherein, the multi-scale feature fusion network is used to fuse the extracted features of carotid ultrasound image samples and carotid MRI image samples based on carotid ultrasound image samples and carotid MRI image samples to obtain fused features; the attention classification network is used to obtain the vulnerability level of carotid plaques based on the fused features and the carotid ultrasound image sample features / carotid MRI image sample features.

[0061] It should be noted that the data acquisition module, output module, and model building module described above are the same examples and application scenarios implemented in Embodiment 1, but are not limited to the content disclosed in Embodiment 1. It should also be noted that these modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0062] Example 3

[0063] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the carotid artery vulnerability grading method based on multimodal radiomics as described in Embodiment 1 above.

[0064] Example 4

[0065] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the carotid artery vulnerability grading method based on multimodal radiomics as described in Embodiment 1 above.

[0066] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0067] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0068] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0069] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0070] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0071] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A carotid artery vulnerability grading method based on multimodal radiomics, characterized in that, include: Obtain at least one of carotid artery ultrasound images and MRI images; Based on at least one of carotid ultrasound images and MRI images, a carotid plaque grading model is used to obtain the vulnerability level of carotid plaques; The carotid plaque grading model includes a multi-scale feature fusion network and an attention classification network. The multi-scale feature fusion network is used to fuse extracted features from carotid ultrasound and MRI images to obtain fused features. The attention classification network is used to determine the vulnerability level of the carotid plaque based on the fused features and the carotid ultrasound / MRI image features. The specific process for obtaining the fusion features includes: A multi-scale feature fusion network is used to extract the first-layer features of carotid ultrasound image samples and carotid MRI image samples respectively. Then, the first-layer features of the carotid ultrasound image samples and the first-layer features of the carotid MRI image samples are fused to obtain the first-layer fused features. A multi-scale feature fusion network was used to extract the second-layer features of carotid ultrasound image samples and carotid MRI image samples respectively. Then, the second-layer features of carotid ultrasound image samples, the second-layer features of carotid MRI image samples and the first-layer fusion features were fused to obtain the second-layer fusion features. A multi-scale feature fusion network is used to extract the third-layer features of carotid ultrasound image samples and carotid MRI image samples respectively. Then, the third-layer features of carotid ultrasound image samples, the third-layer features of carotid MRI image samples and the second-layer fusion features are fused to obtain the third-layer fusion features, i.e., the fusion features. The carotid artery ultrasound image samples were graded to obtain four vulnerability levels for carotid artery plaques based on ultrasound; the carotid artery MRI image samples were also graded to obtain four vulnerability levels for carotid artery plaques based on MRI. The carotid ultrasound image samples were labeled according to the vulnerability levels of four types of carotid plaques for ultrasound, resulting in vulnerability labels for the four types of carotid plaques for ultrasound. The carotid MRI image samples were labeled according to the vulnerability levels of four types of carotid plaques for MRI, resulting in vulnerability labels for the four types of carotid plaques for MRI. The attention classification network includes: an ultrasound image attention classification network and a magnetic resonance imaging attention network; The ultrasound image attention classification network is used to connect the high-level features of the carotid ultrasound image samples extracted by the multi-scale feature fusion network with the fused features in parallel, and then learn the weight values ​​according to the label guidance, and then train to obtain the vulnerability level of the carotid plaque.

2. The carotid artery vulnerability grading method based on multimodal radiomics according to claim 1, characterized in that, The magnetic resonance attention classification network is used to connect the high-level features of the carotid artery magnetic resonance image samples extracted by the multi-scale feature fusion network with the fused features in parallel, and then learn the weight values ​​according to the label guidance, and then train to obtain the vulnerability level of the carotid artery plaque.

3. A carotid artery vulnerability grading system based on multimodal radiomics, based on the carotid artery vulnerability grading method based on multimodal radiomics as described in any one of claims 1-2, characterized in that, include: The data acquisition module is configured to acquire at least one of a carotid ultrasound image and an MRI image. The output module is configured to: obtain the vulnerability level of carotid plaques based on at least one of carotid ultrasound images and MRI images using a carotid plaque grading model; The model building module is configured as follows: the carotid plaque grading model includes a multi-scale feature fusion network and an attention classification network; wherein, the multi-scale feature fusion network is used to fuse the extracted features of carotid ultrasound image samples and carotid MRI image samples based on carotid ultrasound image samples and carotid MRI image samples to obtain fused features; the attention classification network is used to obtain the vulnerability level of carotid plaques based on the fused features and the carotid ultrasound image sample features / carotid MRI image sample features.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the steps in the carotid artery vulnerability grading method based on multimodal radiomics as described in any one of claims 1-2.

5. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the carotid artery vulnerability grading method based on multimodal radiomics as described in any one of claims 1-2.

Images