A multimodal image fusion-based coronary artery lesion region identification system

Abstract

This invention relates to the field of image data processing technology, specifically to a multimodal image fusion-based coronary artery lesion region identification system. The system's steps include: determining abnormal regions in the vessel wall based on structural analysis of the vessel wall and lumen in optical coherence tomography (OCT) images of the user's coronary arteries; determining the fibrous cap region within the abnormal regions based on the pixel gradient and grayscale features of the fibrous cap, and designating the image region behind the fibrous cap region in the vessel wall as "behind the fibrous cap"; determining the lipid core abnormal region based on signal attenuation information behind the fibrous cap and the lipid distribution probability in near-infrared spectral data; and determining the target abnormal region in the OCT images based on the image features of the lipid core abnormal region in intravascular ultrasound images of the user's coronary arteries. This invention improves the accuracy of lesion identification and allows for more precise segmentation of lesion regions.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and more specifically to a coronary artery lesion region identification system based on multimodal image fusion. Background Technology

[0002] Coronary imaging techniques, such as coronary angiography (CAG), computed tomography coronary angiography (CTCA), and magnetic resonance imaging (MRI), have been widely used in the diagnosis of coronary artery disease. Each technique has its specific advantages and limitations; a single technique cannot obtain comprehensive information. Multimodal image fusion techniques are used to combine information from different image sources. This information can provide richer lesion features, such as tissue structure and lesion severity. Common image fusion methods include image registration, weighted averaging, and deep learning techniques.

[0003] To perform coronary artery lesion region identification through multimodal image fusion (the lesion identification mentioned here does not diagnose the disease, but rather assists medical staff in enhancing image recognition), multimodal images of the coronary arteries were acquired, and OCT (Optical Coherence Tomography) images were used to extract the volume and depth of the lesions. However, because the lipid core in the lesion can scatter the infrared light in the OCT images, lesions behind the fibrous cap in the OCT images will appear as blurred low signals, which can lead to confusion with image artifacts and inflammation, resulting in inaccurate lesion identification and inaccurate segmentation of the lesion region. Summary of the Invention

[0004] To address the technical problems of existing fusion imaging technologies failing to accurately identify coronary artery lesions and exhibiting inaccurate segmentation of lesion regions, the present invention aims to provide a coronary artery lesion region identification system based on multimodal image fusion. The specific technical solution adopted is as follows:

[0005] This invention provides a multimodal image fusion-based coronary artery lesion region identification system, the system comprising:

[0006] The feature analysis module is used to identify abnormal regions in the vessel wall based on the structural analysis of the vessel wall and lumen in the optical coherence tomography images of the user's coronary arteries; and to identify the fibrous cap region in the abnormal regions based on the pixel gradient features and grayscale features of the fibrous cap, and to record the image region in the vessel wall located behind the fibrous cap region as the fibrous cap rear region.

[0007] The image fusion module is used to determine the lipid core abnormal region behind the fibrous cap based on the signal attenuation information behind the fibrous cap and the lipid distribution probability in the near-infrared spectral data; and to determine the target abnormal region in the optical coherence tomography image based on the image characteristics of the lipid core abnormal region in the intravascular ultrasound image of the user's coronary artery.

[0008] Furthermore, the determination of abnormal regions in the vessel wall based on structural analysis of the vessel wall and lumen in optical coherence tomography images of the user's coronary arteries includes:

[0009] Acquire optical coherence tomography images of the user's coronary arteries and extract the vessel wall and lumen to determine the junction between the vessel wall and lumen.

[0010] Determine the straight line of pixels passing through the junction of the blood vessel lumen and the blood vessel wall, and take each pixel segment in the straight line of pixels on the blood vessel wall as the blood vessel wall pixel segment.

[0011] The structural integrity of the blood vessel wall pixel segment is determined by using the grayscale value of the pixel segment, and the abnormal areas in the blood vessel wall are identified by using the structural integrity.

[0012] Furthermore, the step of determining the structural integrity of blood vessel wall pixel segments using grayscale values ​​includes:

[0013] Obtain the grayscale value fitting curve of the blood vessel wall pixel segment, and determine the corresponding first derivative function curve;

[0014] Determine the average ratio between each maximum point and the mean of the ordinate in the first derivative curve, and determine the time abscissa difference between the maximum point and the starting point of the gray value fitting curve.

[0015] The structural integrity of the blood vessel wall pixel segment is determined by using the mean of the ratio and the difference of the time abscissa.

[0016] Furthermore, the method of utilizing structural integrity to determine the regions of abnormality in the blood vessel wall includes:

[0017] First blood vessel wall pixel segment with structural integrity less than a preset integrity threshold is identified; second blood vessel wall pixel segment belonging to the blood vessel wall junction is identified from the first blood vessel wall pixel segment.

[0018] The connected domains of the second blood vessel wall pixel segments on the blood vessel wall are considered as abnormal regions in the blood vessel wall.

[0019] Further, determining the fiber cap region within the abnormal region based on the pixel gradient features and grayscale features of the fiber cap includes:

[0020] Determine the straight line passing through the center of the blood vessel lumen and the abnormal area, and take each pixel segment in the straight line portion of the abnormal area as a depth segment.

[0021] By using the pixel gradient values ​​and pixel grayscale values ​​in the depth line segment, the fiber cap region in the abnormal area is determined.

[0022] Furthermore, the step of determining the fiber cap region within the abnormal region using pixel gradient values ​​and pixel grayscale values ​​in the depth line segment includes:

[0023] Determine the maximum value of the gradient of the pixel in the depth line segment, and use any maximum value as the first segmentation point and any maximum value after the first segmentation point as the second segmentation point.

[0024] Determine the average gradient value and average gray value of the pixels between the first segmentation point and the second segmentation point, as well as their gradient-gray value ratio.

[0025] The loss value of the depth line segment is determined by using the mean gradient of the pixel and the gradient-grayscale ratio. The first and second segmentation points in the depth line segment corresponding to the minimum loss value are then determined.

[0026] By using the first and second segmentation points in the depth segment corresponding to the minimum loss value, the fiber cap region in the anomaly area is determined.

[0027] Further, the step of determining the abnormal lipid core region behind the fiber cap based on signal attenuation information behind the fiber cap and combined with the lipid distribution probability in near-infrared spectral data includes:

[0028] Align the near-infrared spectral data map with the optical coherence tomography image to obtain the pixels behind the fiber cap in the aligned near-infrared spectral data map;

[0029] The signal attenuation rate behind the fiber cap is determined, and the lipid attenuation rate behind the fiber cap is obtained by combining the lipid distribution probability mean of the pixels behind the fiber cap.

[0030] The area behind the fibrous cap where the lipid decay rate is greater than a preset decay threshold is defined as the abnormal lipid core region behind the fibrous cap.

[0031] Further, determining the signal attenuation rate behind the fiber cap includes:

[0032] Determine the grayscale value, gradient value, and pixel distance between the pixel behind the fibrous cap and the center point of the blood vessel lumen;

[0033] Arrange the gray values ​​and gradient values ​​according to the pixel distance from smallest to largest to obtain the gray value fitting curve and the gradient value fitting curve.

[0034] The signal attenuation rate behind the fiber cap is determined by using the absolute values ​​of the average slopes of the grayscale fitting curve and the gradient fitting curve.

[0035] Further, the step of determining the target abnormal region in the optical coherence tomography image based on the image characteristics of the lipid core abnormal region in the intravascular ultrasound image of the user's coronary artery includes:

[0036] Align the intravascular ultrasound image of the user's coronary artery with the optical coherence tomography image, identify the lipid core abnormal region in the aligned intravascular ultrasound image and record it as the ultrasound lipid core abnormal region.

[0037] Determine the straight line of pixels passing through the center of the vascular lumen and the abnormal region of the ultrasound lipid core, and determine the straight line segments of each pixel in the straight line portion of the abnormal region of the ultrasound lipid core.

[0038] The gradient and grayscale values ​​of the straight line segments of pixels are used to determine the target anomalous region in optical coherence tomography (OCT) images.

[0039] Furthermore, the step of determining the target anomalous region in the optical coherence tomography image using the gradient value and grayscale value in the straight line segment of the pixel includes:

[0040] Determine the maximum gradient value point in the straight line segment of the pixel and the difference between the mean gray values ​​on both sides of the maximum gradient value point;

[0041] The segmentation coefficient of the gradient maximum point is determined by the difference between the maximum gradient point and the mean gray value on both sides of it.

[0042] The maximum gradient value with the largest segmentation coefficient is determined and used as the target maximum value. The target maximum value of the straight line segment of each pixel is fitted and projected onto the optical coherence tomography image to obtain the target abnormal region.

[0043] The present invention has the following beneficial effects:

[0044] This invention aims to identify coronary artery lesion regions through multimodal image fusion. It acquires multimodal images of the coronary arteries and uses OCT images to extract the volume and depth of lesions. This method obtains OCT images, NIRS (Near-Infrared Spectroscopy) data, and IVUS (Intravenous Ultrasound) images of the patient's coronary arteries. The presence of lesions (abnormal areas) is identified by the changes in the three-layer structure of the vessel wall in the OCT images. Furthermore, the fibrous cap portion is identified based on its image representation. The lesion behind the fibrous cap is determined by the weakening behind it combined with lipid distribution probability in the NIRS data. Finally, the fibrous cap and lesion regions in the OCT images are determined by changes in lipid reflectance in the IVUS images. Compared to traditional methods using OCT images for lesion analysis, this method also combines NIRS data and IVUS images for lipid feature analysis, improving the accuracy of lesion identification and enabling more precise identification and segmentation of lesion regions. Attached Figure Description

[0045] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 The flowchart below shows the steps of a multimodal image fusion coronary artery lesion region identification system provided in one embodiment of the present invention.

[0047] Figure 2 A detailed flowchart of step S1 in a multimodal image fusion coronary lesion region identification system provided in an embodiment of the present invention;

[0048] Figure 3 A detailed flowchart of step S13 in a multimodal image fusion coronary lesion region identification system provided in an embodiment of the present invention;

[0049] Figure 4 A detailed flowchart of step S2 in a multimodal image fusion coronary lesion region identification system provided in an embodiment of the present invention;

[0050] Figure 5 A detailed flowchart of step S3 in a multimodal image fusion coronary lesion region identification system provided in an embodiment of the present invention;

[0051] Figure 6 A detailed flowchart of step S4 in a multimodal image fusion coronary lesion region identification system provided in an embodiment of the present invention;

[0052] Figure 7 This is a schematic diagram of the hardware operating environment of the coronary artery lesion region identification device for multimodal image fusion involved in the embodiments of the present invention;

[0053] Figure 8 This is a schematic diagram of the framework structure of the coronary artery lesion region identification system based on multimodal image fusion involved in the embodiments of the present invention. Detailed Implementation

[0054] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a multimodal image fusion coronary artery lesion region identification system proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0056] The specific scheme of the coronary artery lesion region identification system based on multimodal image fusion provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0057] Example 1:

[0058] For a multimodal image fusion coronary artery lesion region identification system provided by this invention, please refer to [link to relevant documentation]. Figure 1 and Figure 8 , Figure 1 A flowchart illustrating the steps of a coronary artery lesion region identification system based on multimodal image fusion provided in an embodiment of the present invention is shown. Figure 8 This is a schematic diagram of the framework structure of the coronary artery lesion region identification system based on multimodal image fusion involved in the embodiments of the present invention.

[0059] The multimodal image fusion coronary lesion region identification system (hereinafter referred to as the "coronary lesion region identification system" or the "system") includes:

[0060] The feature analysis module A10 is used to determine the abnormal regions in the vessel wall based on the structural analysis of the vessel wall and vessel lumen in the optical coherence tomography images of the user's coronary arteries; and to determine the fibrous cap region in the abnormal regions based on the pixel gradient features and grayscale features of the fibrous cap, and to record the image region in the vessel wall located behind the fibrous cap region as the fibrous cap rear region.

[0061] The image fusion module A20 is used to determine the lipid core abnormal region behind the fibrous cap based on the signal attenuation information behind the fibrous cap and the lipid distribution probability in the near-infrared spectral data; and to determine the target abnormal region in the optical coherence tomography image based on the image characteristics of the lipid core abnormal region in the intravascular ultrasound image of the user's coronary artery.

[0062] The various method steps corresponding to the multimodal image fusion coronary artery lesion region identification system include:

[0063] Step S1: Based on the structural analysis of the vessel wall and vessel lumen in the optical coherence tomography images of the user's coronary arteries, determine the abnormal areas in the vessel wall.

[0064] In interventional procedures for patients with acute coronary syndrome (ACS), especially those with lesions located at the bifurcation of the left anterior descending artery or in a tortuous segment of the vessel, an integrated OCT-NIRS-IVUS trimodal fusion catheter (outer diameter ≤0.68mm) was inserted via the radial artery approach and automatically withdrawn at a uniform speed of 20mm / s within the target vessel, while simultaneously acquiring the following data:

[0065] OCT structural data (near-infrared wavelength 1300nm, resolution 10μm) covers the 360° circumference of blood vessels, which is used to capture the blurred boundaries of fibrous cap ruptures, superficial lipid cores and signal attenuation gradients.

[0066] NIRS chemical composition data (wavelength 800-2500 nm) were used to generate lipid core loading index (LCBI) maps, which quantified the lipid probability distribution using a red-yellow spectrum (yellow indicates high-probability lipid core regions).

[0067] IVUS depth data (40MHz ultrasound) penetrates deep into the lipid core to obtain plaque load and vascular adventitia boundary, compensating for the penetration depth deficiency of OCT;

[0068] Throughout the process, the scanning density is increased in low shear stress regions (such as the side opposite the bifurcation ridge), and the spatial alignment of the three-modal images is ensured by spatiotemporal registration technology (error <0.1mm). Finally, a three-dimensional fused image (including lipid core angle, depth range and volume quantification) is output for subsequent plaque risk assessment.

[0069] Specifically, please refer to Figure 2 Step S1 includes:

[0070] Step S11: Acquire optical coherence tomography images of the user's coronary arteries and extract the vessel wall and lumen from them to determine the vessel wall junction between the vessel wall and the lumen.

[0071] Step S12: Determine the straight line of pixels passing through the junction of the center of the blood vessel lumen and the blood vessel wall, and take each pixel segment in the straight line of pixels on the blood vessel wall as the pixel segment of the blood vessel wall.

[0072] Step S13: Determine the structural integrity of the blood vessel wall pixel segment using the grayscale value of the pixel segment, and determine the abnormal area in the blood vessel wall using the structural integrity.

[0073] More specifically, in one embodiment, please refer to Figure 3 Step S13, which uses the grayscale values ​​of the blood vessel wall pixel segments to determine the structural integrity of the blood vessel wall pixel segments, includes:

[0074] Step S131: Obtain the gray value fitting curve of the blood vessel wall pixel segment and determine the corresponding first derivative function curve;

[0075] Step S132: Determine the average ratio between each maximum point in the first derivative curve and the mean of the vertical axis, and determine the time abscissa difference between the maximum point in the gray value fitting curve and the starting point of the curve.

[0076] Step S133: Use the average ratio and the time abscissa difference to determine the structural integrity of the blood vessel wall pixel segment.

[0077] In this embodiment, lipid core lesions (lesions also referred to as "abnormalities") may exist in the coronary arteries, which are a very critical and dangerous component inside atherosclerotic plaques;

[0078] Near-infrared light from OCT is easily absorbed and scattered when penetrating the lipid core, resulting in signal attenuation and inability to identify deep boundaries. This makes it impossible for the system to accurately measure the overall volume and depth of the lipid core. Furthermore, the "gold standard" labels (such as fibrous cap thickness or lipid angle) that OCT needs to extract in multimodal image fusion have uncertainties in the depth direction. Therefore, analysis based on NIRS and IVUS data is required.

[0079] In OCT images, normal blood vessel walls present a three-layer structure (intima, media, and adventitia). However, when atherosclerotic plaques are present, space-occupying lesions appear in the blood vessel walls, which can lead to the loss of the three-layer structure of the blood vessel walls.

[0080] Therefore, in order to locate lipid core lesions in OCT images, it is necessary to perform vascular wall structure analysis;

[0081] Acquire patient coronary artery OCT images and use neural networks to extract the lumen and wall portions (existing technology).

[0082] Canny edge detection was used to obtain the edges and pixel gradients of the blood vessel wall in the OCT image.

[0083] It also obtains the center point of the blood vessel lumen in the OCT image, and obtains the pixel points between the blood vessel wall and the blood vessel lumen, which are recorded as the blood vessel wall junction (the blood vessel wall junction is a closed pixel line).

[0084] Furthermore, obtain several straight lines passing through the center point of the blood vessel lumen and the junction of the blood vessel wall, i.e., pixel point lines. For each pixel point line, obtain several pixel point segments on the line passing through the blood vessel wall, denoted as blood vessel wall pixel point segments.

[0085] Obtain the grayscale value fitting curve q1 of the blood vessel wall pixel segment (the horizontal axis is time and the vertical axis is grayscale value), and obtain the first derivative function curve q1' in the grayscale value fitting curve;

[0086] Since the blood vessel wall has a three-layer structure, and the outward signal of OCT images will gradually weaken due to the blood vessel wall tissue, the mean value z of the vertical coordinate of the curve q1' of the blood vessel wall pixel segment is obtained, as well as the maximum value points in the curve q1'; and the mean value b of the ratio between each maximum value point and the mean value z is obtained.

[0087] Obtain the maximum (grayscale) point in curve q1 of the blood vessel wall pixel segment, and obtain the time abscissa difference c between the maximum point and the starting point of curve q1;

[0088] Therefore, the structural integrity of the blood vessel wall pixel segments can be obtained. Specifically through The reciprocal of the normalized value is used to calculate the value, where i represents the i-th pixel segment of the blood vessel wall, and represents any pixel segment of the blood vessel wall. The smaller the value, the smaller the difference between the maximum value and the mean value z of the pixel segment of the blood vessel wall, and the closer the maximum gray value is to the inside of the blood vessel wall, thus indicating that the structure of the blood vessel wall is more complete.

[0089] More specifically, in another embodiment, step S13, which utilizes structural integrity to determine the region of abnormality in the blood vessel wall, includes:

[0090] First blood vessel wall pixel segment with structural integrity less than a preset integrity threshold is identified; second blood vessel wall pixel segment belonging to the blood vessel wall junction is identified from the first blood vessel wall pixel segment.

[0091] The connected domains of the second blood vessel wall pixel segments on the blood vessel wall are considered as abnormal regions in the blood vessel wall.

[0092] Based on the above embodiments, for the structural integrity WZ of each obtained blood vessel wall pixel segment, the blood vessel wall pixel segments are clustered into two types of pixel segment clusters, and the portion with smaller structural integrity WZ is obtained. Smaller structural integrity WZ refers to the first blood vessel wall pixel segment whose structural integrity is less than a preset integrity threshold. The preset integrity threshold can be a fixed threshold, such as 0.5, or it can be a percentile, such as the 50th percentile, that is, the range of the last 50% of the structural integrity from largest to smallest.

[0093] Since the lesion causes structural changes in the blood vessel wall, a segment of the second blood vessel wall pixel point segment is obtained from the cluster of parts with less structural integrity in the junction of the blood vessel wall; and the connected domain of this segment of the blood vessel wall is obtained and recorded as the lesion area, that is, the abnormal area in the blood vessel wall.

[0094] Step S2: Determine the fibrous cap region in the abnormal area based on the pixel gradient features and grayscale features of the fibrous cap, and record the image area in the blood vessel wall located behind the fibrous cap region as the area behind the fibrous cap.

[0095] The above embodiments obtain the lesion location in the blood vessel wall based on the structural impact of the lesion on the blood vessel wall. In order to further accurately identify the lesion area, it is necessary to obtain the fibrous cap in the lipid core.

[0096] In OCT images, the fibrous cap of the lesion appears highly reflective, while the lipid core beneath the fibrous cap exhibits rapid signal attenuation due to signal absorption. Therefore, in OCT images, the fibrous cap appears as a high-grayscale band-like structure on the vessel wall.

[0097] Specifically, please refer to Figure 4 Step S2, determining the fiber cap region in the abnormal area based on the pixel gradient features and grayscale features of the fiber cap, includes:

[0098] Step S21: Determine the straight line of pixels passing through the center of the blood vessel lumen and the abnormal area, and take each pixel segment in the straight line of pixels in the abnormal area as a depth segment.

[0099] Step S22: Using the pixel gradient values ​​and pixel grayscale values ​​in the depth line segment, determine the fiber cap region in the abnormal area.

[0100] More specifically, step S22 includes:

[0101] Determine the maximum value of the gradient of the pixel in the depth line segment, and use any maximum value as the first segmentation point and any maximum value after the first segmentation point as the second segmentation point.

[0102] Determine the average gradient value and average gray value of the pixels between the first segmentation point and the second segmentation point, as well as their gradient-gray value ratio.

[0103] The loss value of the depth line segment is determined by using the mean gradient of the pixel and the gradient-grayscale ratio. The first and second segmentation points in the depth line segment corresponding to the minimum loss value are then determined.

[0104] By using the first and second segmentation points in the depth segment corresponding to the minimum loss value, the fiber cap region in the anomaly area is determined.

[0105] In this embodiment, several straight lines of pixels passing through the center of the blood vessel lumen and the lesion area are obtained, and several line segments of pixels passing through the straight lines and located in the lesion area are recorded as depth line segments of the lesion area in the blood vessel wall (the depth line segments are arranged from the inside to the outside of the blood vessel wall).

[0106] Obtain the gradient value and grayscale value of pixels in the depth line segment, and obtain the maximum value of the gradient value of pixels in the depth line segment;

[0107] In OCT images, the fibrous cap appears as a high-grayscale band-like structure. Therefore, one side edge of the fibrous cap will have a higher gradient value and will transition from the lower grayscale vascular tissue to the higher grayscale fibrous cap pixels.

[0108] For line segments of arbitrary depth, the point with the maximum value of any pixel gradient value is recorded as the first segmentation point;

[0109] Obtain the gradient value of the pixel at the first segmentation point, and also obtain the average value of the pixels before the first segmentation point, denoted as h1;

[0110] The second dividing point is defined as any maximum point in the depth segment that is located after the first dividing point.

[0111] Obtain the average gradient value t of the pixels between the first and second segmentation points, and obtain the average gray value h2 of the pixels located between the first and second segmentation points;

[0112] Further, the gradient-grayscale ratio h' between the average gradient t of the pixels between the first and second segmentation points and the average grayscale h2 of the pixels on both sides is obtained;

[0113] Therefore, all first and second segmentation points in all depth segments are obtained, and the loss value S of the depth segments is obtained. Specifically, the loss value S is obtained through... The inverse of the normalized value is obtained, where j represents the j-th depth segment, refers to any depth segment, and J is the number of depth segments. This represents the gradient-grayscale ratio between the first and second dividing points of the j-th depth line segment. denoted as the mean gradient of pixels between the first and second dividing points of the j-th depth segment, and norm represents normalization.

[0114] And for The larger the value, the greater the gray level of the pixels at the first and second segmentation points compared to the surrounding gray levels, and the larger the gradient value. Therefore, the first and second pixels are more likely to be pixels that segment the fiber cap, and thus the smaller the loss value.

[0115] For the above operations, obtain the loss value, obtain the first and second segmentation points of each depth segment corresponding to the minimum loss value; and fit these first and second segmentation points respectively, and record the region between the obtained fitted curves as the fiber cap region.

[0116] Step S3: Determine the abnormal lipid core region behind the fiber cap based on the signal attenuation information behind the fiber cap and the lipid distribution probability in the near-infrared spectral data.

[0117] Specifically, please refer to Figure 5 Step S3 includes:

[0118] Step S31: Align the near-infrared spectral data map with the optical coherence tomography image to obtain the pixel points behind the fiber cap in the aligned near-infrared spectral data map.

[0119] Step S32: Determine the signal attenuation rate behind the fiber cap, and combine it with the lipid distribution probability mean of the pixels behind the fiber cap to obtain the lipid attenuation rate behind the fiber cap.

[0120] More specifically, step S32, determining the signal attenuation rate behind the fiber cap, includes:

[0121] Determine the grayscale value, gradient value, and pixel distance between the pixel behind the fibrous cap and the center point of the blood vessel lumen;

[0122] Arrange the gray values ​​and gradient values ​​according to the pixel distance from smallest to largest to obtain the gray value fitting curve and the gradient value fitting curve.

[0123] The signal attenuation rate behind the fiber cap is determined by using the absolute values ​​of the average slopes of the grayscale fitting curve and the gradient fitting curve.

[0124] Step S33: The area behind the fiber cap where the lipid decay rate is greater than a preset decay threshold is defined as the lipid core abnormal region behind the fiber cap.

[0125] In this embodiment, the fibrous cap of the lipid core lesion in the OCT image obtained by the above operation is the part that separates the lesion from the inside of the blood vessel, and the thickness of the fibrous cap is related to the risk of the lesion. Therefore, the lesion area can be further identified by the obtained fibrous cap.

[0126] The lipid core lesion area behind the fibrous cap will appear blurred and low signal due to lipid absorption. Therefore, lesion analysis can be performed based on the blurred low signal behind the fibrous cap.

[0127] However, imaging artifacts and macrophage infiltration in OCT images cause the blurred low-signal area behind the fibrous cap to be a lipid core. Therefore, lesion verification based on chemical composition in NIRS data is necessary to ensure the "gold standard" of OCT images.

[0128] Acquire the image portion of the blood vessel wall located behind the fibrous cap region, denoted as "behind the fibrous cap";

[0129] Obtain the grayscale value and gradient value of the pixel behind the fibrous cap, as well as the distance between the pixel behind the fibrous cap and the center point of the blood vessel lumen;

[0130] The pixels behind the fiber cap are arranged in ascending order of their pixel distance from the center point of the blood vessel lumen, and the gray value fitting curve q3 and gradient value fitting curve q4 are obtained according to the arrangement.

[0131] Obtain the absolute values ​​of the average slopes k3 and k4 of the fitted curves q3 and q4, respectively.

[0132] Therefore, the signal attenuation rate s behind the fibrous cap is obtained by normalization with k3×k4. For k3 and k4, the larger the value, the faster the gray value of the pixel behind the fibrous cap decreases towards the outside of the blood vessel, thus indicating a higher signal attenuation rate behind the fibrous cap.

[0133] To acquire NIRS data, the NIRS system emits multi-wavelength light in the range of 800-2500 nm, analyzes the characteristic absorption peaks of lipids in the blood vessel wall reflectance spectrum, and obtains the lipid presence probability map for each pixel.

[0134] Align the NIRS map with the OCT image and obtain the pixels behind the fiber cap in the aligned NIRS map;

[0135] Obtain the mean lipid distribution probability z of the pixels behind the fiber cap;

[0136] Therefore, the lipid attenuation rate ZS in the region behind the fibrous cap can be obtained, specifically by normalizing s×z. For s×z, the larger the value, the more obvious the signal attenuation of the tissue behind the fibrous cap in the OCT image and the greater the probability that the pixel behind the fibrous cap in the NIRS atlas is lipid. Therefore, the lipid core attenuation rate in the region behind the fibrous cap is greater.

[0137] The obtained lipid weakening rate is compared with a preset weakening threshold of 0.5 (which can be adjusted). When ZS > 0.5, it means that there is a lipid core lesion behind the fibrous cap, that is, an abnormal lipid core area, which needs to be further identified in the future.

[0138] Step S4: Based on the image characteristics of the lipid core abnormal region in the intravascular ultrasound image of the user's coronary artery, determine the target abnormal region in the optical coherence tomography image.

[0139] Specifically, please refer to Figure 6 Step S4 includes:

[0140] Step S41: Align the intravascular ultrasound image of the user's coronary artery with the optical coherence tomography image, identify the lipid core abnormal region in the aligned intravascular ultrasound image and record it as the ultrasound lipid core abnormal region.

[0141] Step S42: Determine the straight line of pixels passing through the center of the vascular lumen and the abnormal area of ​​the ultrasound lipid core, and determine the straight line segments of each pixel in the straight line part of the abnormal area of ​​the ultrasound lipid core.

[0142] Step S43: Use the gradient value and gray value in the straight line segment of the pixel to determine the target abnormal area in the optical coherence tomography image.

[0143] More specifically, step S43 includes:

[0144] Determine the maximum gradient value point in the straight line segment of the pixel and the difference between the mean gray values ​​on both sides of the maximum gradient value point;

[0145] The segmentation coefficient of the gradient maximum point is determined by the difference between the maximum gradient point and the mean gray value on both sides of it.

[0146] The maximum gradient value with the largest segmentation coefficient is determined and used as the target maximum value. The target maximum value of the straight line segment of each pixel is fitted and projected onto the optical coherence tomography image to obtain the target abnormal region.

[0147] In this embodiment, after the above operations determine that the lipid core (abnormal lipid core region) is behind the fibrous cap, it is necessary to further identify the lesion region; however, since the lipid core absorbs more signals in OCT images, the signal intensity behind the fibrous cap is low, and the edge of the lesion cannot be obtained; therefore, it is necessary to combine IVUS images for analysis.

[0148] IVUS images of the patient's coronary arteries are acquired. IVUS utilizes the difference in reflection of 30-40MHz high-frequency ultrasound in vascular tissue and has a penetration depth of 5-8mm, overcoming the signal attenuation caused by lipid absorption in OCT.

[0149] The acquired IVUS images of the patient's coronary arteries were aligned with the OCT images; and the region behind the fibrous cap in the IVUS images, i.e. the lipid core abnormal region (the region behind the fibrous cap has been identified as the lipid core abnormal region in the above embodiment), was recorded as the ultrasound lipid core abnormal region for easy distinction.

[0150] Neural networks are used to extract the adventitia (i.e., the outer boundary of the blood vessel) behind the fibrous cap in IVUS images.

[0151] In IVUS images, the acoustic impedance of the lipid core is significantly lower than that of fibrous tissue or calcified plaques. Therefore, the acoustic reflection of the lipid region in the image is weak, resulting in a lower gray value.

[0152] Therefore, several pixel line segments are obtained in the posterior part of the fibrous cap in the IVUS image, where each pixel line segment passes through the center of the blood vessel lumen. In other words, the pixel line segments passing through the center of the blood vessel lumen and the abnormal area of ​​the ultrasound lipid core are determined, and each pixel line segment in the pixel line segment of the abnormal area of ​​the ultrasound lipid core is determined.

[0153] For the above pixel line segments, obtain the gradient value and gray value, and obtain the maximum gradient value point in the pixel line segments;

[0154] For any maximum gradient point, obtain its gradient value a, and obtain the difference x between the mean gray values ​​on both sides of the maximum gradient point.

[0155] Behind the fibrous cap, due to the presence of the outer edge of the blood vessel, a high gray band caused by the blood vessel wall will appear behind the lipid core behind the fibrous cap.

[0156] Therefore, for any maximum gradient value of a pixel line segment, its segmentation coefficient FG can be obtained, specifically by normalizing a×x. For a×x, the larger the value, the larger the gradient value of the maximum gradient value point, and the larger the gray value on both sides of the maximum gradient value point. Therefore, it is the larger the segmentation coefficient for segmenting the low echo of lipid core and the high echo of blood vessel.

[0157] The maximum gradient value of the segmentation point with the largest coefficient in the straight line segment of the pixel is obtained as the target maximum point. The target maximum point selected in the straight line segment of the pixel behind the fibrous cap is fitted to obtain the edge region of the lipid core lesion region in the IVUS image (i.e., locate the target abnormal region in the IVUS image). It is then projected onto the OCT image to obtain the target abnormal region in the OCT image.

[0158] This invention aims to identify coronary artery lesion regions through multimodal image fusion. It acquires multimodal images of the coronary arteries and uses OCT images to extract the volume and depth of lesions. This method obtains OCT images, NIRS (Near-Infrared Spectroscopy) data, and IVUS (Intravenous Ultrasound) images of the patient's coronary arteries. The presence of lesions (abnormal areas) is identified by the changes in the three-layer structure of the vessel wall in the OCT images. Furthermore, the fibrous cap portion is identified based on its image representation. The lesion behind the fibrous cap is determined by the attenuation behind the cap combined with the lipid distribution probability in the NIRS data. Finally, the fibrous cap and lesion regions in the OCT images are determined by changes in lipid reflectance in the IVUS images. Compared to traditional methods using OCT images for lesion analysis, this method also combines NIRS data and IVUS images for lipid feature analysis, improving the accuracy of lesion identification and enabling more precise segmentation of lesion regions.

[0159] Example 2:

[0160] This invention also proposes a multimodal image fusion-based coronary artery lesion region identification device. The device can be a data processing device such as a computer or server, or a combination of multiple devices.

[0161] like Figure 7 As shown, Figure 7 This is a schematic diagram of the hardware operating environment of the coronary artery lesion region identification device involving multimodal image fusion according to an embodiment of the present invention.

[0162] like Figure 7As shown, the multimodal image fusion coronary artery lesion region identification device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display or an input unit such as a control panel; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM or a stable, non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001. The memory 1005, as a computer storage medium, may include a coronary artery lesion region identification program.

[0163] Those skilled in the art will understand that Figure 7 The hardware structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0164] Continue to refer to Figure 7 , Figure 7 The memory 1005, which is a computer-readable storage medium, may include an operating device, a user interface module, a network communication module, and a coronary artery lesion area identification program.

[0165] exist Figure 7 In this embodiment, the network communication module is mainly used to connect to the server and can communicate with the server for data; while the processor 1001 can call the coronary artery lesion area identification program stored in the memory 1005 and execute the steps in the above embodiments.

[0166] The hardware structure of the coronary artery lesion region identification device based on the above-mentioned multimodal image fusion is used to implement various embodiments of the coronary artery lesion region identification system based on multimodal image fusion of the present invention.

[0167] Furthermore, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a coronary artery lesion region identification program, wherein, when executed by a processor, the coronary artery lesion region identification program implements the steps of the multimodal image fusion coronary artery lesion region identification system described above.

[0168] The method implemented when the coronary artery lesion region identification procedure is executed can be referred to in various embodiments of the coronary artery lesion region identification system of the multimodal image fusion of the present invention, and will not be repeated here.

[0169] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0170] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0171] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] The above description is only a preferred embodiment of the present invention and does not limit the scope of protection of the present invention. All equivalent structural / method transformations made under the inventive concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the scope of protection of the present invention.

Claims

1. A coronary artery lesion region identification system based on multimodal image fusion, characterized in that, The system includes: The feature analysis module is used to identify abnormal regions in the vessel wall based on the structural analysis of the vessel wall and lumen in the optical coherence tomography images of the user's coronary arteries; and to identify the fibrous cap region in the abnormal regions based on the pixel gradient features and grayscale features of the fibrous cap, and to record the image region in the vessel wall located behind the fibrous cap region as the fibrous cap rear region. The image fusion module is used to determine the lipid core abnormal region behind the fibrous cap based on the signal attenuation information behind the fibrous cap and the lipid distribution probability in the near-infrared spectral data; and to determine the target abnormal region in the optical coherence tomography image based on the image characteristics of the lipid core abnormal region in the intravascular ultrasound image of the user's coronary artery.

2. The coronary artery lesion region identification system based on multimodal image fusion according to claim 1, characterized in that, The structural analysis of the vessel wall and lumen in the optical coherence tomography (OCT) images of the user's coronary arteries identifies abnormal regions in the vessel wall, including: Acquire optical coherence tomography images of the user's coronary arteries and extract the vessel wall and lumen to determine the junction between the vessel wall and lumen. Determine the straight line of pixels passing through the junction of the blood vessel lumen and the blood vessel wall, and take each pixel segment in the straight line of pixels on the blood vessel wall as the blood vessel wall pixel segment. The structural integrity of the blood vessel wall pixel segment is determined by using the grayscale value of the pixel segment, and the abnormal areas in the blood vessel wall are identified by using the structural integrity.

3. The coronary artery lesion region identification system based on multimodal image fusion according to claim 2, characterized in that, The method of determining the structural integrity of blood vessel wall pixel segments using grayscale values ​​includes: Obtain the grayscale value fitting curve of the blood vessel wall pixel segment, and determine the corresponding first derivative function curve; Determine the average ratio between each maximum point and the mean of the ordinate in the first derivative curve, and determine the time abscissa difference between the maximum point and the starting point of the gray value fitting curve. The structural integrity of the blood vessel wall pixel segment is determined by using the mean of the ratio and the difference of the time abscissa.

4. The coronary artery lesion region identification system based on multimodal image fusion according to claim 2, characterized in that, The method of using structural integrity to determine the abnormal regions in the blood vessel wall includes: First blood vessel wall pixel segment with structural integrity less than a preset integrity threshold is identified; second blood vessel wall pixel segment belonging to the blood vessel wall junction is identified from the first blood vessel wall pixel segment. The connected domains of the second blood vessel wall pixel segments on the blood vessel wall are considered as abnormal regions in the blood vessel wall.

5. The coronary artery lesion region identification system based on multimodal image fusion according to claim 1, characterized in that, The step of determining the fiber cap region in the abnormal area based on the pixel gradient features and grayscale features of the fiber cap includes: Determine the straight line passing through the center of the blood vessel lumen and the abnormal area, and take each pixel segment in the straight line portion of the abnormal area as a depth segment. By using the pixel gradient values ​​and pixel grayscale values ​​in the depth line segment, the fiber cap region in the abnormal area is determined.

6. The coronary artery lesion region identification system based on multimodal image fusion according to claim 5, characterized in that, The method of determining the fiber cap region within the abnormal area by utilizing the pixel gradient values ​​and pixel grayscale values ​​in the depth line segment includes: Determine the maximum value of the gradient of the pixel in the depth line segment, and use any maximum value as the first segmentation point and any maximum value after the first segmentation point as the second segmentation point. Determine the average gradient value and average gray value of the pixels between the first segmentation point and the second segmentation point, as well as their gradient-gray value ratio. The loss value of the depth line segment is determined by using the mean gradient of the pixel and the gradient-grayscale ratio. The first and second segmentation points in the depth line segment corresponding to the minimum loss value are then determined. By using the first and second segmentation points in the depth segment corresponding to the minimum loss value, the fiber cap region in the anomaly area is determined.

7. The coronary artery lesion region identification system based on multimodal image fusion according to claim 1, characterized in that, The step of determining the abnormal lipid core region behind the fiber cap based on signal attenuation information behind the fiber cap and combined with the lipid distribution probability in near-infrared spectral data includes: Align the near-infrared spectral data map with the optical coherence tomography image to obtain the pixels behind the fiber cap in the aligned near-infrared spectral data map; The signal attenuation rate behind the fiber cap is determined, and the lipid attenuation rate behind the fiber cap is obtained by combining the lipid distribution probability mean of the pixels behind the fiber cap. The area behind the fibrous cap where the lipid decay rate is greater than a preset decay threshold is defined as the abnormal lipid core region behind the fibrous cap.

8. The coronary artery lesion region identification system based on multimodal image fusion according to claim 7, characterized in that, Determining the signal attenuation rate behind the fiber cap includes: Determine the grayscale value, gradient value, and pixel distance between the pixel behind the fibrous cap and the center point of the blood vessel lumen; Arrange the gray values ​​and gradient values ​​according to the pixel distance from smallest to largest to obtain the gray value fitting curve and the gradient value fitting curve. The signal attenuation rate behind the fiber cap is determined by using the absolute values ​​of the average slopes of the grayscale fitting curve and the gradient fitting curve.

9. The coronary artery lesion region identification system based on multimodal image fusion according to claim 1, characterized in that, The process of determining the target abnormal region in optical coherence tomography (OCT) images based on the image characteristics of the lipid core abnormal region in the user's intravascular ultrasound images of the coronary arteries includes: Align the intravascular ultrasound image of the user's coronary artery with the optical coherence tomography image, identify the lipid core abnormal region in the aligned intravascular ultrasound image and record it as the ultrasound lipid core abnormal region. Determine the straight line of pixels passing through the center of the vascular lumen and the abnormal region of the ultrasound lipid core, and determine the straight line segments of each pixel in the straight line portion of the abnormal region of the ultrasound lipid core. The gradient and grayscale values ​​of the straight line segments of pixels are used to determine the target anomalous region in optical coherence tomography (OCT) images.

10. The coronary artery lesion region identification system based on multimodal image fusion according to claim 9, characterized in that, The method of determining the target anomalous region in an optical coherence tomography (OCT) image by utilizing the gradient and grayscale values ​​of pixel line segments includes: Determine the maximum gradient value point in the straight line segment of the pixel and the difference between the mean gray values ​​on both sides of the maximum gradient value point; The segmentation coefficient of the gradient maximum point is determined by the difference between the maximum gradient point and the mean gray value on both sides of it. The maximum gradient value with the largest segmentation coefficient is determined and used as the target maximum value. The target maximum value of the straight line segment of each pixel is fitted and projected onto the optical coherence tomography image to obtain the target abnormal region.

Images