A coronary artery inner and outer membrane boundary segmentation method based on adjacent scale complementarity
Through the multi-scale processing module and adjacent scale complementary module in the deep learning network, the inner and outer membrane boundaries of the coronary artery in the IVUS image are automatically segmented, solving the problem of time-consuming and inefficient manual segmentation and achieving fast and accurate segmentation results.
Patent Information
- Application Number
- CN202210208869.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-03
- Publication Date
- 2025-05-06
- Estimated Expiration
- 2042-03-03
AI Technical Summary
In clinical practice, it is time-consuming and inefficient for doctors to manually segment the coronary endarterial and epimembrane in IVUS images.
Using a deep learning-based approach, the inner and outer membrane boundaries of the coronary artery are automatically segmented through multi-scale densely linked hollow convolution modules, complementary information learning modules at adjacent scales and contextual supplementary modules at adjacent scales.
It realizes rapid and accurate segmentation of the inner and outer membrane boundaries of the coronary artery, improves segmentation efficiency, and improves the accuracy of segmentation results.
Smart Images
Figure CN114612405B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of image processing and detection, and in particular to a method for segmenting inner and outer membrane boundaries of coronary arteries based on adjacent scale complementarity. Background Art
[0002] Identification of the intima and adventitia of coronary arteries is the first step in quantifying coronary artery section data in clinical practice. Quantified vascular indicators are important reference indicators for guiding clinical diagnosis. Therefore, accurate identification of the intima and adventitia of coronary arteries is very important for the diagnosis of patients with coronary artery disease. Clinically, intravascular ultrasound imaging technology (IVUS) can clearly show the cross-section and internal structure of the coronary arteries, so IVUS images are widely used in clinical practice to detect the intima and adventitia of the coronary arteries. However, it is very time-consuming and inefficient for doctors to manually segment the intima and adventitia of the coronary arteries in IVUS images. Summary of the invention
[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a coronary artery inner and outer membrane boundary segmentation method based on adjacent scale complementarity, which can quickly and accurately segment the coronary artery inner and outer membrane boundary.
[0004] The first technical solution adopted by the present invention is: a method for segmenting the inner and outer membrane boundaries of coronary arteries based on adjacent scale complementarity, comprising the following steps:
[0005] Acquire the IVUS image to be tested and input the IVUS image to be tested into the deep learning network;
[0006] The deep learning network includes a multi-scale densely linked hole convolution module, an adjacent scale complementary information learning module and an adjacent scale context supplement module;
[0007] The IVUS images to be tested are processed by a multi-scale densely linked dilated convolution module, which extracts features from images at multiple scales and establishes the corresponding relationship between the IVUS images and the intima and adventitia of the coronary arteries.
[0008] The complementary information learning module based on adjacent scales obtains complementary information between scales, and determines the size and position of the coronary artery intima and adventitia according to the complementary information between scales;
[0009] The contextual supplementation module based on adjacent scales obtains complementary information of the context and uses the complementary information of the context to supplement the missing relationships between the inner membrane, the outer membrane and the vascular structure at different scales.
[0010] The segmentation results are output according to the correspondence between the IVUS image and the coronary intima and adventitia, the size and position of the coronary intima and adventitia, and the relationship between the missing intima and adventitia and the vascular structure.
[0011] Furthermore, before the step of acquiring the IVUS image to be tested and inputting the IVUS image to be tested into the deep learning network, the step further includes:
[0012] Construct a training data set and train the deep learning network based on the training data set to obtain a trained deep learning network.
[0013] Furthermore, the step of constructing a training data set and training a deep learning network based on the training data set to obtain a trained deep learning network comprises the following specific steps:
[0014] Collect IVUS images of different health conditions and the corresponding coronary artery intima and adventitia segmentation results as training data sets;
[0015] The deep learning network is trained with IVUS images of different health conditions as input and the corresponding coronary artery intima and adventitia segmentation results as output;
[0016] The network parameters of the deep learning network are adjusted until the error rate reaches a preset range to obtain a trained deep learning network.
[0017] Furthermore, the network parameters of the deep learning network include the number of convolution layers, the number of void convolution layers, the number of BN layers, the type of activation function, the size and number of convolution kernels, the number of pooling layers, the number of upsampling layers, the initial weights and bias values.
[0018] Furthermore, the step of obtaining complementary information between scales by the complementary information learning module based on adjacent scales and determining the size and position of the coronary artery intima and adventitia according to the complementary information between scales specifically includes:
[0019] The complementary information learning module based on adjacent scales obtains the complementary information between scales extracted by the multi-scale densely connected hole convolution module;
[0020] Based on the complementary information between scales, the adjacent large scales guide the adjacent small scales and the adjacent small scales guide the adjacent large scales to determine the size and position of the coronary artery intima and adventitia.
[0021] Furthermore, the adjacent large scale guides the adjacent small scale, and the formula is expressed as follows:
[0022]
[0023]
[0024] In the above formula, represents the small-scale output features, represents the large-scale output feature, S i+1 represents the adjacent small-scale features, S i-1 represents the adjacent large-scale features, λ up ,λ down Indicates a reference to details during the interaction phase, represents element-wise addition, represents the fusion stage through the convolutional layer, F up represents the upsampling process, F down represents the downsampling process.
[0025] Furthermore, the step of obtaining complementary information of the context by the adjacent scale-based context supplementation module and using the complementary information of the context to supplement the missing relationship between the inner membrane, the outer membrane and the vascular structure at different scales specifically includes:
[0026] The contextual supplementation module based on adjacent scales obtains complementary information of the image context at adjacent scales through an interactive method;
[0027] According to the complementary information of the image context, the adjacent large-scale context is integrated to supplement the small-scale context and the adjacent small-scale context is integrated to supplement the large-scale context, and the relationship between the missing inner membrane, outer membrane and vascular structure at different scales is obtained.
[0028] Furthermore, the adjacent large-scale context supplements the small-scale context, and the formula is expressed as follows:
[0029]
[0030]
[0031] In the above formula, S i (u,v) represents small-scale features, S i+1 (h,j) represents the adjacent large-scale features, λ1 represents the adjacent large-scale and adjacent small-scale global context, S w represents the weighted scale, I(·) represents the normalization function, Represents output.
[0032] The beneficial effect of the method of the present invention is as follows: the present invention establishes a correspondence between the IVUS image and the segmentation results of the inner membrane and the outer membrane of the coronary artery by utilizing the self-learning ability of the deep learning network, and determines the segmentation results of the inner membrane and the outer membrane of the current coronary artery corresponding to the current image feature through the correspondence. In addition, the mutual learning of complementary information between scales reduces the interference within different scales, and the context is used to supplement the missing relationships at different scales, thereby improving the segmentation efficiency and making the segmentation results more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 It is a flow chart of the steps of a method for segmenting the inner and outer membranes of coronary arteries based on complementary adjacent scales of the present invention;
[0034] Figure 2 It is a structural block diagram of a deep learning network according to a specific embodiment of the present invention;
[0035] Figure 3 is a schematic diagram of segmentation results in a specific embodiment of the present invention;
[0036] Figure 4 is a schematic diagram of the structure of a complementary information learning module for adjacent scales according to a specific embodiment of the present invention;
[0037] Figure 5 4 is a schematic diagram of the structure of the context supplement module of adjacent scales according to a specific embodiment of the present invention. DETAILED DESCRIPTION
[0038] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.
[0039] like Figure 1 As shown, the present invention provides a method for segmenting the inner and outer membrane boundaries of coronary arteries based on adjacent scale complementarity, the method comprising the following steps:
[0040] S0. Construct a training data set and train the deep learning network based on the training data set to obtain a trained deep learning network;
[0041] S0.1. Collect IVUS images of different health conditions and the corresponding coronary artery intima and adventitia segmentation results as training data sets;
[0042] S0.2, using IVUS images of different health conditions as input and the corresponding coronary artery intima and adventitia segmentation results as output, training a deep learning network;
[0043] S0.3. Adjust the network parameters of the deep learning network until the error rate reaches a preset range to obtain a trained deep learning network.
[0044] Specifically, the network parameters of the deep learning network include the number of convolution layers, the number of void convolution layers, the number of BN layers, the type of activation function, the size and number of convolution kernels, the number of pooling layers, the number of upsampling layers, the initial weights and bias values.
[0045] S1, obtaining the IVUS image to be tested and inputting the IVUS image to be tested into the deep learning network;
[0046] The deep learning network includes a multi-scale densely linked hole convolution module, an adjacent scale complementary information learning module and an adjacent scale context supplement module;
[0047] S2, a multi-scale densely linked dilated convolution module is used to process the IVUS image to be tested, extract features from the images at multiple scales, and establish the corresponding relationship between the IVUS image and the intima and adventitia of the coronary artery;
[0048] S3, a complementary information learning module based on adjacent scales acquires complementary information between scales, and determines the size and position of the coronary artery intima and adventitia according to the complementary information between scales;
[0049] S3.1, the complementary information learning module based on adjacent scales obtains the complementary information between scales extracted by the multi-scale densely linked hole convolution module;
[0050] S3.2. Based on the complementary information between scales, the adjacent large scale guides the adjacent small scale and the adjacent small scale guides the adjacent large scale to determine the size and position of the coronary artery intima and adventitia. For the structural diagram, refer to Figure 4 .
[0051] Adjacent large scales guide adjacent small scales and can be expressed as follows:
[0052]
[0053]
[0054] The adjacent small scale guides the adjacent large scale and can be expressed as:
[0055]
[0056]
[0057] In the above formula, represents the small-scale output features, represents the large-scale output feature, S i+1represents the adjacent small-scale features, S i-1 represents the adjacent large-scale features, λ up ,λ down Indicates a reference to details during the interaction phase, represents element-wise addition, represents the fusion stage through the convolutional layer, F up represents the upsampling process, F down represents the downsampling process.
[0058] S4, a context supplementation module based on adjacent scales obtains complementary information of the context, and uses the complementary information of the context to supplement the missing relationships between the inner membrane and the outer membrane and the vascular structure at different scales;
[0059] S4.1, the context supplementation module based on adjacent scales obtains complementary information of the context of the image at adjacent scales through an interactive method;
[0060] S4.2. Based on the complementary information of the context, the adjacent large-scale context is integrated to supplement the small-scale context and the adjacent small-scale context is integrated to supplement the large-scale context, and the relationship between the missing inner membrane, outer membrane and vascular structure at different scales is obtained. The structural diagram is referred to Figure 5 .
[0061]
[0062]
[0063] In the above formula, S i (u,v) represents small-scale features, S i+1 (h,j) represents the adjacent large-scale features, λ1 represents the adjacent large-scale and adjacent small-scale global context, S w represents the weighted scale, I(·) represents the normalization function, Represents output.
[0064] Adjacent small-scale contexts complement large-scale contexts:
[0065] λ2=σ(-||S i -S i+1 ||)
[0066]
[0067] In the above formula, S i represents the small-scale context, S i+1 Represents the large-scale context, the similarity λ2 is normalized by the function σ, and the small-scale similarity multiplication is normalized. Finally, Represents output.
[0068] S5. Output the segmentation result according to the correspondence between the IVUS image and the coronary intima and adventitia, the size and position of the coronary intima and adventitia, and the relationship between the missing intima and adventitia and the vascular structure.
[0069] A coronary artery inner and outer membrane boundary segmentation device based on adjacent scale complementarity:
[0070] at least one processor;
[0071] at least one memory for storing at least one program;
[0072] When the at least one program is executed by the at least one processor, the at least one processor implements the coronary artery inner and outer membrane boundary segmentation method based on adjacent scale complementarity as described above.
[0073] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0074] A storage medium stores processor-executable instructions, wherein the processor-executable instructions are used to implement the above-mentioned method for segmenting the inner and outer membrane boundaries of coronary arteries based on complementary adjacent scales when executed by the processor.
[0075] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0076] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A coronary artery inner and outer membrane boundary segmentation method based on adjacent scale complementarity, characterized in that: The following steps are involved: Acquire the IVUS image to be tested and input the IVUS image to be tested into the deep learning network; The deep learning network includes a multi-scale densely linked hole convolution module, an adjacent scale complementary information learning module and an adjacent scale context supplement module; The IVUS images to be tested are processed by a multi-scale densely linked dilated convolution module, which extracts features from images at multiple scales and establishes the corresponding relationship between the IVUS images and the intima and adventitia of the coronary arteries. The complementary information learning module based on adjacent scales obtains complementary information between scales, and determines the size and position of the coronary artery intima and adventitia according to the complementary information between scales; The contextual supplementation module based on adjacent scales obtains complementary information of the context and uses the complementary information of the context to supplement the missing relationships between the inner membrane, the outer membrane and the vascular structure at different scales. The segmentation results are output according to the correspondence between the IVUS image and the coronary intima and adventitia, the size and position of the coronary intima and adventitia, and the relationship between the missing intima and adventitia and the vascular structure.
2. The method for segmenting the inner and outer membrane boundaries of coronary arteries based on adjacent scale complementarity according to claim 1, characterized in that: Before the step of acquiring the IVUS image to be tested and inputting the IVUS image to be tested into the deep learning network, the method further includes: Construct a training data set and train the deep learning network based on the training data set to obtain a trained deep learning network.
3. The method for segmenting the inner and outer membrane boundaries of coronary arteries based on adjacent scale complementarity according to claim 2, characterized in that: The step of constructing a training data set and training a deep learning network based on the training data set to obtain a trained deep learning network comprises the following specific steps: Collect IVUS images of different health conditions and the corresponding coronary artery intima and adventitia segmentation results as training data sets; The deep learning network is trained with IVUS images of different health conditions as input and the corresponding coronary artery intima and adventitia segmentation results as output; The network parameters of the deep learning network are adjusted until the error rate reaches a preset range to obtain a trained deep learning network.
4. The method for segmenting the coronary artery inner and outer membrane boundary based on adjacent scale complementarity according to claim 3, characterized in that: The network parameters of the deep learning network include the number of convolution layers, the number of hole convolution layers, the number of BN layers, the type of activation function, the size and number of convolution kernels, the number of pooling layers, the number of upsampling layers, the initial weights and bias values.
5. The method for segmenting the inner and outer membrane boundaries of coronary arteries based on adjacent scale complementarity according to claim 4, characterized in that: The step of obtaining complementary information between scales by the complementary information learning module based on adjacent scales and determining the size and position of the coronary artery intima and adventitia according to the complementary information between scales specifically includes: The complementary information learning module based on adjacent scales obtains the complementary information between scales extracted by the multi-scale densely connected hole convolution module; Based on the complementary information between scales, the adjacent large scales guide the adjacent small scales and the adjacent small scales guide the adjacent large scales to determine the size and position of the coronary artery intima and adventitia.
6. The method for segmenting the inner and outer membrane boundaries of coronary arteries based on adjacent scale complementarity according to claim 5, characterized in that: The adjacent large scale guides the adjacent small scale, and the formula is as follows: The adjacent small scale guides the adjacent large scale as follows: In the above formula, represents the small-scale output features, represents the large-scale output feature, S i+1 represents the adjacent small-scale features, S i-1 represents the adjacent large-scale features, λ up ,λ down Indicates a reference to details during the interaction phase, represents element-wise addition, represents the fusion stage through the convolutional layer, F up represents the upsampling process, F down represents the downsampling process.
7. The method for segmenting the inner and outer membrane boundaries of coronary arteries based on adjacent scale complementarity according to claim 6, characterized in that: The step of obtaining complementary information of the context based on the adjacent scale context supplementation module and using the complementary information of the context to supplement the missing relationship between the inner membrane, the outer membrane and the vascular structure at different scales specifically includes: The contextual supplementation module based on adjacent scales obtains complementary information of the image context at adjacent scales through an interactive method; According to the complementary information of the image context, the adjacent large-scale context is integrated to supplement the small-scale context and the adjacent small-scale context is integrated to supplement the large-scale context, and the relationship between the missing inner membrane, outer membrane and vascular structure at different scales is obtained.
8. A method for segmenting the inner and outer membrane boundaries of coronary arteries based on adjacent scale complementarity according to claim 7, characterized in that: The adjacent large-scale context supplements the small-scale context, and the formula is as follows: In the above formula, S i (u,v) represents small-scale features, S i+1 (h,j) represents the adjacent large-scale features, λ1 represents the adjacent large-scale and adjacent small-scale global context, S w represents the weighted scale, I(·) represents the normalization function, Indicates output.