Method for determining exact boundary of outer elastic membrane of blood vessel lacuna in IVUS image

Abstract

The invention discloses a method for determining a precise boundary of an outer elastic membrane of a blood vessel lacuna in an IVUS image. The core of the invention lies in that an IVUS image segmentation framework is introduced, the framework comprises three main innovative modules: a WCSR module, an MaHS module and a DFCH bridge module, through the three modules, the framework takes wavelet transform as a sampling basis, time frequency analysis is carried out on spatial information, and the time frequency of the spatial information is calculated; meanwhile, the ability of the model to capture global features and local details is enhanced through a multi-attention mechanism, spatial information recovery in the feature up-sampling process is enhanced through convolution with different expansion rates and a multi-layer feature fusion technology, and finally, through an enhanced unsupervised learning method, the spatial information recovery in the feature up-sampling process is enhanced. The model can effectively learn and simulate a visual information vacancy area in the IVUS image without a large amount of annotated data, so that the segmentation precision is improved. Compared with the prior art, the method provided by the invention has the advantages that the image is processed by using wavelet transform, and structural features in the image can be better captured and utilized through innovative application of different modules, so that the segmentation performance is improved.

Description

Technical Field

[0001] The present invention belongs to the fields of medical image processing, deep learning, and signal processing technology, and relates to an IVUS (intravascular ultrasound) image processing technology based on deep learning, and specifically to a method for determining the precise boundary of the external elastic membrane of a vascular cavity in an IVUS image. Background Art

[0002] Atherosclerosis is the leading cause of cardiovascular diseases such as coronary heart disease, myocardial infarction and stroke. According to the World Health Organization (WHO), cardiovascular disease is the number one cause of death worldwide. In 2016, approximately 17.9 million people died from cardiovascular disease worldwide, accounting for 31% of global deaths. Of these deaths, 85% were caused by coronary artery disease (CAD) and stroke. This number is expected to increase to approximately 23.6 million by 2030. Atherosclerosis is an inflammatory disease characterized by the gradual accumulation of lipid plaques in the arterial wall. Although the exact mechanism of atherosclerotic plaque formation is not fully understood, studies have shown that thin-cap fibroatheroma (TCFA) may be a precursor lesion to plaque rupture. Therefore, early assessment of TCFA is crucial for developing treatment and intervention strategies to avoid plaque rupture and effectively manage CAD. It can be seen that the assessment of lumen and vessel wall size and plaque burden plays a key role in the early diagnosis of TCFA.

[0003] With the development of technology, several other invasive intravascular imaging techniques are now available, and IVUS imaging has become the preferred technique in coronary artery assessment due to its unique advantages. The recent large-scale randomized controlled trial IVUS-ACS provides strong support for IVUS imaging technology. The study tested the effects of IVUS-guided implantation of modern drug-eluting stents in a large number of patients with acute coronary syndrome and found that IVUS guidance significantly improved patients' long-term prognosis compared with traditional angiography guidance. Compared with other imaging techniques, IVUS provides higher spatial resolution and penetration depth, can accurately display abnormalities of the coronary arteries, and provide real-time cross-sectional views of the arterial wall, including morphology and pathological features.

[0004] However, due to the complexity of the vascular wall and lumen, traditional image processing methods such as threshold segmentation and edge detection often fail to meet the requirements of accuracy and real-time processing of IVUS images. Although deep learning methods have shown excellent performance in image segmentation tasks in recent years, it is still a huge challenge to accurately identify the boundary between the vascular lumen and the external elastic membrane (EEM) in IVUS images. These challenges include, but are not limited to, the heterogeneity of the vascular wall, calcified plaques in the blood vessels, and noise in vascular images. Summary of the invention

[0005] In the field of interventional cardiovascular imaging, it is extremely critical to accurately determine the boundaries of the external elastic membrane of the vascular cavity in IVUS images. The existing technology often faces challenges in processing such images due to the complexity of the blood vessels and the variation of image quality, such as the unevenness of the vascular wall, the interference of calcified plaques, and the noise in the image, which seriously affect the accuracy of boundary recognition and the processing speed. In order to meet these challenges, the present invention provides a method for determining the precise boundaries of the external elastic membrane of the vascular cavity in IVUS images. The present invention proposes an innovative framework based on deep learning and image processing technology, which combines a multi-channel attention mechanism, wavelet transform, feature calibration, and a novel jump connection strategy to achieve high-precision and high-efficiency automatic segmentation of vascular cavities and external elastic membranes in IVUS images.

[0006] The core technology of the present invention is to design a wavelet-integrated multidimensional tensor harmonic framework WiMuTH (Wavelet-Integrated Multidimensional Tensorial-Harmonic) for IVUS image segmentation, which combines multiple modules to improve the segmentation effect on IVUS boundary segmentation data sets. The present invention includes four key innovative modules: wavelet-based cardiovascular robustness module WCSR (Wavelet-based Cardiovascular Robustness System), MaHS (Multi-Axial Heterogeneous Synthesis) module, intensity distribution corrector module IDC (Intensity Distribution Corrector) and deep feature convergence coordination bridge DFCH (Deep Feature Convergence Harmonization Bridge). In view of the limitations of the existing technology, the present invention provides a new solution that effectively improves segmentation accuracy, compatibility and robustness.

[0007] By applying this technology, the present invention can significantly improve the segmentation effect of IVUS images. Experimental results show that WiMuTH significantly outperforms multiple methods in the field of medical image segmentation in the challenging vascular cavity and external elastic membrane boundary segmentation tasks.

[0008] A method for determining the precise boundary of the external elastic membrane of a vascular cavity in an IVUS image comprises the following steps:

[0009] Step 1: Prepare the initial image and build the dataset:

[0010] Step 2: Construct the wavelet integrated multidimensional tensor harmonic (WiMuTH) framework;

[0011] The wavelet integrated multidimensional tensor harmonic framework is referred to as the WiMuTH framework, which consists of an encoder path, a decoder path and a jump connection part. The backbone network of the framework is a six-layer CNN neural network, and the effect is improved by combining the designed modules on the basis of the six-layer CNN neural network.

[0012] Step 3: Process the input image through the encoder path;

[0013] The structures of the top three layers and the bottom three layers in the encoder path are slightly different. Each of the top three layers consists of a WCSR (Wavelet-based Cardiovascular Robustness System) module and a Conv layer, while each of the bottom three layers consists of a WCSR module and a MaHS (Multi-Axial Heterogeneous Synthesis) module.

[0014] Step 4: Get the predicted segmentation result through the decoder path part;

[0015] The structure of the decoder is similar to that of the encoder. The bottom three layers consist of a WSCR module and a MaHS module, and the top three layers consist of a WSCR module and a Conv layer. The feature parameters in the decoder are continuously upsampled and transmitted upward, and the predicted segmentation results are finally output.

[0016] Step 5: The spatial information is restored during the upsampling process by setting a skip connection between the encoder path and the decoder path;

[0017] The jump connection part of the framework is implemented by the DFCH bridge (Deep Feature Convergence Harmonization Bridge) module, which accepts the feature parameters of the same-layer encoder, the next-layer DFCH bridge module and the previous-layer decoder, and outputs the feature parameters to the same-layer decoder and the previous-layer DFCH bridge module to achieve jump connection.

[0018] Step 6: Perform model training and testing based on the data set obtained in step 1.

[0019] Furthermore, the specific method of step one is as follows:

[0020] The initial image is obtained by intravascular ultrasound (IVUS) technology, and the data set is constructed and divided into training data set and test data set. The initial image contains the morphological features of the vascular lumen, vascular wall and atherosclerotic plaque. The initial image is taken as input and represented as X∈R H×W×D , where H, W, and D represent the height, width, and number of RGB channels of the initial image, respectively.

[0021] Furthermore, the specific method of step three is as follows:

[0022] The prepared initial image is first transmitted to a WCSR module, which uses wavelet transform as the sampling basis and performs wavelet-like transform calculations on the spatial information of the channel dimension in the input image. The specific operation is as follows: First, four filters with convolution are used to filter the input image. The four filters with convolution are low-pass filters f LL , high pass filter f LH 、f HL 、f HH , the filter is defined as follows:

[0023]

[0024] Through these four filters, four feature maps x are obtained respectively. LL 、x LH 、x HL 、x HH ,definition:

[0025]

[0026] in, represents the convolution operation, ↓2 means the convolution step size in the filter is 2. Finally, the Concat function is used to combine the four built-in feature maps x LL 、x LH 、x HL 、x HH Fusion is performed to obtain the output of the current layer WCSR module, i.e., the feature map F WCSR 1, and then pass through a 3*3 Conv layer, and at the same time, the input of the current layer WSCR module is also input into the Conv layer for processing. Finally, the input and output of the current layer WSCR module are concatenated through the Conv layer and the output of the current layer WSCR module through the Concat function to obtain the feature map in the wavelet domain. After that, it is passed to the skip connection part and the WCSR module of the next layer of the encoder. Except for the WCSR module at the top layer in the encoder path and the WCSR module at the bottom layer in the decoder path, the remaining WCSR modules have two inputs, which are the output of the previous encoder concatenated by the Concat function and the output of the Conv layer. In these modules, the two inputs are filtered and then get F WCSR (i+1), and finally concatenate through the Concat function to get the output of the WCSR module of the current layer.

[0027] In the lower three layers of the encoder, the MaHS (Multi-Axial Heterogeneous Synthesis) module is used to replace the Conv layer in the previous layer. The MaHS module first receives the feature map output by the Concat function from the previous encoder. The feature map from the previous layer contains important spatial and frequency information. The feature map is evenly split in the dimension of the number of channels C and divided into four sub-feature maps F in , n = 1, 2, 3, 4, so that they can be processed separately. Different attention mechanisms are applied to the four feature maps, including self-attention module, convolutional block attention module, squeeze-excitation attention module and Conv layer. Three of the attention formulas are as follows:

[0028] Self-attention: For the input sub-feature map, a transformation is performed to produce a query Q, key K, and value V matrix. The formula for calculating the attention score A is:

[0029]

[0030] Among them, d k is the dimension of the key vector, which helps stabilize the gradients during training.

[0031] Convolutional Block Attention (CBA): CBA is implemented through two sub-modules: channel attention and spatial attention.

[0032] The formula of the channel attention module is:

[0033] M c =σ(MLP(AvgPool(F in ))+MLP(MaxPool(F in )))

[0034] The formula of the spatial attention module is:

[0035] M s =σ(f 7×7 ([AvgPool(F in );MaxPool(F in )]))

[0036] Where σ represents the Sigmoid function, f 7×7 It is a 7×7 convolution operation.

[0037] Squeeze-and-Excitation Attention (SEA): SEA first squeezes the input sub-feature map through global average pooling, and then excites the features through two fully connected layers. The formula is as follows:

[0038] s=σ(W 2 δ(W 1 (AvgPool(F in ))))

[0039] Where δ is the ReLU function, σ is the Sigmoid function, and W 1 and W 2 are the weights of the fully connected layer.

[0040] Finally, the output of the WCSR module and the MaHS module of the last layer of the encoder is concatenated by the Concat function to obtain the output F of the bottom layer of the encoder. i .

[0041] Furthermore, the specific method of step 4 is as follows:

[0042] The structure of the decoder is similar to that of the encoder. The bottom three layers are composed of a WSCR module and a MaHS module, and the top three layers are composed of a WSCR module and a Conv layer. The output of the encoder bottom layer is transmitted to the WSCR module at the bottom of the decoder, and then upsampled and transmitted upward to the MaHs module, and then transmitted to the WCSR module of the next layer. After that, the WCSR module of each layer will receive the feature parameters from the DCFH bridge module and the parameters output by the previous decoder layer. In the top three layers of the decoder, the Conv layer is used to replace the MaHS module. At the same time, the MaHS module and Conv layer of each layer will transmit the feature parameters to the DFCH bridge module of the jump connection part of the same layer. The feature parameters in the decoder are continuously upsampled and transmitted upward, and finally the predicted segmentation result is output.

[0043] Furthermore, the skip connection part is as follows:

[0044] The jump connection is realized by the DFCH bridge module. First, the output feature map F of the encoder bottom layer is i The DFCH bridge module at the bottom layer also receives the output feature map F from the encoder bottom layer. i , the output feature map of the same-layer encoder spliced ​​by the Concat function, and the input from the IDC module. The DFCH bridge modules at other levels simultaneously receive the output of the lower-layer DFCH bridge module, the output feature map of the same-layer encoder spliced ​​by the Concat function, and the input from the IDC module. The feature map input to the DFCH bridge module is split into multiple sub-feature sets by the feature splitting module (Align Split, AS). The feature splitting module first converts the output F from the lower-layer DFCH bridge module into a sub-feature set. DFCH(i+1) passes through a Conv layer and an AvgPool+Share_Conv layer respectively, and the outputs of the two are concatenated through the Concat function, and then F DFCH (i+1) After the Hadamard product operation is performed on the output of the Share_Conv layer and the Concat function, it is evenly divided into four parts on the channel dimension C, denoted as

[0045] use represents the feature map output by the i-th layer encoder in the downsampling process, It is also divided into four equal parts on the channel dimension C, recorded as These four sub-feature maps are respectively Merge and add the single-layer features F transmitted from the intensity distribution corrector module IDC (The Integro-Dimensional Calibration) IDC(i+1) , this single-layer feature is obtained by the feature parameters transmitted by the MaHS module or Conv layer of the previous decoder through the IDC module. The IDC module combines Conv2d (two-dimensional convolution layer) and Group Normalization to generate a 1×H×W feature factor by manipulating the input features. Here, the ReLU function is specified as group normalization. The specific formula is as follows:

[0046] F IDCi =ReLU(Conv2d(F i ))

[0047] These three sets of characteristic data: F IDCn , n=1, 2, 3, 4, four sets of feature data are obtained through Concat operation, and then projected using G 0 , G 1 , G 2 and G 3 The four projection methods are processed separately. These four projection methods are composed of Conv2d with different dilations (1, 3, 5 and 7) and layer normalization. Finally, the feature maps obtained by the four projection methods are connected in series through the Concat function to produce the final output of the DFCH bridge module of the i-th layer. The output of each layer of DFCH bridge is transmitted to the WCSR module of the corresponding layer of the decoder and the DFCH bridge module of the previous layer to complete the jump connection operation.

[0048] Furthermore, the specific method of step six is ​​as follows:

[0049] During the training process, the WiMuTH framework consisting of the encoder path, the jump connection part, and the decoder path is trained with the training data in the dataset obtained in step 1, and the edema area in the unseen IVUS images is identified and predicted by the WiMuTH framework. The MSE loss function is selected in the training process to evaluate the difference between the model output and the actual annotation, and the weight of the model is continuously optimized through back propagation, gradient descent, dynamic learning rate adjustment, and regularization techniques. After the training is completed, the WiMuTH framework is tested with the test data in the dataset.

[0050] The beneficial effects of the present invention are as follows:

[0051] The core technology of the present invention is the introduction of an IVUS image segmentation framework WiMuTH Framework, which contains four key innovations. WCSR enhances the model's frequency domain analysis capability of images through wavelet transform, effectively improving the processing accuracy of dynamic images, especially dynamic artifacts in cardiac images; MaHS significantly improves the feature capture capability of complex vascular areas by integrating multiple attention mechanisms and convolution operations; IDC ensures the integrity of information during transmission through deep normalization, improving the accuracy and consistency of image segmentation; DFCH bridge utilizes multi-layer features and convolutions with different expansion rates to enhance the ability to capture image edges and details, especially the recovery of spatial information during upsampling. These innovations improve the overall performance and application effect of image analysis. Compared with the prior art, this method helps to better utilize wavelet transform to process image features and improve segmentation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a general framework diagram of an embodiment of the present invention.

[0053] Figure 2 A diagram of the framework of the small modules in the module designed for the present invention.

[0054] Figure 3 This is a comparison chart of the model effects.

[0055] Figure 4 The specific verification results of different network models on Lumen and EEM segmentation tasks on the HDS-IVUS dataset are shown. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0057] Glossary

[0058] IVUS (Intravascular Ultrasound): It is a medical imaging technology that uses the principle of ultrasound to transmit and receive ultrasound through a probe inserted into a blood vessel to obtain high-resolution two-dimensional cross-sectional images of the inner wall of the blood vessel and surrounding tissues. This technology is mainly used in the cardiovascular field, especially in the evaluation and treatment of coronary artery disease. IVUS can provide doctors with detailed information on coronary artery stenosis, plaque composition and its distribution to help formulate treatment strategies. In addition, IVUS can also be used to monitor the progression of vascular disease and the effectiveness of interventional surgery.

[0059] Wavelet Transform: It is a mathematical function processing technology that provides a detailed description of the local features within the signal by analyzing the data or signal in time and frequency simultaneously. Compared with the traditional Fourier transform, the wavelet transform can provide better time resolution at the signal mutation point, so it is particularly suitable for analyzing non-stationary signals. By expanding and translating the mother wavelet (basis function), the wavelet transform can perform multi-scale analysis on the signal, thereby capturing features at different scales. This transform has a wide range of applications in many fields such as image compression, denoising, feature extraction, etc., such as in medical imaging, seismic data analysis and multimedia processing.

[0060] Segmentation of Lumen and External Elastic Lamina Boundaries: This is an image processing technology that is mainly used in medical image analysis, especially in the diagnosis and research of vascular diseases. By accurately segmenting the lumen (cavity) and external elastic membrane (boundary structure outside the blood vessel) of the blood vessel, we can better understand the structure and functional status of the blood vessels and evaluate the extent of vascular lesions. This segmentation task usually requires high precision and high sensitivity because the boundaries of the blood vessels may not be obvious in medical images or may be interfered with by surrounding tissues. In practical applications, this task usually uses advanced computer vision technology and machine learning algorithms, such as deep learning models, to perform accurate boundary detection in an automated manner, and is widely used in the diagnosis and treatment planning of cardiovascular diseases.

[0061] Specifically, the present invention proposes an automatic segmentation method for determining the boundaries of lumens and EEMs in IVUS images, which aims to learn to understand the original IVUS input and map the original pixels to the corresponding categories. In order to enable the model to learn more complete and detailed position feature information and position feature relationships, we propose a Wavelet Convolutional Spectrum-Reshaper (WCSR) module and a MaHS module, which aim to inject the model with the simulated representation of IVUS images in the wavelet domain and use multi-head heterogeneous attention to emphasize the relationship between positions. WiMuTH is divided into two branches, namely the adaptive position-aware spatial path and the guided position mapping spatial path; the adaptive position-aware spatial path of WiMuTH uses discrete IVUS images X i As input, it generates the model parameters θ of the location-aware sensitivity enhancement i As output. To achieve this goal, we first introduced a frequency domain-based encoder to mask the original IVUS image using the edge information in the IVUS image. Secondly, we used the backbone network to process the masked feature map F and generate the reconstructed IVUS image. The backbone network uses the input as a label to update the output model parameters θ i Guided position mapping spatial path: In order to effectively solve the blurring problem caused by a large number of artifacts and angiography in IVUS images due to the influence of blood, plaque components, vascular curvature, and imaging devices, we integrated a model parameter θ with position awareness i This fusion can improve the model's sensitivity to blank areas and can more accurately understand and restore the texture information and morphological structure of blank areas in IVUS images. On the basis of fusion, the labels are replaced with the results of manual annotation. Then, the backbone network is used to map the original image pixels to the segmentation feature map, and the segmentation boundary of the segmentation feature map is processed. We obtain the final EEM and Lumen boundary. The overall model structure is as follows: Figure 1 shown. Figure 2 A diagram of the framework of the small modules in the module designed for the present invention.

[0062] The core of the present invention is to introduce an IVUS image segmentation framework, which uses wavelet transform as the sampling basis to perform time-frequency analysis on spatial information, and at the same time enhances the model's ability to capture global features and local details through a multi-attention mechanism. Then, by using convolutions with different expansion rates and multi-layer feature fusion technology, the spatial information recovery in the feature upsampling process is enhanced. Finally, through an enhanced unsupervised learning method, the model can effectively learn and simulate visual information vacant areas in IVUS images without a large amount of labeled data, thereby improving segmentation accuracy.

[0063] The specific implementation process is as follows:

[0064] Step 1: Prepare the initial image and build the dataset:

[0065] The initial image is obtained through intravascular ultrasound (IVUS) technology, and the data set is constructed and divided into training data set and test data set. This technology can capture the cross-sectional image of the blood vessel. The initial image contains the morphological characteristics of the vascular lumen, vascular wall and atherosclerotic plaque. The initial image is taken as input and represented as X∈R H×W×D , where H, W, and D represent the height, width, and number of RGB channels of the initial image, respectively.

[0066] Step 2: Construct the wavelet integrated multidimensional tensor harmonic (WiMuTH) framework;

[0067] The wavelet integrated multidimensional tensor harmonic framework, referred to as WiMuTH, consists of an encoder path, a decoder path and a jump connection part. The backbone network of the framework is a six-layer CNN neural network, which is combined with the designed modules on the basis of the six-layer CNN neural network to improve the effect. In the encoder path and the decoder path, the structures of the upper three layers and the lower three layers are slightly different. Each of the upper three layers consists of a WCSR (Wavelet-based Cardiovascular Robustness System) module and a Conv layer, while each of the lower three layers consists of a WCSR module and a MaHS (Multi-Axial Heterogeneous Synthesis) module. The jump connection part of the framework is implemented by the DFCH bridge (Deep Feature Convergence Harmonization Bridge) module, which accepts the feature parameters of the same-layer encoder, the next-layer DFCH bridge module and the previous-layer decoder, and outputs the feature parameters to the same-layer decoder and the previous-layer DFCH bridge module to achieve jump connection.

[0068] Step 3: Step 3: Encoder path part;

[0069] The prepared initial image is first transmitted to a WCSR module, which uses wavelet transform as the sampling basis and performs wavelet-like transform calculations on the spatial information of the channel dimension in the input image. The specific operation is as follows: First, four filters with convolution are used to filter the input image. The four filters with convolution are low-pass filters f LL , high pass filter f LH 、f HL 、f HH , the filter used in the present invention is defined as follows:

[0070]

[0071] Through these four filters, four feature maps x are obtained respectively. LL 、x LH 、x HL 、x HH ,definition:

[0072]

[0073] in, represents the convolution operation, ↓2 means the convolution step size in the filter is 2. Finally, the Concat function is used to combine the four built-in feature maps x LL 、x LH 、x HL 、x HH Fusion is performed to obtain the output of the current layer WCSR module, i.e., the feature map F WCSR 1, and then pass through a 3*3 Conv layer, and at the same time, the input of the current layer WSCR module is also input into the Conv layer for processing. Finally, the input and output of the current layer WSCR module are concatenated through the Conv layer and the output of the current layer WSCR module through the Concat function to obtain the feature map in the wavelet domain. After that, it is passed to the skip connection part and the WCSR module of the next layer of the encoder. Except for the WCSR module at the top layer in the encoder path and the WCSR module at the bottom layer in the decoder path, the remaining WCSR modules have two inputs, which are the output of the previous encoder concatenated by the Concat function and the output of the Conv layer. In these modules, the two inputs are filtered and then get F WCSR (i+1), and finally concatenate through the Concat function to get the output of the current layer WCSR module.

[0074] In the lower three layers of the encoder, the MaHS (Multi-Axial Heterogeneous Synthesis) module is used to replace the Conv layer in the previous layer. The MaHS module first receives the feature map output by the Concat function from the previous encoder. The feature map from the previous layer contains important spatial and frequency information. The feature map is evenly split in the dimension of the number of channels C and divided into four sub-feature maps F in , n = 1, 2, 3, 4, so that they can be processed separately. Different attention mechanisms are applied to the four feature maps, including self-attention module, convolutional block attention module, squeeze-excitation attention module and Conv layer. Three of the attention formulas are as follows:

[0075] Self-attention: For the input sub-feature map, a transformation is performed to produce a query Q, key K, and value V matrix. The formula for calculating the attention score A is:

[0076]

[0077] Among them, d k is the dimension of the key vector, which helps stabilize the gradients during training.

[0078] Convolutional Block Attention (CBA): CBA is implemented through two sub-modules: channel attention and spatial attention.

[0079] The formula of the channel attention module is:

[0080] M c =σ(MLP(AvgPool(F in ))+MLP(MaxPool(F in )))

[0081] The formula of the spatial attention module is:

[0082] M s =σ(f 7×7 ([AvgPool(F in );MaxPool(F in )]))

[0083] Where σ represents the Sigmoid function, f 7×7 It is a 7×7 convolution operation.

[0084] Squeeze-and-Excitation Attention (SEA): SEA first squeezes the input sub-feature map through global average pooling, and then excites the features through two fully connected layers. The formula is as follows:

[0085] s=σ(W 2 δ(W 1 (AvgPool(F in ))))

[0086] Where δ is the ReLU function, σ is the Sigmoid function, and W 1 and W 2 are the weights of the fully connected layer.

[0087] Finally, the output of the WCSR module and the MaHS module of the last layer of the encoder is concatenated by the Concat function to obtain the output F of the bottom layer of the encoder. i .

[0088] Step 4: The structure of the decoder is similar to that of the encoder. The bottom three layers are composed of a WSCR module and a MaHS module, and the top three layers are composed of a WSCR module and a Conv layer. The output of the encoder bottom layer is transmitted to the WSCR module at the bottom of the decoder, and then upsampled and transmitted upward to the MaHs module, and then transmitted to the WCSR module of the next layer. After that, the WCSR module of each layer will receive the feature parameters from the DCFH bridge module and the parameters output by the previous decoder layer. In the top three layers of the decoder, the Conv layer is used to replace the MaHS module. At the same time, the MaHS module and Conv layer of each layer will transmit the feature parameters to the DFCH bridge module of the jump connection part of the same layer. The feature parameters in the decoder are continuously upsampled and transmitted upward, and finally the predicted segmentation result is output.

[0089] Step 5: Jump connection part;

[0090] In the present invention, the jump connection is realized by the DFCH bridge module. First, the output feature map F of the encoder bottom layer is i The DFCH bridge module at the bottom layer also receives the output feature map F from the encoder bottom layer. i , the output feature map of the same-layer encoder spliced ​​by the Concat function, and the input from the IDC module. The DFCH bridge modules at other levels simultaneously receive the output of the lower-layer DFCH bridge module, the output feature map of the same-layer encoder spliced ​​by the Concat function, and the input from the IDC module. The feature map input to the DFCH bridge module is split into multiple sub-feature sets by the feature splitting module (AlignSplit, AS). The feature splitting module firstly outputs the output F from the lower-layer DFCH bridge module. DFCH (i+1) passes through a Conv layer and an AvgPool+Share_Conv layer respectively, and the outputs of the two are concatenated through the Concat function, and then F DFCH (i+1) After the Hadamard product operation is performed on the output of the Share_Conv layer and the Concat function, it is evenly divided into four parts on the channel dimension C, denoted as

[0091] use represents the feature map output by the i-th layer encoder in the downsampling process, It is also divided into four equal parts on the channel dimension C, recorded as These four sub-feature maps are respectively Merge and add the single-layer features F transmitted from the intensity distribution corrector module IDC (The Integro-Dimensional Calibration)IDC(i+1) , this single-layer feature is obtained by the feature parameters transmitted by the MaHS module or Conv layer of the previous decoder through the IDC module. This module is used to reduce the impact of temporal forgetting. The IDC module combines Conv2d (two-dimensional convolution layer) and GroupNormalization (group normalization) to generate a 1×H×W feature factor by manipulating the input features. Here, the ReLU function is specified as group normalization. The specific formula is as follows:

[0092] F IDCi =ReLU(Conv2d(F i ))

[0093] These three sets of characteristic data: F IDCn , n=1, 2, 3, 4, four sets of feature data are obtained through Concat operation, and then projected using G 0 , G 1 , G 2 and G 3 The four projection methods are processed separately. These four projection methods are composed of Conv2d with different dilations (1, 3, 5 and 7) and layer normalization. Finally, the feature maps obtained by the four projection methods are connected in series through the Concat function to produce the final output of the DFCH bridge module of the i-th layer. The output of each layer of DFCH bridge is transmitted to the WCSR module of the corresponding layer of the decoder and the DFCH bridge module of the previous layer to complete the jump connection operation.

[0094] Except for the bottom DFCH layer, the rest of the DFCH layer modules They are all directly generated by the output of the DFCH layer module of the upper layer. The jump connection operation is completed by transmitting parameters through these modules.

[0095] Step 6: At the output of the network, the probability of each pixel belonging to the edema area can be obtained. The output is usually a segmentation map of the same size as the input image, and the value of each pixel represents the probability that the pixel belongs to the edema area.

[0096] During the training process, the labeled training data obtained in the preprocessing stage is used, which contains IVUS images and their corresponding edema area annotations. With this data, the model can be trained to identify and predict edema areas in unseen IVUS images. The MSE loss function is selected for training to evaluate the difference between the model output and the actual annotation, and the model weights are continuously optimized through back propagation, gradient descent, dynamic learning rate adjustment and regularization techniques.

[0097] The framework mentioned in the present invention was experimented on the public data set HDS-IVUS. Specifically, different pixel contents in the data set are different pixel values. At the beginning of training, the category will be assigned a probability value of 1, and the background part will be assigned a probability value of 0. The corresponding probability value will be output through the forward propagation of the model, and then the loss for each pixel is obtained through the MSE loss function. The losses calculated for all pixels of the entire image or batch are accumulated to obtain the final loss value. Through back propagation, the gradient of the loss relative to each model parameter is calculated. This gradient represents the sensitivity of the loss function to each parameter under the current model parameters. The optimizer adjusts the value of each parameter in the model according to the calculated gradient and optimizes the model weights. Finally, by continuously iteratively updating the parameters, the minimum value of the loss function is gradually approached to achieve the optimization of the segmentation effect.

[0098] The trained model can be used for inference, that is, to make predictions on new IVUS images. This process involves putting the new image through the same preprocessing, then using the trained model to segment it, and finally generating a segmentation result, outputting an image predicted to be the edema part. In the experiment, the WiMuTH framework achieved an accuracy of 92.29% for the Lumen segmentation and 77.85% for the EEM segmentation in the HDS-IVUS dataset, both of which are the best compared to other networks. The specific results are shown in the figure below. Figure 3 Visually, the figure shows the calcification IVUS segmentation results for the HDS-IVUS dataset, including: (a) AttUNet; (b) EUNet; (c) FCN-8s; (d) GCN; (e) LEDNet; (f) OCNet; (g) PSPNet; (h) TransUNet; (i) UNet; (j) BiseNet; (k) Deeplab; (l) WiMuTH. The light green solid line in the figure represents the machine annotation, and the red dotted line represents the annotation of human experts. Figure 4 The experiments conducted on the HDS-IVUS public dataset are shown, and it can be seen that the framework proposed in the present invention has achieved multiple best segmentation results in Lumen and EEM.

[0099] Figure 4 The specific verification results of different network models on Lumen and EEM segmentation tasks on the HDS-IVUS dataset are shown. Figure 4The table in shows the performance metrics of various segmentation models, including HD (Hausdorff distance), 95HD (95% Hausdorff distance), Dice coefficient and IoU (intersection over union). The comparison models include models based on U-Net and its variants; Attention U-Net and TransNet; Classical semantic segmentation network; Fully Convolutional Network (FCN-8s), Pyramid Scene Paring Network (PSPNet), DeepLab and Object Context Network (OCNet). Lightweight network: ENet, LEDNet and Bilateral Segmentation Network (BiseNet). Each model is evaluated based on its performance in accurately delineating Lumen and EEM regions in IVUS images. Models are listed by their reference year. Our proposed model WiMuTH framework shows competitive performance, especially showing top metric scores in the cells marked in red, while the cells marked in blue indicate suboptimal metric scores.

[0100] The core of the present invention is to introduce an IVUS image segmentation framework, which includes three main innovative modules: WCSR module, MaHS module and DFCH bridge module. Through these three modules, the framework uses wavelet transform as the sampling basis to perform time-frequency analysis on spatial information. At the same time, the model's ability to capture global features and local details is enhanced through the multi-attention mechanism. Then, by using convolution with different expansion rates and multi-layer feature fusion technology, the spatial information recovery in the feature upsampling process is enhanced. Finally, through the enhanced unsupervised learning method, the model can effectively learn and simulate the visual information vacant areas in IVUS images without a large amount of labeled data, thereby improving the segmentation accuracy. Compared with the prior art, this method uses wavelet transformation to process images, and through the innovative application of different modules, it helps to better capture and utilize the structural features in the image, thereby improving the segmentation performance.

[0101] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0102] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A method for determining the precise boundary of the external elastic membrane of a vascular cavity in an IVUS image, characterized in that: The steps include: Step 1: Prepare the initial image and build the dataset: Step 2: Construct a wavelet integrated multidimensional tensor harmonic framework; The wavelet integrated multidimensional tensor harmonic framework is referred to as the WiMuTH framework, which consists of an encoder path, a decoder path and a jump connection part; the backbone network of the framework is a six-layer CNN neural network, and the effect is improved by combining the designed modules on the basis of the six-layer CNN neural network; Step 3: Process the input image through the encoder path; The structures of the top three layers and the bottom three layers in the encoder path are slightly different. Each of the top three layers consists of a WCSR module and a Conv layer, while each of the bottom three layers consists of a WCSR module and a MaHS module. Step 4: Get the predicted segmentation result through the decoder path part; The structure of the decoder is similar to that of the encoder. The bottom three layers consist of a WSCR module and a MaHS module, and the top three layers consist of a WSCR module and a Conv layer. The feature parameters in the decoder are continuously upsampled and transmitted upward, and the predicted segmentation results are finally output. Step 5: The spatial information is restored during the upsampling process by setting a skip connection between the encoder path and the decoder path; The jump connection part of the framework is realized by the DFCH bridge module, which accepts the characteristic parameters of the same-layer encoder, the next-layer DFCH bridge module and the previous-layer decoder, and outputs the characteristic parameters to the same-layer decoder and the previous-layer DFCH bridge module to realize the jump connection; Step 6: Perform model training and testing based on the data set obtained in step 1.

2. A method for determining the precise boundary of the external elastic lamina of a vascular cavity in an IVUS image according to claim 1, characterized in that: Step 1: The initial image is obtained by intravascular ultrasound IVUS technology, and the data set is constructed and divided into a training data set and a test data set; the initial image contains the morphological characteristics of the vascular cavity, vascular wall and atherosclerotic plaque; the initial image is used as input, represented as X∈R H×W×D , where H, W, and D represent the height, width, and number of RGB channels of the initial image, respectively.

3. A method for determining the precise boundary of the external elastic lamina of a vascular cavity in an IVUS image according to claim 1, characterized in that: Step 3: The prepared initial image is first transmitted to a WCSR module, which uses wavelet transform as a sampling basis and performs wavelet-like transform calculations on the spatial information of the channel dimension of the input image. The specific operations are as follows: First, four filters with convolution are used to filter the input image, and the four filters with convolution are low-pass filters f LL , high pass filter f LH 、f HL 、f HH , the filter is defined as follows: Through these four filters, four feature maps x are obtained respectively. LL 、x LH 、x HL 、x HH ,definition: in, Represents the convolution operation, ↓2 means the convolution step size in the filter is 2; finally, the Concat function is used to combine the four built-in feature maps x LL 、x LH 、x HL 、x HH Fusion is performed to obtain the output of the current layer WCSR module, i.e., the feature map F WCSR 1, and then pass through a 3*3 Conv layer, and at the same time input the input of the current layer WSCR module into the Conv layer for processing; finally, the input and output of the current layer WSCR module are concatenated through the Conv layer and the output of the current layer WSCR module through the Concat function to obtain the feature map Fi in the wavelet domain d After that, it is passed to the jump connection part and the WCSR module of the next layer of the encoder; except for the WCSR module at the top of the encoder path and the WCSR module at the bottom of the decoder path, the remaining WCSR modules have two inputs, which are the output of the previous encoder spliced ​​by the Concat function and the output of the Conv layer. In these modules, the two inputs are obtained after passing through the filter F WCSR (i+1), and finally concatenate them through the Concat function to get the output of the WCSR module of the current layer; In the lower three layers of the encoder, the MaHS module is used to replace the Conv layer in the previous layer. The MaHS module first receives the feature map output by the Concat function from the previous encoder. The feature map from the previous layer contains important spatial and frequency information. The feature map is evenly divided in the dimension of the number of channels C into four sub-feature maps F in , n = 1, 2, 3, 4, so as to facilitate separate processing; different attention mechanisms are applied to the four feature maps, including self-attention module, convolutional block attention module, squeeze-excitation attention module and Conv layer; three of the attention formulas are as follows: Self-attention: For the input sub-feature map, a transformation is performed to produce a query Q, key K, and value V matrix; the formula for calculating the attention score A is: Among them, d k is the dimension of the key vector, which helps stabilize the gradient during training; Convolutional Block Attention (CBA): CBA is implemented through two sub-modules: channel attention and spatial attention; The formula of the channel attention module is: M c =σ(MLP(AvgPool(F in ))+MLP(MaxPool(F in ))) The formula of the spatial attention module is: M s =σ(f 7×7 ([AvgPool(F in );MaxPool(F in )])) Where σ represents the Sigmoid function, f 7×7 It is a 7×7 convolution operation; Squeeze-Excite Attention SEA: SEA first squeezes the input sub-feature map through global average pooling, and then excites the features through two fully connected layers. The formula is as follows: s=σ(W2δ(W1(AvgPool(F in )))) Where δ is the ReLU function, σ is the Sigmoid function, W1 and W2 are the weights of the fully connected layer; Finally, the output of the WCSR module and the MaHS module of the last layer of the encoder is concatenated by the Concat function to obtain the output F of the bottom layer of the encoder. i .

4. A method for determining the precise boundary of the external elastic lamina of a vascular cavity in an IVUS image according to claim 1, characterized in that: Step 4: The specific method is as follows: The structure of the decoder is similar to that of the encoder. The bottom three layers are composed of a WSCR module and a MaHS module, and the top three layers are composed of a WSCR module and a Conv layer. The output of the encoder's bottom layer is transmitted to the WSCR module at the bottom of the decoder, and then upsampled and transmitted upward to the MaHs module, and then transmitted to the WCSR module of the next layer. After that, the WCSR module of each layer will receive the feature parameters from the DCFH bridge module and the parameters output by the previous decoder layer. In the top three layers of the decoder, the Conv layer is used to replace the MaHS module. At the same time, the MaHS module and Conv layer of each layer will transmit the feature parameters to the DFCH bridge module of the jump connection part of the same layer. The feature parameters in the decoder are continuously upsampled and transmitted upward, and finally the predicted segmentation result is output.

5. The method for determining the precise boundary of the external elastic lamina of a vascular cavity in an IVUS image according to claim 1, characterized in that: The skip connection part is as follows: The jump connection is realized by the DFCH bridge module. First, the output feature map F of the encoder bottom layer is i is input to the bottom DFCH bridge module; the bottom DFCH bridge module also receives the output feature map F from the bottom of the encoder i , the output feature map of the encoder at the same layer concatenated by the Concat function, and the input from the IDC module; The DFCH bridge modules at other levels simultaneously receive the output of the DFCH bridge module at the lower level, the output feature map of the encoder at the same level concatenated by the Concat function, and the input from the IDC module; the feature map input to the DFCH bridge module is split into multiple sub-feature sets by the feature splitting module. The feature splitting module first converts the output F from the DFCH bridge module at the lower level into DFCH (i+1) passes through a Conv layer and an AvgPool+Share_Conv layer respectively, and the outputs of the two are concatenated through the Concat function, and then F DFCH (i+1) After the Hadamard product operation is performed on the output of the Share_Conv layer and the Concat function, it is evenly divided into four parts on the channel dimension C, denoted as use represents the feature map output by the i-th layer encoder in the downsampling process, It is also divided into four equal parts on the channel dimension C, recorded as These four sub-feature maps are respectively Merge and add the single-layer features F transferred from the intensity distribution corrector module IDC IDC(i+1) , this single-layer feature is obtained by the feature parameters transmitted by the MaHS module or Conv layer of the previous decoder through the IDC module; the IDC module combines Conv2d and GroupNormalization to generate a 1×H×W feature factor by manipulating the input features; here the ReLU function is specified as group normalization; the specific formula is as follows: F IDCi =ReLU(Conv2d(F i )) These three sets of feature data: Through the Concat operation, four sets of feature data are obtained, and then they are processed respectively using projections G0, G1, G2 and G3. These four projection methods are composed of Conv2d with different expansions and layer normalizations. Finally, the feature maps obtained by the four projection methods are connected in series through the Concat function to generate the final output of the DFCH bridge module of the i-th layer. The output of each layer of DFCH bridge is transmitted to the WCSR module of the corresponding layer of the decoder, as well as the DFCH bridge module of the previous layer to complete the jump connection operation.

6. A method for determining the precise boundary of the external elastic lamina of a vascular cavity in an IVUS image according to claim 1, characterized in that: Step 6: The specific method is as follows: During the training process, the WiMuTH framework consisting of an encoder path, a skip connection part, and a decoder path is trained by the training data in the dataset obtained in step 1, and the edema area in the unseen IVUS image is identified and predicted by the WiMuTH framework; The training process selects the MSE loss function to evaluate the difference between the model output and the actual annotation, and continuously optimizes the model weights through back propagation, gradient descent, dynamic learning rate adjustment, and regularization techniques; after training, the WiMuTH framework is tested using the test data in the dataset.

Images