A hybrid lateral connection based multi-channel network and its applications

Abstract

The application discloses a kind of multi-channel networks based on mixed lateral connection, first, using multimodal encoder extracts multimodal image data, while using SE module increases channel attention. Then using the method of lateral connection, the same size features of different modalities obtained after each down-sampling are fused. Finally, these mixed features are spliced with the features obtained during the up-sampling process of the decoder, so as to maximize the use of the features of each modality. The network structure well solves the feature fusion problem of multiple encoders and can be applied to clinical multi-modal imaging tasks. Experimental results also show that the mixed lateral connection network proposed by the application can achieve the best effect in TC and ET segmentation, and can be popularized to other image segmentation tasks.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multi-channel network based on hybrid lateral connections and its applications. Background Technology

[0002] In recent years, deep learning-based multimodal medical image segmentation has demonstrated state-of-the-art performance, largely due to input-level fusion strategies: directly concatenating multimodal images as multi-channel input, and then training the segmentation network. These works focus on dense or cross-functional fusion of features along deep paths. For example, U-Net uses symmetric connections to effectively fuse high- and low-order features. However, such structures struggle to preserve the features of each modality itself.

[0003] To effectively extract features from various modalities, multi-encoders have begun to be applied in the field of research. For example, Kumar et al. (Kumar A, Fulham M, Feng D, et al. Co-learning feature fusion maps from PET-CT images of lung cancer[J].IEEE Transactions on Medical Imaging,2019,39(1):204-217) used two encoders to encode positron emission tomography (PET) and T1 images respectively, and then performed feature fusion, which can more effectively extract the information contained in the two types of images. Jose Dolz et al. (Dolz J, Gopinath K, Yuan J, et al. HyperDense-Net: a hyper-densely connected CNN for multi-modal image segmentation[J].IEEE transactions on medical imaging,2018,38(5):1116-1126.) used the DenseNet structure with multiple encoders to perform multimodal semantic segmentation of brain images, and achieved competitive performance in brain segmentation.

[0004] Classic networks like U-Net use input-level fusion to effectively combine high- and low-order features. However, such structures struggle to preserve the features of each modality itself. Decision-level fusion decodes by directly concatenating learned features with high-dimensional features from other modalities. While decision-level fusion can learn complex and complementary feature information compared to input-level fusion networks, its decoding method—directly concatenating learned single-modal features with features from other modalities—is too simplistic and fails to effectively utilize high-order features. Further integrating various information sources and effectively improving and applying them to neural networks to enhance lesion segmentation and recognition capabilities remains a challenge in clinical research. Summary of the Invention

[0005] This invention aims to efficiently fuse high-order features of various modes while effectively preserving the characteristics of each mode itself, and proposes a multi-channel network based on hybrid lateral connections and its application.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] On one hand, the present invention provides a multi-channel network based on hybrid lateral connections, with a multimodal encoder as the main structure and a lateral connection structure used in multimodal image decoding, and includes the following steps:

[0008] S1. Multimodal image data is extracted using a multimodal encoder, while channel attention is increased using a squeeze-and-excitation (SE) module.

[0009] S2. Using the Hybrid Lateral Connection method, the same-size features of different modalities obtained after each downsampling are fused to obtain hybrid features;

[0010] S3. The mixed features from step S2 are concatenated with the features obtained during the decoder upsampling process and input into the decoder.

[0011] Furthermore, in step S1, each modal data is encoded using a separate encoder, with a total of four downsampling operations, and then the results from multiple encoders are combined for decoding.

[0012] Furthermore, the encoder consists of an initialization encoder and four downsampling modules. First, the input is channel-adjusted, then normalized using BatchNorm, and activated using the ReLU function.

[0013] Furthermore, the encoder's downsampling module consists of two residual connections and one downsampling layer, with each block in the residual connections containing two convolutional layers and one pooling layer.

[0014] Furthermore, the SE module in step S1 is added after the highest-dimensional feature output of the decoder.

[0015] Furthermore, the SE module in step S1 assigns weights to the channels of the network, capturing features by combining feature map information and channel information.

[0016] Furthermore, the SE module includes the following steps:

[0017] (1) Compression: Global average pooling is performed on the input mixed features to obtain compressed information;

[0018] (2) Incentive: Use the obtained compressed information to obtain channel dependencies, obtain the weights of each channel, and establish a correlation model between channels;

[0019] (3) Scaling: Multiply the weight value of each channel by the two-dimensional matrix of the corresponding channel of the original feature map, and output the result.

[0020] On the other hand, the present invention also provides the application of the above-mentioned multi-channel network based on hybrid lateral connections in the multimodal image feature extraction method.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0022] This invention provides a multi-channel network based on hybrid lateral connectivity. First, a multimodal encoder extracts multimodal image data, while an SE module adds channel attention. Then, a lateral connectivity method is used to fuse features of the same size from different modalities obtained after each downsampling. Finally, these hybrid features are concatenated with features obtained during the decoder's upsampling process, thereby maximizing the utilization of features from each modality. This network structure effectively solves the feature fusion problem of multi-encoders and can be applied to clinical multimodal imaging tasks. Experimental results also show that the proposed hybrid lateral connectivity network achieves the best results in TC and ET segmentation. After demonstrating its good performance in brain tumor segmentation, it can be extended to other image segmentation tasks. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments recorded in this invention, and those skilled in the art can obtain other drawings based on these drawings.

[0024] Figure 1 This is a diagram of a multi-channel network model based on hybrid lateral connections provided in an embodiment of the present invention.

[0025] Figure 2 This is a diagram of a hybrid lateral connection structure provided in an embodiment of the present invention.

[0026] Figure 3 This is a block diagram of the attention mechanism provided in an embodiment of the present invention. Detailed Implementation

[0027] Multimodal brain imaging data can provide a variety of information. However, existing multimodal medical image segmentation schemes mainly employ input-level fusion and decision-level fusion strategies, which are ineffective. Most input-level fusion strategies struggle to preserve the features of each modality. Decision-level fusion methods can better utilize the features of each modality, but they suffer from the problem that features learned from a single modality are not easily combined with features from other modalities. This invention aims to combine the above two methods to effectively extract fusion features from multimodal brain images.

[0028] To better understand this technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.

[0029] like Figure 1 As shown, the present invention provides a multi-channel network based on hybrid lateral connections, with a multimodal encoder as the main structure and a lateral connection structure in multimodal image decoding, and includes the following steps:

[0030] S1. Multimodal encoders are used to extract multimodal image data. At the same time, the SE module is used to increase channel attention at the decoder, which can effectively improve the accuracy of the model.

[0031] S2. Using a hybrid lateral connection method, the same-size features of different modalities obtained after each downsampling are fused to obtain hybrid features;

[0032] S3. The hybrid features from step S2 are concatenated with the features obtained during the decoder upsampling process and input into the decoder to maximize the use of features of each modality.

[0033] 1. About multimodal encoders

[0034] The encoder part uses a residual network as the base model and employs BatchNorm (BN) for normalization, which achieves better results than Group Normalization (GN) when the batch size is large enough (16 in our experiments). We follow common convolutional neural network (CNN) methods, progressively reducing the image size while increasing the feature size. For the reduction method, we use cross-convolution.

[0035] Each modal data is encoded using a separate encoder with the same structure, thus effectively extracting information from each modality. A total of four downsampling operations are performed, and then the results from multiple encoders are combined for decoding.

[0036] Each encoder consists of an initial encoder and four downsampling modules. First, the input undergoes channel adjustment; for example, each encoder takes a 240×240×batch-size image as input and uses a 3x3 convolution to convert the input image channels to 32 channels with a random dropout rate of 0.2. Next, normalization is performed using BatchNorm (BN), and activation is achieved using the ReLU function.

[0037] Each encoder's downsampling module consists of two residual connections and one downsampling layer. Each block in the residual connections contains two convolutional layers and one pooling layer (including Normalize and ReLU operations). For example, the downsampling layer uses a 3x3 convolutional kernel with a stride of 2 to downsample the input, halving the image size and doubling the number of channels. The model uses a total of four downsampling layers. The number of channels output by the initialization layer and the downsampling layer are 32, 64, 128, 256, and 512, respectively.

[0038] Finally, the high-level features of multiple modalities (four in this embodiment) are concatenated, and the channels are adjusted using two 3x3 residual connection modules. The first residual connection block reduces the number of input combined feature channels to 1024, and the second residual connection block reduces it to 512.

[0039] 2. Regarding hybrid lateral connections

[0040] When decoding deep features, we use hybrid lateral connections to fuse low-dimensional features from various modalities. This method can effectively extract feature information of different dimensions from multimodal data.

[0041] In this embodiment, the encoder uses a total of four downsampling operations, corresponding to information from four different dimensions. To effectively fuse the information, we simulate a U-Net structure. During upsampling, four lateral connections are used to merge the channels corresponding to the upsampling.

[0042] Hybrid lateral connection structure such as Figure 2As shown. Information from the same dimension across different modalities is concatenated and then adjusted using multiple channel adjustment modules. This embodiment uses four modalities, where C represents the number of channels for each modality in the current dimension. Let C be the number of channels for each mode, and the combined number of channels is 4C. Feature extraction and channel adjustment use two residual blocks each time, reducing C each time. Taking the first lateral connection as an example, the mixed features have a total of 128 channels, and the outputs after channel reduction at each layer are 96, 48, and 32, respectively.

[0043] 3. On attention mechanisms

[0044] This invention adds the SE module after the highest-dimensional feature output of the decoder. For example... Figure 3 As shown, this module captures features by combining feature map information and channel information. The SE module primarily focuses on channel information, assigning weights to the network's channels. In ordinary convolutional layers, the output of each channel is equal, but through weighting, we can strengthen channels with high information value and suppress channels with low information value.

[0045] The SE module includes the following steps:

[0046] (1) Compression: Global average pooling is performed on the input mixed features to obtain compressed information; after the compression operation is completed, the feature map will be compressed into a 1x1xC vector. The principle is shown in the following formula, where c represents the c-th channel of Z.

[0047]

[0048] Where H is the length of the image, W is the width of the image, u represents the input image patch, and F s For the compression operation, i and j refer to the traversal operation on the length and width, and Z is the output of the compression operation. As shown in the above formula, the c-th channel of Z is obtained by compressing the c-th channel of the input image block u, specifically by averaging all pixels.

[0049] (2) Stimulation: The obtained compressed information is used to obtain the channel dependency relationship. This operation obtains the weight of each channel through the parameter W, and uses W to establish the correlation model between each channel; the stimulation step uses a simple sigmoid activation function to activate.

[0050] s = F ex (z, W)=σ(g(z, W))=σ(W2δ(W1z)) (2)

[0051] Where s is the output of this operation, F exdenoted by , z represents the output of the previous compression step as input, W represents a random convolution kernel used to obtain gradients for the channels during backpropagation, δ represents the ReLU operation, σ represents the sigmoid function, and W1 and W2 represent the convolution kernel parameters corresponding to the two activation operations.

[0052] (3) Scaling: Multiply the weight value of each channel by the two-dimensional matrix of the corresponding channel of the original feature map, and output the result. For example, after obtaining a 1x1xC vector, scaling can be performed.

[0053]

[0054] in, F represents the c-th channel of the final output. scale This indicates the final channel multiplication operation, u c s represents the c-th channel of the original input. c This represents the c-th channel output from the excitation operation in the second step.

[0055] This invention also provides the application of the above-mentioned multi-channel network based on hybrid lateral connections in a multimodal image feature extraction method.

[0056] Input data:

[0057] This invention uses the publicly available multimodal brain tumor dataset BraTS (brain tumorsegmentation challenge) as experimental input data. The dataset includes native pre-contrast maps (T1), post-contrast T1-weighted maps (T1Gd), T2-weighted maps (T2), and T2-fluid attenuation inversion recovery maps (T2-flair). Each patient's 3D image contains 155 images, each 240x240 pixels. The dataset has three labels: whole tumor (WT), tumor core region (TC), and enhanced tumor region (ET). Our experiments use 285 cases from BraTS2018 as the training set and 40 new cases from BraTS2019 as the validation set to validate the performance of our model. Before the experiment, all data were normalized and shuffled after being cut into 2D slices.

[0058] Experimental results:

[0059] This invention uses the Dice Similarity Coefficient (DICE) and Hausdorff distance (HD) (HD95 is selected in this embodiment) as the evaluation criteria, and compares the results with HNF-Net, NVDLMED, and No New-Net (3D) models. The results are shown in the table below:

[0060]

[0061] Experimental results show that our proposed hybrid lateral connectivity network achieves the best performance in segmenting TC and ET. Having demonstrated its excellent performance in brain tumor segmentation, we will extend it to other image segmentation tasks in future work.

[0062] In summary, this invention proposes a network for multimodal image feature extraction. This network uses a multi-encoder architecture as its core, inspired by the U-Net structure, and replaces the long-connection structure in U-Net with lateral connections in multimodal image decoding. This structure effectively solves the feature fusion problem of multi-encoders and can be applied to clinical multimodal imaging tasks.

[0063] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A hybrid side-connection based multi-channel network, characterized in that, A multi-modal encoder is used as the main structure, and a lateral connection structure is used in multi-modal image decoding, and the method comprises the following steps: S1, using a multi-modal encoder to extract multi-modal image data, and using a compression excitation module to increase channel attention; in step S1, each modality data is separately encoded using an encoder, four times of down-sampling are performed, and then the results of the multiple encoders are combined together for decoding; the encoder is composed of an initialization encoder and four down-sampling modules, first, the input is channel adjusted, then normalized using BatchNorm, and activated using a ReLU function; the down-sampling module of the encoder is composed of two residual connections and a down-sampling layer, each block in the residual connection contains two convolutional layers and a pooling layer; the compression excitation module of step S1 is added after the highest dimensional feature output of the decoder; S2, using a hybrid lateral connection method to fuse the same size features of different modalities obtained after each down-sampling to obtain mixed features; S3, the mixed features of step S2 are spliced with the features obtained in the up-sampling process of the decoder and input into the decoder.

2. The multi-channel network based on hybrid lateral connection according to claim 1, wherein, The compression excitation module of step S1 allocates weights to the channels of the network, and captures features by combining feature map information and channel information.

3. A multi-channel network based on hybrid lateral connection according to claim 2, characterized in that, The compression excitation module comprises the following steps: (1) compression: performing global average pooling on the input mixed features to obtain compressed information; (2) excitation: using the obtained compression information to obtain channel dependency, obtaining the weight of each channel, and establishing a correlation model between channels; (3) scaling: multiplying the weight value of each channel by the two-dimensional matrix of the corresponding channel of the original feature map, and outputting the obtained result.

4. Application of the multi-channel network based on hybrid lateral connection according to any one of claims 1-3 in a multi-modal image feature extraction method.

Images