Soft tissue photoacoustic / ultrasonic multi-modal image fusion method based on deep learning

Abstract

Disclosed in the present invention is a soft tissue photoacoustic / ultrasonic multi-modal image fusion method based on deep learning. The method comprises the following steps: an ultrasonic-photoacoustic imaging device collecting a human body soft tissue photoacoustic image and ultrasonic image, and performing size normalization processing; inputting the images into a spatial conversion module to convert same into a YCbCr space; inputting the images into a pre-convolutional module to change the number of data channels; inputting the images into a multi-scale feature extraction module to extract salient features of the source images; inputting the features into a filter prediction module to obtain a multi-scale filter; and inputting the filter into a filtering fusion and adaptive enhancement module and combining same with the input source images, so as to obtain a final fusion result. Compared with several traditional fusion methods and deep-learning-based fusion methods, the method provided in the present invention has a greater fusion effect, and more importantly, has an excellent real-time performance. Moreover, multiple modes of photoacoustic / ultrasonic fusion extension experiments are performed on a photoacoustic / ultrasonic multi-modal imaging system, verifying the effectiveness of the method in the present invention.

Description

Soft tissue photoacoustic / ultrasound multimodal image fusion method based on deep learning TECHNICAL FIELD

[0001] The present application relates to a multi-modal medical image fusion method, in particular to an unsupervised human soft tissue photoacoustic / ultrasound multimodal image fusion method based on multi-channel filtering and adaptive enhancement. BACKGROUND

[0002] In recent years, various medical imaging techniques have been widely used in the diagnosis process of diseases. Generally speaking, it is difficult to obtain comprehensive diagnostic information by relying on a single imaging modality, which is crucial to ensure the accuracy and comprehensiveness of diagnosis. Therefore, the method of multi-modal medical image fusion combines data from different imaging modalities to form a comprehensive image with rich information, providing a solid basis for clinical diagnosis. Multi-modal medical image fusion technology combines images from various sources such as X-ray, computed tomography (CT), single photon emission computed tomography (SPECT), ultrasound (US), magnetic resonance imaging (MRI), infrared, ultraviolet, and positron emission tomography (PET).

[0003] MRI, X-ray, CT, and US imaging techniques can reveal the location, size, shape of lesions, and their impact on surrounding tissue structures. In order to further explore the biological characteristics of tumors, soft tissue state and functional information, the application of positron emission computed tomography, functional magnetic resonance imaging and single photon emission computed tomography is becoming more and more common. By combining functional and structural data, medical image fusion can produce more valuable diagnostic information. In the process of treating specific human organs, medical image fusion plays a key role, which can achieve more accurate monitoring and analysis of diseases.

[0004] Among them, ultrasonic imaging and photoacoustic imaging technology has been widely used in the medical field, especially in the imaging of human soft tissue. These two imaging techniques have their unique advantages, and their combination can provide more comprehensive and in-depth diagnostic information, which is of great significance for clinical diagnosis and treatment. Ultrasonic imaging uses high-frequency sound waves to detect internal structures of the body, and the echoes produced by sound waves at different tissue interfaces are converted into images, which can display the dynamic changes inside the body in real time. The advantage of this technology is that it has no radiation risk, real-time imaging capability, portability and cost-effectiveness, especially suitable for the observation of soft tissue structure. However, it has certain limitations in resolution and depth, especially in the imaging of bone and gas tissue. Photoacoustic imaging combines the advantages of optics and acoustics, and through laser pulse-induced thermal expansion of tissue to produce ultrasonic waves, thus realizing imaging. This technology is known for its high contrast and deep tissue imaging capability, especially suitable for imaging of blood vessels and tissues rich in hemoglobin. Although photoacoustic imaging has certain limitations in imaging depth and equipment requirements, its advantage in resolution is significant. The fusion of ultrasonic imaging and photoacoustic imaging can achieve complementation, improving the accuracy and information richness of human soft tissue imaging. This fusion technology combines the real-time monitoring capability of ultrasonic imaging and the high contrast and high resolution characteristics of photoacoustic imaging, making the observation of soft tissue and surrounding blood vessels more detailed and in-depth.

[0005] The difficulty of ultrasound and photoacoustic image fusion is how to retain as much characteristic information of different modal images as possible. The main fusion methods currently include traditional methods and deep learning-based methods. The fusion process of the traditional method can be summarized as follows: first, the input image is decomposed in the frequency domain or the spatial domain. Then, special fusion rules are designed according to the components obtained by decomposition. Finally, the multi-modal medical image after fusion is obtained by image reconstruction according to the previous decomposition method using the fused component information. Therefore, the traditional fusion method is limited by the complexity of image decomposition, usually requiring a large amount of calculation, and the fusion efficiency and fusion effect are easily affected by the artificially designed fusion rules. In recent years, deep learning has been increasingly widely applied in the field of computer vision, and people have also carried out in-depth research on the application of deep learning in image fusion. However, for multi-modal medical image fusion, due to the lack of standard fusion results and a large number of safe and reliable medical images, supervised deep learning methods are not easy to implement, so it is particularly important to balance the generalization ability and accuracy of the model under unsupervised training. The current fusion methods based on unsupervised deep learning mainly include two categories, namely the fusion method based on weight map and the fusion method based on deep representation. Among them, the fusion method based on weight map first obtains the corresponding weight map of the source image through the same neural network, and then performs weighted fusion according to the weight map and the source image to obtain the fused image; the fusion method based on deep representation first processes different source images through the corresponding neural network, obtains the fused feature map according to the fusion rule, and then further processes the feature map by using a deep neural network to generate the final fusion image. However, the fusion method based on weight map lacks consideration of local information and spatial continuity of the input image. The fusion method based on deep representation is difficult to balance the relationship between network depth and fusion effect. How to balance the high quality of the fusion result and the lightweight of the network structure is an important challenge faced by the image fusion task. In addition, due to the lack of real labels in the image fusion task, it is very valuable to develop a loss function that comprehensively considers the image pixel intensity, feature information, structure information and correlation to ensure the preservation of fusion information in the unsupervised fusion network training process.

[0006] SUMMARY

[0007] In order to solve the problems of poor fusion effect, high time consumption and lack of a loss function that can comprehensively evaluate the fusion effect in the human soft tissue photoacoustic / ultrasound multi-modal image fusion algorithm, the present application provides a soft tissue photoacoustic / ultrasound multi-modal image fusion method based on deep learning.

[0008] The purpose of the present application is achieved by the following technical solutions:

[0009] A soft tissue photoacoustic / ultrasound multi-modal image fusion method based on deep learning comprises the following steps:

[0010] Step (1) obtains the ultrasound and photoacoustic source images of human soft tissue from an ultrasonic photoacoustic multimodal imaging device, and pre-processes the source images through size normalization operation;

[0011] Step (2) the pre-processed source images are converted from RGB space to YCbCr space through a channel space conversion module, and the data of the three channels are further pre-convoluted through a pre-convolution module, which recombines the data of each channel on the channel;

[0012] Step (3) the image processed by the pre-convolution module is input into a multi-scale feature extraction module, and the encoding stage of the multi-scale feature extraction module contains two encoding layers, and the first layer encoding operation is as follows:

[0013] x 11 =ResBlock(x n )

[0014] x down1 =Skip Conv (Concat(x n ,HybridAttention(Down(x 11 ))),HybridAttention(Down(x 11 )))

[0015] Wherein x n represents the image processed by the pre-convolution module, ResBlock(·) is a residual operation, HybridAttention(·) represents a hybrid attention operation, Skip Conv (·) represents a convolution-based Skip operation, Down(·) represents a down-sampling operation, and Concat(·) represents a connection operation;

[0016] The second layer encoding operation is as follows:

[0017] x 12 =ResBlock(x down1 )

[0018] x' down2 =Skip Conv (Concat(x down1 ,HybridAttention(Down(x 12 ))),HybridAttention(Down(x 12 )))

[0019] x down2 =Skip Conv(Concat(x n ,x' down2 ),x' down2 )

[0020] x bottom =ResBlock(x down2 )

[0021] where x bottom is the bottom output of the encoding stage;

[0022] The bottom output outputs features of different scales through two decoding operations, the specific process is as follows:

[0023] x up1 =Skip Conv (Concat(x bottom ,HybridAttention(Up(x bottom ))),HybridAttention(Up(x bottom )))

[0024] x 21 =Skip Res (Concat(x 12 ,ResBlock(x up1 )),ResBlock(x up1 ))

[0025] x up2 =Skip Conv (Concat(x 21 ,HybridAttention(Up(x 21 ))),HybridAttention(Up(x 21 )))

[0026] x' 22 =Skip Conv (Concat(x bottom ,ResBlock(x up2 )),ResBlock(x up2 ))

[0027] x 22 =Skip Res (Concat(x 11 ,x' 22 ),x' 22 )

[0028] where x 21 and x 22 are two different scale feature outputs of the decoding process, SkipRes (·) indicates the Skip operation of applying residual, and Up(·) indicates the upsampling operation;

[0029] The source image is processed by the multi-scale feature extraction module to obtain three features, from large to small: and

[0030] Step (4) The features of three different scales corresponding to the Y, Cb and Cr channels are input into the filter prediction module in pairs. The filter prediction module uses spatial cross attention to dynamically process two input feature maps of the same scale at the same time. and F1 m , and assign weights according to the importance of each position, and then output the corresponding spatial attention weighted feature map. The specific operation is shown in the following formula:

[0031] A=[A0,A1]=Sigmoid(Conv(ReLU(Conv(Concat(F0 m ,F1 m )))))

[0032] Where A is the attention weight, A0 and A1 are and F1 m The corresponding weight components, and F1 m ' is the input feature map and F1 m The corresponding spatial attention weighted feature map, m = 0, 1, 2 are the serial numbers of different scales, ⊙ represents the Hadamard product, Sigmoid(·) represents the Sigmoid activation function, and ReLU(·) represents the ReLU activation function;

[0033] Spatial attention weighted feature map and F1m' and the corresponding source image are input into the kernel prediction network based on the residual structure respectively. The network is learned by and F1 m 'dynamic changes, predict the most effective filter and Filter1 m , the specific operation of filter prediction is shown as follows:

[0034] Among them, Unfold(·) means converting the source image into a column vector, fold(·) means reshaping the feature map to its original size, and sum(·) means the summation operation. yes Corresponding prediction convolution kernel weights; add two filters to obtain the filter Filter of the current scale m ∈2xCxWxH 2 As shown in the following formula:

[0035] Step (5) inputs the Y, Cb and Cr channel data of the two source images and the filters of three different kernel sizes into the filter fusion and adaptive enhancement module for convolution operation, and obtains the fused Y, Cb and Cr channel data by weighted sum of the obtained convolution results, as shown in the following formula:

[0036] Wherein I Y-fuse , I Cb-fuse and I Cr-fuse are the fusion results of the Y channel, Cb channel and Cr channel of the source image respectively, I Cb-0 , I Cb-1 , I Cr-0 , I Cr-1 , I Y-0 and I Y-1 respectively represent the YCbCr channel information of the input source image, and alpha is a training parameter in the network, Indicates a convolution operation;

[0037] Step (6) realizes adaptive enhancement of the fused image by adjusting the brightness factor and contrast factor of the Y channel and the saturation factor of the Cb and Cr channels in the training process, outputs the enhanced Y, Cb and Cr channel data, and reconstructs the fusion result to realize unsupervised human soft tissue photoacoustic / ultrasound multi-modal image fusion.

[0038] Compared with the prior art, the present application has the following advantages:

[0039] Quantitative and qualitative evaluation on open source CT-MRI images, MRI-PET images, MRI-SPECT images and photoacoustic-ultrasound images shows that the method proposed in the present application has more excellent fusion effect compared with several traditional fusion methods and depth learning-based fusion methods, and more importantly, has good real-time performance. And a variety of mode photoacoustic / ultrasound fusion expansion experiments are carried out on the photoacoustic / ultrasound multi-modal imaging system, which verifies the effectiveness of the method of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0040] Fig. 1 is a flowchart of an unsupervised human soft tissue photoacoustic / ultrasound multi-modal image fusion method based on multi-channel filtering and adaptive enhancement;

[0041] Fig. 2 is an unsupervised human soft tissue photoacoustic / ultrasound multimodal image fusion framework based on multi-channel filtering and adaptive enhancement;

[0042] Fig. 3 is a multi-scale feature extraction module framework;

[0043] Fig. 4 is a hybrid attention structure framework;

[0044] Fig. 5 is a filter prediction module framework;

[0045] Fig. 6 is a filter fusion and adaptive enhancement module framework;

[0046] Fig. 7 is the effect of human soft tissue photoacoustic ultrasound fusion imaging and the corresponding histogram. DETAILED DESCRIPTION

[0047] The technical solutions of the present application will be further described below in conjunction with the drawings, but are not limited thereto, and any modification or equivalent replacement of the technical solutions of the present application without departing from the spirit and scope of the present application shall be covered in the protection scope of the present application.

[0048] The present application provides a kind of human soft tissue photoacoustic / ultrasound multimodal image fusion method based on deep learning, as shown in figure 1, including the following steps: ultrasonic photoacoustic imaging equipment acquires human soft tissue photoacoustic image and ultrasound image, and carries out size normalization processing;Input space conversion module is converted to YCbCr space;Input pre-convolution module changes data channel number;Input multi-scale feature extraction module extracts source image significant feature;Input filter prediction module obtains multi-scale filter;Input filter fusion and adaptive enhancement module combines input source image to obtain the final fusion result.As shown in figure 2, the specific steps are as follows:

[0049] Step (1) obtains the ultrasound and photoacoustic source images of human soft tissue from ultrasonic photoacoustic multimodal imaging equipment, and pre-processes the source images through size normalization operation.

[0050] Step (2) the source image after pre-processing is converted from RGB space to YCbCr space by channel space conversion module, and the data of three channels is further processed by pre-convolution module, and the pre-convolution module recombines the data of each channel on the channel.

[0051] Step (3) inputs the image processed by the pre-convolution module into the multi-scale feature extraction module, which is designed to extract significant features representing the source image. It mainly consists of residual blocks, convolution blocks and hybrid attention blocks, as shown in FIG. 3. Compared with UNet, the module adds hybrid attention mechanism and residual structure, and adds learnable parameters in the basic skip operation to adaptively balance the encoding features and decoding features. These optimizations effectively enhance the network's ability to process and extract different scale detail features without increasing complexity. Considering the balance between model performance and complexity, the encoding stage of the multi-scale feature extraction module proposed in the application contains two encoding layers. The first layer encoding operation is shown in formula (1):

[0052] x 11 =ResBlock(x n )

[0053] x down1 =Skip Conv (Concat(x n ,HybridAttention(Down(x 11 ))),HybridAttention(Down(x 11 ))) (1)

[0054] where ResBlock(·) is a residual operation including two convolution operations, instance normalization, LeakyReLU function activation and residual connection, HybridAttention(·) represents hybrid attention operation, and the hybrid attention used is a convolution block attention module, which includes channel attention and spatial attention. The role of channel attention is to build the correlation between different channels, and automatically assign different weights according to the importance of the channel through network learning, which plays a role in strengthening important features and suppressing non-important features. The role of spatial attention is to enhance the feature expression of key regions, which converts the spatial information in the feature map to another space, generates a mask with weights according to the importance of different positions, and plays a role in enhancing the region of interest and weakening the irrelevant region. The structure of hybrid attention is shown in FIG. 4, and the input feature map is first subjected to channel attention and then spatial attention. Skip Conv (·) represents a skip operation applying convolution. Down(·) represents a down-sampling operation. The bottom output after the second layer encoding operation is shown in formula (2):

[0055] where x bottom is the bottom output of the encoding stage. Then, the bottom output needs to be output through two decoding operations to output features of different scales, as shown in formula (3):

[0056] where x 21 and x 22 are the feature outputs of two different scales of the decoding process. Skip Res (·) denotes the Skip operation with the application of the residual. Up(·) denotes the up-sampling operation. Finally, the three features obtained after the source images are processed by the multi-scale feature extraction module are and

[0057] Step (4) refers to the principle of the kernel prediction network. The three different scale features corresponding to the Y, Cb, and Cr channels are input into the filter prediction module (Figure 5) in pairs to predict the filters of different kernel sizes corresponding to each channel through attention mechanisms. The filter prediction module is the key part of the entire fusion network. The main task of this module is to use the three different scale features obtained by the multi-scale feature extraction module to predict the filters used to guide image fusion. Since multi-modal medical image fusion is a process that needs to consider the complementary learning of different source images at the same time, the network has to perform cross-image operations during operation. The filter prediction module first uses spatial cross-attention to dynamically process two input feature maps of the same scale and F1 m at the same time, and assigns weights according to the importance of each position, and then outputs the corresponding spatial attention weighted feature map, as shown in equation (4):

[0058] where A is the attention weight, A0and A1are and F1 m corresponding weight components. and F1 m ' are the input feature maps and F1 m corresponding spatial attention weighted feature maps. m = 0, 1, 2 are the serial numbers of different scales. Secondly, the spatial attention weighted feature maps and F1 m ' and the corresponding source images are input into the kernel prediction network based on the residual structure. Through the dynamic changes of and F1 m ', the network predicts the most effective filters and Filter1 mTo reduce the complexity of the model and improve the performance of the model, instead of using a deep convolutional neural network to extract rich image features, a kernel prediction network based on residual structure is used. On the one hand, the residual structure allows the reuse of features between different layers, improving the utilization rate of the network while also enhancing the generalization ability of the features. On the other hand, for pixel-level operations, the residual structure can maintain the spatial resolution of the input features, preserving some of the details of the upper layer features. The specific operation of filter prediction is shown in formula (5):

[0059] Wherein, Unfold(·) represents converting the source image into a column vector, fold(·) represents restoring the feature map to the original size, sum(·) represents the summation operation, The corresponding predicted convolution kernel weight is m ∈2×C×W×k 2 As shown in formula (6):

[0060] The predicted filter combined with the spatial cross-attention can adaptively process each position of the source image.

[0061] Step (5) inputs the Y, Cb and Cr channel data of the two source images and the filters of three different kernel sizes into the filter fusion and adaptive enhancement module (Figure 6) for convolution operation, and obtains the fused Y, Cb and Cr channel data by weighted summation of the convolution results, as shown in formula (7):

[0062] Wherein I Y-fuse , I Cb-fuse and I Cr-fuse are the fusion results of the source images in the Y channel, the Cb channel and the Cr channel, and α is a training parameter in the network.

[0063] Step (6) realizes adaptive enhancement of the fused image by adjusting the brightness factor and the contrast factor of the brightness (Y) channel and the saturation factor of the color (Cb, Cr) channel during the training process, outputs the enhanced Y, Cb and Cr channel data, and reconstructs the fusion result, realizing unsupervised human soft tissue photoacoustic / ultrasound multimodal image fusion.

[0064] Embodiment;

[0065] ​The embodiment carries out real-time fusion imaging experiment on a photoacoustic-ultrasound imaging system. The ultrasound and photoacoustic multimodal images of human soft tissue output by the imaging system are directly connected to the source image input interface of the fusion method, and through experimental test, the multimodal fusion imaging speed of the whole system is 8 FPS. The fusion imaging result and the corresponding gray histogram are shown in FIG. 7, and the quantitative indexes of the mainstream method are shown in Table 1. The fusion result can clearly restore the details and colors of the source images, and through the gray histogram, it can be seen that the method proposed in the application can also reduce the noise of the photoacoustic image and the ultrasound image to a certain extent.

[0066] Table 1 quantitative indexes of different fusion methods

Claims

1.A deep learning based soft tissue photoacoustic / ultrasound multimodal image fusion method, characterized in that The method comprises the following steps: Step (1) obtaining the ultrasound and photoacoustic source images of human soft tissue from an ultrasonic photoacoustic multimodal imaging device, and pre-processing the source images through size normalization operation; Step (2) the pre-processed source images are converted from RGB space to YCbCr space through a channel space conversion module, and the data of the three channels are further pre-convoluted through a pre-convolution module, which recombines the data of each channel on the channel; Step (3) inputting the images processed by the pre-convolution module into a multi-scale feature extraction module, the encoding stage of the multi-scale feature extraction module comprising two encoding layers, the first layer encoding operation being as shown in the following formula: x 11 = ResBlock(x n ) x down1 = Skip Conv (Concat(x n ,HybridAttention(Down(x 11 ))),HybridAttention(Down(x 11 ))) where x n denotes the image processed by the pre-convolution module, ResBlock(·) is a residual operation, HybridAttention(·) denotes a hybrid attention operation, Skip Conv (·) denotes a Skip operation applying convolution, Down(·) denotes a down-sampling operation, and Concat(·) denotes a concatenation operation; The second layer encoding operation is as shown in the following formula: x 12 = ResBlock(x down1 ) x' down2 = Skip Conv (Concat(x down1 ,HybridAttention(Down(x 12 ))),HybridAttention(Down(x 12 ))) x down2 = Skip Conv (Concat(x n ,x' down2 ),x' down2 ) x bottom = ResBlock(x down2 ) where x bottom is the bottom output of the encoding stage; The bottom layer output outputs features of different scales through two decoding operations, and the specific process is as shown in the following formula: x up1 = Skip Conv (Concat(x bottom ,HybridAttention(Up(x bottom ))),HybridAttention(Up(x bottom ))) x 21 = Skip Res (Concat(x 12 ,ResBlock(x up1 )),ResBlock(x up1 )) x up2 = Skip Conv (Concat(x 21 ,HybridAttention(Up(x 21 ))),HybridAttention(Up(x 21 ))) x' 22 = Skip Conv (Concat(x bottom ,ResBlock(x up2 )),ResBlock(x up2 )) x 22 = Skip Res (Concat(x 11 ,x' 22 ),x' 22 ) where x 21 and x 22 are the feature outputs of two different scales of the decoding process, Skip Res (·) denotes the Skip operation with the residual applied, and Up(·) denotes the up-sampling operation; The source image is processed by the multi-scale feature extraction module to obtain three features, which are in descending order and Step (4) combines the three different scale features corresponding to the Y, Cb and Cr channels two by two into the filter prediction module, and the filter prediction module dynamically processes two input feature maps of the same scale at the same time by using spatial cross attention and F1 m and according to the importance of each position, a weight is assigned, and then the corresponding spatial attention weighted feature map is output, and the specific operation is as shown in the following formula: A = [A0, A1] = Sigmoid(Conv(ReLU(Conv(Concat(F0 m ,F1 m )) )) where A is the attention weight, A0and A1are and F1 m corresponding weight components, and F1 m 'for input feature map and F1 m corresponding spatial attention weighted feature maps, m = 0, 1, 2 are the serial numbers of different scales, represents Hadamard product, Sigmoid(·) represents Sigmoid activation function, and ReLU(·) represents ReLU activation function; Spatial attention weighted feature maps and F1 m and the corresponding source images are input into the kernel prediction network based on the residual structure respectively, and the network learns through and Fl m the dynamic changes of the signal-to-noise ratio, the most effective filter is predicted and Filter 1 m The specific operation of the filter prediction is shown in the following equation: wherein Unfold( ) represents converting the source image into a column vector, fold( ) represents reshaping the feature map to the original size, sum( ) represents a summation operation, is corresponding predicted convolution kernel weights; adding the two filters to obtain a filter Filter m ∈2×C×W×k 2 as follows: Step (5) inputs the Y, Cb and Cr channel data of the two source images and three different kernel sizes of filters into the filter fusion and adaptive enhancement module for convolution operation, and obtains the fused Y, Cb and Cr channel data by weighted sum of the obtained convolution results, as shown in the following formula: where I Y-fuse , I Cb-fuse , and I Cr-fuse are the fusion results of the source images in Y channel, Cb channel, and Cr channel, respectively, I Cb-0 , I Cb-1 , I Cr-0 , I Cr-1 , I Y-0 , and I Y-1 represent the information of YCbCr channels of the input source images, respectively, and a is a training parameter in the network. represents convolution operation; Step (6) realizing adaptive enhancement of the fused image by adjusting the brightness factor and contrast factor of the Y channel and the saturation factor of the Cb and Cr channels during the training process, outputting the data of the enhanced Y, Cb and Cr channels, and reconstructing to generate a fusion result, thereby realizing unsupervised human soft tissue photoacoustic / ultrasound multimodal image fusion.

Images