Feature fusion method of multi-modal deep neural network

Abstract

The invention discloses a feature fusion method for a multi-modal deep neural network, and the method comprises the steps of obtaining a channel attention mask between modals in a multi-modal deep three-dimensional CNN through employing a squeeze and excitation (SE) module on a deep learning feature domain, i.e., in all modals, giving more attention to the channels which are remarkably helpful tothe task target, so that the weight distribution of the multi-modal three-dimensional depth feature map on the channels is established explicitly; and then, calculating by utilizing four-dimensional convolution and a Sigmoid activation function to obtain a spatial attention mask between modals, namely, in the three-dimensional feature map of each modal, which positions in the space need to be moreconcerned, so that the spatial correlation of the multi-modal three-dimensional depth feature map is established explicitly, and more attention is given to the positions with important information inmodes, channels and spaces, so that the diagnosis efficiency of the multi-mode intelligent diagnosis system is improved.

Description

technical field [0001] The invention relates to the fields of medical imaging and deep learning, in particular to a feature fusion method of a multimodal deep neural network. Background technique [0002] Existing tumor detection and diagnosis methods are usually realized through medical imaging technology, including planar X-ray imaging, CT, MRI, PET / CT, ultrasound and other modalities, and tissue biopsy is performed on suspicious lesions found in the images. However, due to the heterogeneity of tumors, their properties cannot be fully characterized by a single modality image. For example, on plane X-ray and CT images, the X-ray absorption degree of tumor tissue is characterized; on MRI images, the hydrogen proton density of tumor tissue is characterized; on FDG PET / CT, the It is the activity of tumor tissue to metabolize glucose; on ultrasound images, it is characterized by the reflection degree of tumor tissue to sound waves. Therefore, more and more clinical research b...

AI Technical Summary

Problems solved by technology

However, these methods did not analyze and process the feature weight distribution of each mode, but simply added, superimposed or fused directly

Images