Multi-modal image synthesis method and device based on recursive fusion conversion unit

Abstract

The application discloses a kind of multi-modal image synthesis method and device based on recursive fusion conversion unit, method includes: fusion conversion unit uses the reasoning form of recursive development, extracts different depth local and global features;Multi-branch decoder utilizes the local feature of each branch intermediate depth to synthesize missing modal image, calculates loss to guide optimization;Utilize the local and global features of last stage to synthesize multiple missing modal reconstruction images;Mixed gate module dynamically fuses multiple missing modal reconstruction images in fusion image space;Fusion conversion unit, multi-branch decoder and mixed gate module are connected to form generator, generator and discriminator constitute generative adversarial network framework;Respectively, the image reconstructed using different branch, different depth feature is calculated to generate adversarial loss and reconstruction loss, and the network is trained by back propagation.The device includes: processor and memory.The application considers the local and global context information of multi-modal image data, and improves the image synthesis quality of missing modal.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image synthesis, and in particular to a multi-modal image synthesis method and device based on recursive fusion transformation unit. BACKGROUND

[0002] In real-world scenarios, people often need as many image modalities as possible to accurately and completely describe the scene in reality. However, due to various factors such as cost, time and safety, it is often difficult to obtain all modalities. A fundamental solution is to synthesize the required modalities through available multi-modal data, to fuse multi-modal complementary information into a single image, to improve image quality, and to make it more useful in subsequent applications such as image analysis, segmentation, target detection and recognition.

[0003] People have made various efforts in multi-modal image synthesis. Traditional methods usually construct a non-linear mapping from source images to target images. However, the performance of these methods is limited by the manually extracted or selected features. With the rapid development of deep learning, many studies have introduced convolutional neural networks (CNN) into the image synthesis task. Through carefully designed loss functions and network structures, convolutional neural networks have made significant progress compared to traditional methods. Later, generative adversarial networks (GAN) were proposed, which can be used for implicit image synthesis. GAN relies on the adversarial game between the generator and the discriminator to estimate the probability distribution of the target, thereby making progress in synthesis performance.

[0004] Despite the significant performance improvement, current synthesis techniques based on CNNs all have inherent limitations: (i) Convolutional operations mainly focus on learning the relationship between pixels within a relatively small neighborhood, which makes it good at extracting local features, but not good at capturing global concepts. In order to ensure that the features have sufficient spatial resolution for accurate reconstruction, the network depth of the image synthesis model is relatively limited. This greatly limits the ability of the image synthesis model to extract context information that reflects global spatial dependencies. (ii) In the task of multi-modal image synthesis, not only the global spatial context within the modalities needs to be captured, but also the global complementary information between the modalities needs to be summarized. However, these CNN-based methods are difficult to capture the cross-modal dependencies of features between modalities, resulting in weak performance in the fusion of global complementary information between all modalities.

[0005] Vision Transformer (ViT) has advantages over CNN in global representation learning thanks to the global self-attention mechanism. By capturing global context within and between modalities, ViT can provide promising synthesis results. However, this also brings new challenges. First, ViT will segment the input image into different patches, encode them into one-dimensional sequences with position encoding, and calculate the output of each position by weighted summation of all patches. This may cause the deterioration of local feature details. Since the multi-modal image synthesis task requires global context information while preserving local details, it is urgent to explore a multi-modal image synthesis method that can balance local and global features, fully utilize intra-modal and inter-modal information, and improve synthesis quality. SUMMARY

[0006] The present application utilizes the sensitivity of convolutional neural networks to local image features and the learning ability of Vision Transformer to long-distance dependencies to design a multi-modal image synthesis method and device based on recursive fusion conversion units, so that the model takes into account the local and global context information of multi-modal image data, and improves the image synthesis quality of the missing modal, as described in detail below:

[0007] In a first aspect, a multi-modal image synthesis method based on recursive fusion conversion units is provided, the method comprising:

[0008] The fusion conversion unit learns local image features through a local representation block, filters effective features from the local image features through a feature conversion gate, converts the effective features to a Transformer feature space, and fuses the effective features with old global features through a global fusion block to infer new global features. The fusion conversion unit adopts a recursive unfolding inference form to extract local and global features at different depths.

[0009] The multi-branch decoder synthesizes the missing modal image using the local features at the intermediate depth of each branch, calculates the loss to guide optimization, and synthesizes multiple missing modal reconstruction images using the local and global features at the last stage. The hybrid gating module dynamically fuses multiple missing modal reconstruction images in the fusion image space.

[0010] The fusion conversion unit, multi-branch decoder, and hybrid gating module are connected in series to form a generator, and the generator and discriminator form a generative adversarial network framework.

[0011] The images reconstructed using features at different branches and different depths are used to calculate the adversarial loss and reconstruction loss, respectively, and the network is trained through backpropagation.

[0012] The local representation block comprises:

[0013] LRB = Conv1(ReLU(ResBlock(x)))

[0014] LRB outputs the local feature map C i-1 The local feature is sent into a 1*1 convolution layer and a gated linear unit, and the effective feature is converted into a transformer feature space.

[0015] The recursive expansion is that the output of the previous recursive stage is taken as the input of the next recursive stage, and multiple inferences are performed in the same module to train the module parameters.

[0016] The multi-branch decoder is:

[0017] The missing modality image is reconstructed by using the input modality multi-branch feature provided by the fusion conversion unit, the missing image is reconstructed by using the global feature of the fusion branch, and the intermediate result is reconstructed by using the local feature in the intermediate recursive inference stage.

[0018] The mixed gating module is a multi-branch synthetic image that calculates adaptive fusion weights, and the final missing modality fusion result is obtained by weighting.

[0019] The second aspect is a multi-modal image synthesis device based on a recursive fusion conversion unit, the device comprises a processor and a memory, the memory stores program instructions, and the processor calls the program instructions stored in the memory to make the device execute the method in any one of the first aspect.

[0020] The third aspect is a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program comprises program instructions, and the program instructions are executed by the processor to make the processor execute the method in any one of the first aspect.

[0021] The beneficial effects of the technical scheme provided by the application are:

[0022] 1、The application utilizes the sensitivity of the convolutional neural network to local image features and the capturing ability of ViT to long-distance dependence, and designs a multi-modal image synthesis method based on a recursive fusion conversion unit, so that the model realizes dynamic compatibility of local and global information in the training process, obtains local image features, and considers the long-distance dependence relationship of local features within and between modalities, finally obtains leading results in the missing modality synthesis task, and realizes a high-quality multi-modal image synthesis model.

[0023] 2. The application designs a feature conversion method, which selects effective features in the past recursive process adaptively through a gated linear unit in training, and automatically converts the effective features into a Transformer feature space, so as to add the old global features of the past recursive process, and further extract dependency relationships.

[0024] 3. The application reduces the parameter quantity of the whole model by recursive unfolding of the fusion conversion unit, reduces the dependence on the training data quantity, and guarantees the accuracy of deep features of the network by adding constraints to each recursive stage, so as to strengthen the quality of the synthetic image under the premise of a small amount of parameters.

[0025] 4. The application designs a method for synthesizing missing modality images based on a hybrid gating module, constructs a hybrid gating module for a specific modality branch and a multi-modality fusion branch, adaptively calculates the weight map of each branch through convolution operation and element-level Softmax normalization, fully utilizes the feature information of the multi-modality branch, flexibly mixes multiple synthesis results in the image space, and obtains the final missing modality synthesis result. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 A flowchart of a multi-modality image synthesis method based on a recursive fusion conversion unit is provided for the application.

[0027] Figure 2 A schematic diagram of the fusion conversion unit module is provided for the application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical scheme and advantages of the application clearer, the embodiments of the application are further described in detail below.

[0029] In order to solve the technical problems existing in the multi-modal image synthesis method, the embodiment of the application provides a multi-modal image synthesis method based on a recursive fusion conversion unit. The method preliminarily extracts local features of an input image through a multi-branch encoder. The method combines the respective advantages of a convolutional neural network and a Transformer network by assembling a fusion conversion unit, and fully extracts local feature information of the image and long-distance dependency between modes. The method designs a recursive unfolding manner to constrain the synthesis result of each recursive stage, thereby greatly reducing the model parameters and improving the model performance without reducing the model performance. The method synthesizes the reconstruction results of multiple branches and multiple recursive stages through a multi-branch decoder to guide the parameter updating direction of the recursive process. The method designs a hybrid gating module to adaptively mix the synthesized images of specific modes and fusion modes in the image space, thereby fully utilizing different information of multi-modal images and improving the synthesis quality of the missing mode. The method connects the fusion conversion unit and the hybrid gating module to form a generator, and balances the best performance in the architecture of the generative adversarial network to obtain a high-quality missing mode synthesis result.

[0030] As shown in Figure 1 A multi-modal image synthesis method based on a recursive fusion conversion unit is composed of six components, and the content of each part is as follows:

[0031] As shown in Figure 2 In the multi-branch encoder component, the embodiment of the application converts multi-modal input data from an image space to a feature space.

[0032] Further, the multi-branch encoder Encoder based on CNN inferences and preliminarily extracts the local features of each mode for subsequent inference use. i ,i∈{1,2,…,M} respectively.

[0033]

[0034] Then The graph element embedding is converted into the input required by the Transformer in the form of an initial information of global fusion. The formal expression of the graph element embedding is as follows: given an input image with a size of HxWxC, the input image is first divided into N non-overlapping graph elements with a size of PxPx C, each graph element is flattened into a one-dimensional vector, and then converted into an embedding vector with a dimension of D through a convolutional mapping. In order to preserve the position information, each graph element embedding is added with the corresponding position encoding PE. The whole process can be represented as: F0=Conv(flatten(C0'))+PE, wherein F0 is the initial fusion input of global fusion, containing the feature information specific to each input mode, and C0' represents the initial feature from different modes splicing:

[0035]

[0036] In the fusion transformation unit component, embodiments of the present invention use local representation blocks to extract local features of specific input modalities, and use feature transformation gates to filter effective features, transform feature shapes, and send them into the global fusion block to extract long-distance dependencies between different modalities.

[0037] Furthermore, the CNN-based Local Representation Block (LRB) specifically comprises: each LRB consists of a ResBlock with two 3×3 convolutional layers, a ReLU activation function, and a 1×1 convolutional layer. The formal expression of the ResBlock is as follows:

[0038] X′=ReLU(BatchNorm(Conv3(X)))

[0039] X″=I)atchNorm(Conv3(X′))

[0040] ResBlock(X) = X″ + X

[0041] Here, X represents the original input of the network, Conv3 represents a 3×3 convolutional layer, and BatchNorm is a batch normalization layer. It uses mini-batch statistics for normalization to prevent internal covariate shifts in the distribution, fixes the distribution range, and introduces learnable parameters to maintain the dataset's unique statistical characteristics, preventing the loss of learned information. ResBlock alleviates the gradient vanishing problem in deep networks through residual connections, allowing the network to effectively extract high-level features by increasing its depth. Then, ResBlock and a 1×1 convolutional layer (Conv1) form a local representation block as follows:

[0042] LRB=Conv1(ReLU(ResBlock(x)))

[0043] Furthermore, the feature transformation gate operation specifically includes: the local feature map C output by the LRB. i-1 The data are fed into 1×1 convolutional layers and gated linear units to filter effective features, which are then transformed into the Transformer feature space for subsequent global fusion. The method for filtering effective features is as follows:

[0044] S i =C i-1 +GLU(Conv1(C i-1 ))

[0045] where C i-1 is the output of LRB in the i-1th recursion stage, S i is the selected feature map, and GLU is defined as:

[0046]

[0047] where W1, W2, b1, b2 are all learnable parameters, is element-wise multiplication, and σ is the Sigmoid function. Since the CNN feature map (H, W, C are the height, width, and channel number of the feature map) in LRB is inconsistent with the Transformer embedding (L, D are the embedding number and embedding dimension of the feature), the feature conversion gate adopts two 3x3 convolution layers to convert the CNN spatial feature into the Transformer feature space, and adopts Batch Normalization (Batch Normalization) for regularization processing.

[0048] CT i = GELU(BatchNorm(Conv3(S i ))

[0049] Further, the Transformer-based global fusion block (Global Fusion Block, GFB) specifically includes: a Multi-Head Self-Attention (Multi-Head Self-Attention, MHSA) and a Multilayer Perceptron (Multilayer Perceptron, MLP) to form a Transformer layer, and multiple Transformer layers are stacked to form a GFB:

[0050] x' = MHSA(LayerNorm(x)) + x

[0051] TransformerLayer = MLP(LayerNorm(x')) + x'

[0052] GFB = TransformerLayer2(TransformerLayer1(x))

[0053] Here, x is the input to the GFB, the MLP consists of multiple linear mappings and nonlinear activation functions, and LayerNorm represents Layer Normalization. Its principle is the same as BatchNorm, but instead of using a batch as the unit, it uses a single sample as the unit, calculating the mean and variance of features in each channel to perform distribution normalization and train learnable parameters. Compared to BatchNorm, LayerNorm is less sensitive to batch size and is more suitable for normalizing variable-length sequences in Transformers.

[0054] MHSA employs multiple independent self-attention heads (SAs), each calculating a weighted sum of all elements in the input sequence. The results from different heads are combined and linearly mapped to obtain the output of MHSA.

[0055] MHSA(Q,K,V)=Concat(SA1,SA2,SA3,...,SA h W O

[0056]

[0057] in, Let d represent the query, key, and value calculated through learnable linear projection, respectively. k For the dimension K, W O This represents a learnable linear mapping matrix.

[0058] In the recursive expansion component, the output of the previous recursive stage is used as the input of the next recursive stage in this embodiment of the invention, and multiple inferences are performed in the same module to enable more thorough training of the module parameters.

[0059] Furthermore, in order to propagate local features during the recursive phase, LRB will use the output C from the previous phase. i-1 The output C0 of the encoder is concatenated with the input. Local features C1 are obtained by relying on C0, while C2 is obtained by relying on both C0 and C1, and so on. The output of LRB in stage i is formally expressed as:

[0060] C i =LRB(concat(C i-1 ,C0)),i∈{1,2,...,R}

[0061] Furthermore, in addition to fusing modal local feature information from different LRBs, GFB also needs to preserve the long-range dependencies extracted in previous recursive stages. Therefore, GFB will extract CT data from the feature transformation gate. i T compared to the previous recursive GFB outputi-1 Element-wise summation is performed as input:

[0062] T i = GFB(T i- 1+∑CT i )

[0063] In the multi-branch decoder component, embodiments of the present application utilize the input modal multi-branch feature reconstruction to reconstruct the missing modal image, utilize the fusion branch global feature reconstruction to reconstruct the missing image, and utilize the local feature reconstruction in the intermediate recurrent inference stage to reconstruct the intermediate result.

[0064] Further, the multi-branch decoder utilizes the local feature extracted by the local representation block in the final stage for each input modal branch Synthesize the missing modal image

[0065]

[0066] Further, for the inference process with R recurrent stages, the decoder will reconstruct the feature map of M modalities in each stage based on the local feature of the corresponding recurrent stage, resulting in Mx(R-1) intermediate results And calculate the loss with the label image of the corresponding modal To supervise the parameter update direction in the recurrent process:

[0067]

[0068] Where y i is the real label image data of a specific modality. Subsequently, it will be backpropagated together with other losses.

[0069] Further, the multi-branch decoder also performs image synthesis work for the fusion branch based on global features. For the fusion branch that has gone through R inference stages, the final fusion feature T R can be obtained. Since the decoder is a CNN architecture, T R needs to be restored to the feature space of through graph element expansion. Similar to graph element embedding, the N dimension of T is first restored to the two-dimensional space of HxW through reshaping, and then through convolution mapping, the D-dimensional vector is mapped to the C-dimensional channel to obtain the converted fusion feature

[0070]

[0071] In the mixed gate module component, the embodiment of the application calculates adaptive fusion weights for the synthesized image of the multiple branches, and obtains the final missing modality fusion result through weighting.

[0072] Further, for the M+1 branch images synthesized by the decoder, i.e. M and 1 The mixed gate module calculates the weight map of each branch adaptively to fully use the information of all input modality branches and global fusion branches in the image space, and obtain accurate synthesis results:

[0073]

[0074] wherein i is an element of {1, 2,..., M+1}, and the Softmax function normalizes the M+1 weight maps of each pixel.

[0075] Further, according to the weight map learned by the gate module, multiple synthesized images are summarized to obtain the final target image:

[0076]

[0077] wherein, is element multiplication.

[0078] In the adversarial generation network component, the aforementioned components collectively constitute a generator, which promotes the performance of each other through mutual confrontation with a discriminator, and calculates an adversarial loss to guide the optimization direction.

[0079] Further, the discriminator D, as a part of the adversarial generation network architecture that dynamically confronts the generator G and promotes each other, is composed of multiple convolution blocks. Each convolution block includes a 3D convolution layer, a nonlinear activation function ReLU, and an InstancenNorm (Instance Normalization). The discriminator D finally outputs a binary classification result through a Sigmoid function. The generative adversarial network guides the entire dynamic confrontation and optimization process by the adversarial loss. The adversarial loss of the generator G is:

[0080]

[0081] The adversarial loss of the discriminator is:

[0082]

[0083] wherein X t represents a real label image of the missing modality.

[0084] Further, during the training process, the generator based on the fusion conversion unit and the hybrid gating module learns the modal feature information contained in the multi-modal input sufficiently to synthesize the fake-looking image. The discriminator perceives and classifies the synthesized image to assist the generator in adjusting the parameter update direction during the gradient backpropagation process, supplement the high-frequency details in the synthesized image, and update its own parameters to improve the ability to distinguish images.

[0085] Embodiment 1

[0086] The embodiment of the present application provides a multi-modal image synthesis method based on a recursive fusion conversion unit, which comprises the following steps:

[0087] 101: The images of each input modal are converted from the image space to the feature space by a multi-branch encoder to obtain initial local image features. Meanwhile, the local features of each modal are converted into the Transformer feature space through mutual splicing and graph element embedding operations as initial fusion features.

[0088] Specifically, the graph element embedding divides the input image of HxWxC into N non-overlapping graph elements of PxPx C, each graph element is expanded into a one-dimensional vector, and then C channels are converted into an embedding vector of dimension D through convolution mapping. The converted features are added with the corresponding position encoding PE to retain the position information.

[0089] 102: The fusion conversion unit learns the local image features through the local representation block, filters and converts the effective features to the Transformer feature space through the feature conversion gate, and combines the effective features from multiple input modalities and the old global features in the Transformer feature space through the global fusion block, and infers the new deep global features;

[0090] Specifically, the feature conversion gate filters the features of the local representation block through a 1x1 convolution layer and a gated linear unit, retains the effective information, and converts the feature form through a 3x3 convolution layer and batch normalization. The global fusion block is consistent with the ViT structure, splices the local feature information from multiple modal branches and the fusion output of the previous stage as the input of this stage, and captures the dependency relationship through the multi-head self-attention mechanism.

[0091] 103: The fusion conversion unit adopts a recursive unfolding inference form to gradually extract local image features and global fusion features of different depths without increasing the number of parameters;

[0092] Specifically, the local representation block extracts deep features in the current stage using the local feature information output by the previous stage as input. The global fusion block captures the intra-modal and inter-modal dependencies of local features by taking the output of the local representation block of the previous stage after feature conversion gate transformation and the output of the global fusion block of the previous stage as input.

[0093] 104: The multi-branch decoder synthesizes the intermediate results of the missing modal using the local image features obtained by the local representation blocks of different modal branches at different recursive stages. The global features extracted by the global fusion block of the fusion branch are also synthesized into the missing modal image after graph element expansion.

[0094] Specifically, the image expansion first reshapes the N dimension of the global feature in the form of NxD to the two-dimensional space of HxW, and then restores the D dimension to C channels through convolution operation. In this way, the global fusion feature is converted back to the CNN feature space.

[0095] 105: The hybrid gating module further synthesizes the missing modal image in each branch using the multi-branch decoder, and dynamically fuses the synthesis results of different modalities with the results of the fusion branch in the fusion image space.

[0096] Specifically, the M specific input modal features extracted by the local representation block will be synthesized into the missing modal image by the decoder, and the fusion features learned by the global fusion block will also be reconstructed into the missing modal by the decoder. The obtained M+1 synthesized images are connected by residual connection and normalized by Softmax, giving each image a pixel-level weight to flexibly fuse the modal information of different branches.

[0097] 106: The fusion conversion unit and the hybrid gating module are connected in series to form the generator. The discriminator and the generator constitute an adversarial generation network framework, and the discriminator takes the missing modal real image data and the synthesized data of the generator as input, and outputs different classification results, which are used for subsequent calculation of adversarial loss.

[0098] 107: The adversarial loss and reconstruction loss values of the fusion reconstruction image of different modal branches and different recursive depths are calculated, summed, and then backpropagated to train the network.

[0099] The obtained fusion reconstruction image can calculate the adversarial loss and reconstruction loss of the corresponding modal. The reconstruction loss adopts the form of L1 loss, including the loss between the intermediate results in the recursive process and the real label image, and the loss between the final fusion result and the real label image.

[0100] To sum up, the embodiment of the application gradually generates high-quality missing modal images through six parts of multi-branch encoder, fusion conversion unit, recursive unfolding, multi-branch decoder and hybrid gating module; wherein the sensitivity of the convolutional neural network to local image features and the learning ability of the Transformer to long-distance dependency between graph elements are used to fully extract multi-modal image information; the inference form of recursive unfolding is used to realize a low-parameter and high-performance feature extraction module (compared with stacking multiple modules with the same structure in series, the parameter amount is less, and the same sample amount of the dataset can be used to train more fully and obtain better results); the hybrid gating module is used to calculate the weight in the image space, flexibly fuse the synthesized image results from different modalities, and fully train multiple modules in the generative adversarial network framework to obtain high-quality missing modal images that are more consistent with human visual perception.

[0101] Embodiment 2

[0102] The embodiment of the application provides a recursive unfolding fusion conversion unit inference method. The fusion conversion unit combines the advantages of convolutional neural network and Transformer, and has good feature extraction capability. However, the Transformer often has a large number of parameters, and a large-scale dataset is needed to find the global optimal solution in training. In order to reduce the dependence of the Transformer on a large amount of data, the embodiment of the application introduces a recursive unfolding inference method, which reduces the network parameter amount by reducing the cascaded Transformer blocks, and guarantees the effectiveness of network parameter training by repeatedly unfolding six times. The method takes the output of the last inference of the local representation block in the fusion conversion unit as the input of the next inference of the local representation block, and takes the conversion result spliced from the last global fusion block and the last local representation block output as the input of the next global fusion block to complete recursive inference. In order to ensure the effective transmission of local features in the recursive process, the decoder is used to reconstruct the local features extracted by each modal branch at each recursion, and the loss is calculated with the real label image to update the network parameters.

[0103] As shown in Table 1, the embodiment of the present application is designed to test the effectiveness in three task scenarios. Among them, w / o RU represents the reasoning mode without recursive expansion, but using six cascaded fusion conversion units for calculation; w / o Lr represents that the loss supervision network parameter optimization is no longer calculated for the intermediate results of each modality in the recursive expansion reasoning. As can be seen from the four evaluation indexes in the table, the algorithm design of recursive expansion achieves better results than cascading multiple fusion conversion units, and the parameter quantity is reduced from 137.6M to 39.51M, greatly reducing the demand for the number of training samples; and the constraint of the intermediate results of recursion effectively guides the iteration direction of the network parameters in the recursive process, so that the synthesis has more real and natural, high-quality results.

[0104] Table 1

[0105]

[0106] Embodiment 3

[0107] The scheme in Embodiment 1 is further introduced in combination with specific examples and calculation formulas, and details are described as follows:

[0108] I. Data preparation

[0109] The embodiment of the present application can be extended to any scene of missing modality image synthesis task. The embodiment of the present application verifies the effectiveness of the method on two brain disease public data sets and one indoor scene data set. Among them, the brain tumor segmentation data set (Brain Tumor Segmentation 2020 dataset, BraTS2020), the ischemic stroke lesion segmentation data set (Ischemic Stroke Lesion Segmentation 2015 dataset, ISLES2015) and the indoor scene data set (Taskonomy).

[0110] BraTS2020 is a public brain tumor segmentation data set with good calibration and alignment. It includes 369 training samples and 125 test samples. Each sample contains T1, T1Gd, T2, T2-FLAIR four modalities and corresponding segmentation annotations.

[0111] ISLES2015 is a public ischemic stroke lesion segmentation data set. It includes: subacute ischemic stroke lesion segmentation (SISS) and acute stroke outcome (SPES), and the SISS part is selected in the embodiment, and 45 samples are divided into 28 training samples and 17 test samples, wherein each sample contains: T1, T2, T2-FLAIR, DWI four kinds of modality data.

[0112] Taskonomy is a public indoor scene dataset that helps to verify the generalization of the method of the present application. Each sample in the dataset has more than ten modalities, including: RGB images, texture images, depth images, edge images, etc. Five modalities are selected for the experiment in this embodiment, including: depth images, shadow images, normal images, texture images and RGB color images. The training set includes 1000 samples, and the test set includes 500 samples.

[0113] II. Network structure of the multi-modal image synthesis method based on recursive fusion conversion unit

[0114] The network structure of the multi-modal image synthesis method based on recursive fusion conversion unit in the embodiment of the present application, as shown in Figure 1 , is composed of six components, and the content of each part is as follows:

[0115] As shown in Figure 2 , in the multi-branch encoder component, the embodiment of the present application converts the multi-modal input data from the image space to the feature space.

[0116] Further, the multi-branch encoder Encoder based on CNN infers and preliminarily extracts the local features of each modality for subsequent inference use. i , i∈{1,2,…,M}, respectively.

[0117]

[0118] Then is converted into the input form required by the Transformer as the initial information of global fusion. The formalization of the graph element embedding is as follows: given an input image with size H×W×C, it is first divided into N non-overlapping graph elements with size P×P×C, each graph element is flattened into a one-dimensional vector, and then converted into an embedding vector with dimension D through a convolutional mapping. In order to preserve the position information, each graph element embedding is added with the corresponding position encoding PE. The whole process can be represented as: F0=Conv(flatten(C0'))+PE, where F0 is the initial fusion input of global fusion, containing the feature information specific to each input modality, and C0' represents the splicing of the initial features from different modalities

[0119]

[0120] ​In the fusion conversion unit component, the embodiment of the present application uses a local representation block to extract specific input modal local features, and filters effective features and converts feature shapes through a feature conversion gate, and sends them into a global fusion block to extract long-distance dependency relationships between different modalities.

[0121] Further, the CNN-based local representation block specifically comprises: each local representation block (LRB) is composed of a ResBlock with two 3x3 convolution layers, an activation function ReLU, and a 1x1 convolution layer. The formal expression of ResBlock is:

[0122] X' = ReLU(BatchNorm(Conv3(X)))

[0123] X'' = BatchNorm(Conv3(X'))

[0124] ResBlock(X) = X'' + X

[0125] wherein X represents the original input of the network, Conv3 represents a 3x3 convolution layer, BatchNorm is a batch normalization layer, which prevents distribution internal covariate shift by normalizing using batch statistics, fixes the distribution range and introduces learnable parameters to maintain the statistical characteristics specific to the data set, and prevents the loss of learned information. ResBlock alleviates the problem of gradient disappearance in deep networks through residual connection, so that the network can effectively extract high-level features by increasing the depth. Then ResBlock and 1x1 convolution layer (Conv1) constitute a local representation block in the following form:

[0126] LRB = Conv1(ReLU(ResBlock(x)))

[0127] Further, the feature conversion gate operation specifically comprises: the local feature map C i-1 output by the LRB is sent into a 1x1 convolution layer and a gated linear unit to filter effective features and convert the effective features into a Transformer feature space for subsequent global fusion. The method for filtering effective features is:

[0128] S i = C i-1 + GLU(Conv1(C i-1 ))

[0129] wherein C i-1 is the output of the LRB in the i-1th recursive stage, and S iFor selected feature maps, GLU is defined as:

[0130]

[0131] where W1, W2, b1, b2 are learnable parameters, is element-wise multiplication, and σ is Sigmoid function. Given that the CNN feature maps in LRB (H, W, C are height, width, and channel number of feature maps) are inconsistent with the Transformer embedding in the Global Fusion Block (L, D are the number of embeddings and embedding dimension of features), the feature conversion gate adopts two 3x3 convolution layers to convert the CNN spatial features into the Transformer feature space, and adopts Batch Normalization (Batch Normalization) for regularization processing.

[0132] CT i = GELU(BatchNorm(Conv3(S i ))

[0133] Further, the Global Fusion Block (GFB) based on the Transformer specifically includes: the Multi-Head Self-Attention (MHSA) and the Multilayer Perceptron (MLP) form a Transformer layer, and multiple Transformer layers are stacked to form the GFB:

[0134] x' = MHSA(LayerNorm(x)) + x

[0135] TransformerLayer = MLP(LayerNorm(x')) + x'

[0136] GFB = TransformerLayer2(TransformerLayer1(x))

[0137] where x is the input of the GFB, the MLP is composed of multiple linear mappings and nonlinear activation functions, and LayerNorm represents Layer Normalization. The principle is the same as BatchNorm, but it is no longer in a batch unit, but in a single sample unit, and the mean and variance of the features in each channel are calculated to perform distribution normalization and learnable parameter training. Compared with BatchNorm, LayerNorm is not sensitive to batch size, and is more suitable for sequence normalization of indefinite length in the Transformer.

[0138] MHSA adopts multiple independent self-attention heads (SA), each of which calculates the weighted sum of all elements in the input sequence, and the results of different heads are spliced to obtain the output of MHSA after linear mapping:

[0139] MHSA(Q, K, V) = Concat(SA1, SA2, SA3,..., SA h )W O

[0140]

[0141] wherein, respectively represent the query, key and value calculated by a learnable linear projection, d k is the dimension of K, and W O represents a learnable linear mapping matrix.

[0142] In the recursive unfolding component, the output of the previous recursive stage is taken as the input of the next recursive stage, and multiple inferences are performed in the same module to make the module parameters more fully trained.

[0143] Further, in order to propagate local features in the recursive stage, the LRB splices the output C i-1 of the previous stage and the output C0 of the encoder as input, and the local feature C1 is obtained depending on C0, and C2 is obtained depending on C0 and C1 together, and so on. The output of the LRB in the i-th stage is formalized as:

[0144] C i = LRB(concat(C i-1 , C0)), i e {1, 2,..., R}

[0145] Further, in addition to the need to fuse modal local feature information from different LRBs, the GFB also needs to retain the long-distance dependency relationships extracted in the past recursive stages. Therefore, the GFB performs element-wise addition of the CT i extracted by the feature transformation gate and the T i-1 output by the last recursive GFB as input:

[0146] T i = GFB(T i-1 +∑CT i )

[0147] In the multi-branch decoder component, embodiments of the present invention utilize the multi-branch features of the input modality provided by the fusion conversion unit to reconstruct the missing modality image, utilize the global features of the fusion branch to reconstruct the missing image, and utilize the local features in the intermediate recursive inference stage to reconstruct the intermediate result.

[0148] Furthermore, the multi-branch decoder utilizes the final-stage local features extracted from local representation blocks for each input modality branch. Synthetic missing modality images

[0149]

[0150] Furthermore, for an inference process with R recursive stages, the decoder will reconstruct the feature maps of the M modalities in each stage based on the local features of the corresponding recursive stage, resulting in M×(R-1) intermediate results. The loss is calculated by comparing the loss image with the label image of the corresponding modality to supervise the direction of parameter update during the recursive process.

[0151]

[0152] Among them, y i Real-world labeled image data for a specific modality. It will then be added to other losses and propagated backwards.

[0153] Furthermore, the multi-branch decoder also performs image synthesis for the fusion branch based on global features. For the fusion branch after R inference stages, the final fusion feature T can be obtained. R Because the decoder uses a CNN architecture, therefore T R It needs to be restored to its original state through primitive expansion. In the feature space. Primitive expansion is similar to primitive embedding. The N-dimensional vector is first reshaped back to a two-dimensional space of H×W, and then, through convolution mapping, the D-dimensional vector is mapped to the C-dimensional channel to obtain the transformed fused feature. Used to decode and generate images

[0154]

[0155] In the hybrid gating module component, this embodiment of the invention calculates adaptive fusion weights for multi-branch synthetic images and obtains the final missing modality fusion result through weighting.

[0156] Furthermore, for the M+1 branch images synthesized by the decoder, that is, M branches... With 1 The mixed gate module adaptively calculates the weight map of each branch to fully use the information of all input modal branches and the global fusion branch in the image space, and obtains an accurate synthesis result:

[0157]

[0158] wherein i is an element of {1, 2,..., M+1}, and the Softmax function normalizes the M+1 weight maps of each pixel.

[0159] Further, according to the weight map learned by the gate module, multiple synthesized images are summarized to obtain a final target image:

[0160]

[0161] wherein, is element multiplication.

[0162] In the adversarial generative network component, the aforementioned components collectively constitute a generator, which promotes the performance of each other through mutual confrontation with a discriminator, and calculates an adversarial loss to guide the optimization direction.

[0163] Further, the discriminator D, as a part of dynamic confrontation with the generator G in the adversarial generative network architecture, promotes each other and is composed of multiple convolutional blocks. Each convolutional block includes a 3D convolution layer, a nonlinear activation function ReLU, and an InstancenNorm (Instance Normalization). The discriminator D finally outputs a binary classification result through a Sigmoid function. The generative adversarial network guides the entire dynamic confrontation and optimization process by the adversarial loss. The adversarial loss of the generator G is:

[0164]

[0165] The adversarial loss of the discriminator is:

[0166]

[0167] wherein X t represents a real label image of a missing modality.

[0168] Further, in the training process, the generator based on the fusion conversion unit and the mixed gate module fully learns the modal feature information contained in the multi-modal input to synthesize a fake image. The discriminator perceives and classifies the synthesized image to assist the generator in adjusting the parameter update direction in the gradient backpropagation process, supplement the high-frequency details in the synthesized image, and update its own parameters to improve the ability to distinguish images.

[0169] III. Evaluation indicators and protocols

[0170] In order to accurately evaluate the quality of the synthesized image, four general image quality evaluation indexes are adopted in the embodiment of the present application, including: Peak Signal-to-Noise Ratio (PSNR), Feature Similarity Index Measurement (FSIM), Structural Similarity Index Measurement (SSIM), and Normalized Mean Squared Error (NMSE).

[0171] wherein the PSNR is for a given real image y and a synthesized image is defined as:

[0172]

[0173] wherein N is the number of pixels of the image. The PSNR can objectively reflect the distortion degree of the image, and the greater the value is, the less the distortion is, and the better the quality is.

[0174] wherein the FSIM is for a given real image y and a synthesized image is defined as:

[0175]

[0176] S L (x) = [S PC (x)] α [S GM (x)] β

[0177]

[0178] wherein α and β are usually taken as 1. The FSIM combines the phase consistency and the gradient amplitude, and thus has good evaluation ability when the image has more serious noise or blur degradation. The greater the value is, the better the synthesis quality is.

[0179] wherein the SSIM is for a given real image y and a synthesized image is defined as:

[0180]

[0181] wherein is the image mean value, is the image variance, is the covariance, and c1 and c2 are constants to prevent division by zero error. The greater the SSIM value is, the more similar the synthesized image is to the real image in structure.

[0182] where NMSE for a given real image y and synthesized image is defined as:

[0183]

[0184] The smaller the NMSE value, the smaller the error between the synthesized image and the real result, and the better the quality.

[0185] Four, the use details of the model

[0186] 1. Data supplement: The embodiments of the present application use different modalities in multiple data sets as input to verify the synthesis performance of missing modalities in different scenarios, and are divided into two types of tasks of single modality input scene and multi-modality input scene. Specifically, for the single modality input scene, the BraTS2020 data set and the ISLES2015 data set adopt the FLAIR modality as the input, which corresponds to the samples of synthesizing T1, T2, T1Gd and DWI modalities respectively; adopt T1 modality as input, which corresponds to the sample of synthesizing FLAIR modality. The Taskonomy data set adopts Shade modality to synthesize RGB, and Normal modality to synthesize RGB. For the multi-modality input scene, the BraTS2020 data set and the ISLES2015 data set adopt T2+FLAIR to synthesize T1, adopt T1+FLAIR to synthesize T2 and T1Gd or DWI, and adopt T1+T2 to synthesize FLAIR. The Taskonomy data set adopts Shade+Texture to synthesize RGB, and Normal+Depth to synthesize RGB. All input samples are unified to 224x224.

[0187] 2. Model optimization: The embodiments of the present application adopt Adam optimizer, and the parameters β1=0.5, β2=0.999, and the learning rate is set to 2.0x10-4. In order to ensure sufficient training, the network parameters will be trained for 600 rounds.

[0188] 3. Hyperparameter setting: The number of recursive expansion layers R of the embodiments of the present application is 6, and the weighting ratio λ in the loss function is 100.

[0189] 4. Loss setting: The embodiments of the present application contain the adversarial loss (L adv ), the reconstruction loss (L1) and the recursive loss (L r). The adversarial loss and the reconstruction loss are common loss constraints in the generative adversarial network, responsible for providing high-frequency and low-frequency information for the generator and guiding the parameter optimization direction. In addition to the above two common loss functions, the embodiment of the present application also designs a recursive loss, which calculates the L1 difference between the intermediate results generated by the decoder and the real label image in each recursive expansion process of the fusion conversion unit, to guide the transmission direction of local features in the recursive process. The total loss function of the network is L total =L adv +L1+L r .

[0190] In order to solve the technical problems existing in the multi-modal image synthesis method, the embodiment of the present application proposes a multi-modal image synthesis method based on recursive fusion conversion unit, which fully extracts the local feature information of the image and the long-distance dependence relationship between the modes by combining the convolutional neural network and the Transformer network structure to form a fusion conversion unit. The method realizes the adaptive selection of effective features in different recursive depths and the feature conversion between different network structures by designing a feature conversion gate. The method reduces the model parameters and improves the model performance without reducing the model performance by designing the recursive expansion method of the fusion conversion unit to constrain the synthesis results of each recursive stage. The method adaptively mixes the synthesis images of specific modes and fusion modes in the image space by designing a hybrid gating module, thereby fully utilizing the different information of multi-modal images and improving the synthesis quality of missing modes.

[0191] The embodiments of the present application have the following four key points:

[0192] I. A multi-modal image synthesis method based on recursive fusion conversion unit is proposed

[0193] Technical effect: The method uses the sensitivity of the convolutional neural network to local image features and the capture ability of ViT to long-distance dependence relationship, and designs a multi-modal image synthesis method based on recursive fusion conversion unit, so that the model realizes dynamic compatibility of local and global information during training, obtains local image features, and considers the long-distance dependence relationship within and between modes of local features. Ultimately, leading results are obtained in the missing mode synthesis task, and a high-quality multi-modal image synthesis model is realized;

[0194] II. A feature conversion method is proposed

[0195] Technical effect: The feature conversion method uses the gating linear unit to adaptively select effective features in the previous recursive process during training, and automatically converts the features of the convolutional neural network into a form suitable for ViT to further extract dependence relationships;

[0196] III. A recursive unfolding reasoning method is proposed

[0197] Technical effects: By recursively unfolding the fusion conversion unit, the parameter quantity of the overall model is reduced, the dependence on the training data quantity is reduced, and by adding constraints to each recursive stage, the accuracy of the deep network features is ensured, thereby enhancing the quality of the synthesized image under the premise of a small number of parameters.

[0198] IV. A method for synthesizing missing modality images based on a hybrid gating module is proposed

[0199] Technical effects: The method constructs a hybrid gating module for specific modality branches and multi-modal fusion branches, adaptively calculates the weight map of each branch through convolution operation and element-level Softmax normalization, fully utilizes the feature information of the multi-modal branch, flexibly mixes multiple synthesis results in the image space, and obtains the final missing modality synthesis result.

[0200] In summary, the embodiment of the present application provides a multi-modal image synthesis method based on a recursive fusion conversion unit, designs a feature conversion method to realize the conversion and transmission of features between different network structures, recursively unfolds the fusion conversion unit to reduce the parameter quantity of the overall model, and enhances the quality of the synthesized image under the premise of a small number of parameters. A method for synthesizing missing modality images based on a hybrid gating module is designed to flexibly mix multiple synthesis results in the image space and obtain the final missing modality synthesis result.

[0201] Example 4

[0202] The method proposed in the embodiment of the present application is compared with a plurality of technical methods in single modality input and multi-modality input on two brain medical data sets and indoor scene data sets.

[0203] I. Single modality input comparison

[0204] For the single modality input scene, the experimental results are shown in Table 2: the image synthesis performance is evaluated in four quantitative indicators based on the embodiment of the present application. The multi-modal image synthesis method based on the recursive fusion conversion unit is superior to other popular image synthesis methods, which proves the utilization of local and global features of the image by the method and the superior performance of the missing modality image synthesis. In addition, the excellent performance on the two medical data sets shows that the method has stable synthesis performance, and the indicators on the indoor scene data also prove that the method has good generalization performance and can be migrated to different scenes to complete the task.

[0205] Table 2

[0206]

[0207] II. Multi-modality input comparison

[0208] The experimental results for the multi-modal input scene are shown in Table 3: the embodiment of the application uses four quantitative indicators to evaluate the image synthesis performance. The multi-modal image synthesis method based on the recursive fusion conversion unit achieves leading synthesis performance in popular multi-modal image synthesis methods, and the evaluation indicators on three data sets all reach the best scores, which again proves that the method effectively utilizes the image structure and high and low frequency features, and can synthesize real and natural high-quality missing modalities.

[0209] Table 3

[0210]

[0211]

[0212] A multi-modal image synthesis device based on a recursive fusion conversion unit, the device comprising: a processor and a memory, the memory storing program instructions, the processor invoking the program instructions stored in the memory to make the device execute the following method steps in embodiment 1:

[0213] The fusion conversion unit learns local image features through a local representation block, filters effective features from the local image features through a feature conversion gate, converts the effective features to a Transformer feature space, and fuses the effective features with old global features through a global fusion block to infer new global features; the fusion conversion unit adopts a recursive unfolding inference form to extract local and global features at different depths;

[0214] The multi-branch decoder synthesizes missing modality images using local features at intermediate depths in each branch, calculates loss to guide optimization, and synthesizes multiple missing modality reconstructed images using local and global features at the last stage; the hybrid gating module dynamically fuses multiple missing modality reconstructed images in the fusion image space;

[0215] The fusion conversion unit, the multi-branch decoder and the hybrid gating module are connected in series to form a generator, and the generator and the discriminator form a generative adversarial network framework;

[0216] The image reconstructed using features of different branches and different depths is calculated respectively to calculate the adversarial loss and the reconstruction loss, and the network is trained through back propagation.

[0217] The local representation block is:

[0218] LRB = Conv1(ReLU(ResBlock(x)))

[0219] The local feature map C output by the LRB i-1The effective features are sent into a 1x1 convolutional layer and a gated linear unit, and the effective features are converted into a transformer feature space.

[0220] The recursive expansion is that the output of a previous recursive stage is taken as the input of a subsequent recursive stage, and multiple inferences are performed in the same module to train the module parameters.

[0221] The multi-branch decoder is:

[0222] The missing modality image is reconstructed by using the input modality multi-branch feature provided by the fusion conversion unit, the missing image is reconstructed by using the fusion branch global feature, and the intermediate result is reconstructed by using the local feature in the intermediate recursive inference stage.

[0223] The mixed gating module is a multi-branch synthetic image that calculates adaptive fusion weights, and the final missing modality fusion result is obtained by weighting.

[0224] It should be pointed out that the device description in the above embodiments corresponds to the method description in the embodiments, and the embodiments of the present application will not be repeated here.

[0225] The execution subject of the processor and the memory described above can be a computer, a single-chip microcomputer, a microcontroller, etc. having a computing function, and in specific implementation, the embodiments of the present application do not limit the execution subject, and the selection is made according to the actual application needs.

[0226] The memory and the processor transmit data signals through a bus, and the embodiments of the present application will not be repeated here.

[0227] Based on the same inventive concept, the embodiments of the present application also provide a computer readable storage medium, the storage medium includes a stored program, and when the program runs, the device where the storage medium is located executes the method steps in the above embodiments.

[0228] The computer readable storage medium includes but is not limited to a flash memory, a hard disk, a solid state disk, etc.

[0229] It should be pointed out that the readable storage medium description in the above embodiments corresponds to the method description in the embodiments, and the embodiments of the present application will not be repeated here.

[0230] In the above embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, it can be realized in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the flow or function according to the embodiments of the present application is generated in whole or in part.

[0231] The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in or transmitted by a computer-readable storage medium, such as a magnetic medium, an optical medium, a semiconductor medium, etc. The computer-readable storage medium can be a server, a data center, or other data storage device that includes one or more of the available media integrated.

[0232] The model of each device in the embodiments of the present application is not limited, except for the special description, and the device that can complete the above functions can be used.

[0233] Those skilled in the art can understand that the drawings are only schematic diagrams of a preferred embodiment, and the above-mentioned serial numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0234] The above is only a preferred embodiment of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-modal image synthesis method based on recursive fusion transformation unit, characterized in that, The method comprises: The fusion conversion unit learns local image features through local representation blocks, filters effective features from the local image features through a feature conversion gate, converts the effective features to a Transformer feature space, and fuses the effective features with old global features through a global fusion block to infer new global features; the fusion conversion unit adopts a recursive unfolding inference form to extract local and global features at different depths; The multi-branch decoder synthesizes intermediate results of missing modalities using local image features obtained by local representation blocks of different modal branches at different recursion stages, and synthesizes missing modality images after the global features extracted by the global fusion blocks of the branches are expanded through the decoder, and calculates a loss to guide optimization; the last-stage local and global features are used to synthesize multiple missing modality reconstructed images; the hybrid gating module dynamically fuses multiple missing modality reconstructed images in a fusion image space; The fusion conversion unit, the multi-branch decoder, and the hybrid gating module are connected in series to form a generator, and the generator and the discriminator form a generative adversarial network framework; The images reconstructed using features of different branches and different depths are respectively calculated for adversarial loss and reconstruction loss, and the network is trained through back propagation; The local representation block is: LRB = Conv1(ReLU(ResBlock(x))) Local feature maps C output by the LRB i-1 The effective features are filtered and converted into a Transformer feature space by being fed into a 1x1 convolutional layer and a gated linear unit. The multi-branch decoder is: The missing modality images are reconstructed using the input modality multi-branch features provided by the fusion conversion unit, the missing images are reconstructed using the global features of the fusion branches, and the intermediate results are reconstructed using the local features in the intermediate recursion inference stage. The hybrid gating module calculates adaptive fusion weights for the synthesized images of the multi-branches, and obtains the final missing modality fusion result through weighting. 2.The multi-modal image synthesis method based on recursive fusion transform unit according to claim 1, wherein, The recursive unfolding is: the output of the previous recursion stage is used as the input of the next recursion stage, and multiple inferences are performed in the same module to train the module parameters.

3. A multi-modal image synthesis apparatus based on recursive fusion transform unit, characterized by, The device comprises a processor and a memory, the memory stores program instructions, and the processor invokes the program instructions stored in the memory to make the device execute the method of any one of claims 1-2.

4. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, the computer program comprises program instructions, and the program instructions are executed by the processor to make the processor execute the method of any one of claims 1-2.

Images