Multi-modal image synthesis method and device based on recursive fusion conversion unit

Abstract

The invention discloses a multi-modal image synthesis method and device based on a recursive fusion conversion unit, and the method comprises the steps: the fusion conversion unit employs a recursive expansion reasoning mode, and extracts local and global features of different depths; the multi-branch decoder synthesizes a missing modal image by using the local feature of the middle depth of each branch, and calculates loss to guide optimization; synthesizing a plurality of missing modal reconstruction images by using the local and global features of the final stage; the hybrid gating module dynamically fuses a plurality of missing modal reconstruction images in the fusion image space; the fusion conversion unit, the multi-branch decoder and the hybrid gating module are connected in series to form a generator, and the generator and the discriminator form a generative adversarial network framework; and calculating adversarial loss and reconstruction loss for images reconstructed by using features of different branches and different depths, and training the network through back propagation. The device comprises a processor and a memory. According to the method, local and global context information of the multi-modal image data is considered, and the image synthesis quality of the missing modal is improved.

Description

Technical Field

[0001] The present invention relates to the field of image synthesis, and in particular to a multimodal image synthesis method and device based on a recursive fusion conversion unit. Background Art

[0002] In real-world scenarios, people usually need as many image modalities as possible to accurately and completely describe real-world scenes. However, due to various factors such as cost, time, and safety, it is often difficult to obtain all modalities. A fundamental solution is to synthesize the required modalities through available multimodal data, fuse multimodal complementary information into a single image, improve image quality, and make it more useful in subsequent applications such as image analysis, segmentation, object detection and recognition.

[0003] Various efforts have been made in multimodal image synthesis. Traditional methods usually construct a nonlinear mapping from the source image to the target image. However, the performance of these methods is limited by manually extracted or selected features. With the rapid development of deep learning, many studies have introduced convolutional neural networks (CNN) into image synthesis tasks. Through carefully designed loss functions and network structures, convolutional neural networks have made significant progress over traditional methods. Later, generative adversarial networks (GAN) were proposed and can be used for implicit image synthesis. GAN relies on the adversarial game between the generator and the discriminator to estimate the probability distribution of the target, thereby making progress in synthesis performance.

[0004] Despite the significant performance improvements, most current CNN-based synthesis techniques have inherent limitations: (i) The convolution operation mainly focuses on learning the relationship between pixels in a relatively small neighborhood, which makes it good at extracting local features but poor at capturing global concepts. In order to ensure that the features have sufficient spatial resolution for accurate reconstruction, the network depth of image synthesis models is relatively limited. This greatly limits the ability of image synthesis models to extract contextual information that reflects global spatial dependencies. (ii) In multimodal image synthesis tasks, it is necessary not only to capture the global spatial context within the modality, but also to summarize the global complementary information between modalities. However, these CNN-based methods have difficulty capturing the cross-modal dependencies of inter-modal features, resulting in weak fusion performance of all inter-modal global complementary information.

[0005] Thanks to the global self-attention mechanism, Vision Transformer (ViT) has an advantage over CNN in global representation learning. By capturing the global context within and between modalities, ViT can provide very promising synthesis results. However, this also brings new challenges. First, ViT divides the input image into different fragments, encodes it into a one-dimensional sequence using position encoding, and calculates the output of each position by weighted summing all fragments. This may lead to the degradation of local feature details. Since the multimodal image synthesis task requires global context information while retaining local details, it is urgent to explore a multimodal image synthesis method that can take into account both local and global features, and can make full use of the information within and between modalities to improve the synthesis quality. Summary of the invention

[0006] The present invention utilizes the sensitivity of convolutional neural networks to local image features and the learning ability of Vision Transformer to long-distance dependencies, and designs a multimodal image synthesis method and device based on a recursive fusion transformation unit, so that the model takes into account both local and global context information of multimodal image data, and improves the image synthesis quality of the missing modality, as described below:

[0007] In a first aspect, a multimodal image synthesis method based on a recursive fusion transformation unit comprises:

[0008] The fusion conversion unit learns local image features through the local representation block, selects effective features from the local image features through the feature conversion gate, and converts the effective features to the Transformer feature space. The effective features are fused with the old global features through the global fusion block to infer the new global features. The fusion conversion unit adopts the recursive expansion reasoning form to extract local and global features of different depths.

[0009] The multi-branch decoder uses the local features of the intermediate depth of each branch to synthesize the missing modality image and calculates the loss to guide the optimization; the local and global features of the final stage are used to synthesize multiple missing modality reconstructed images; the hybrid gating module dynamically fuses multiple missing modality reconstructed images in the fused image space;

[0010] The fusion conversion unit, multi-branch decoder and hybrid gating module are connected in series to form a generator, and the generator and the discriminator form a generative adversarial network framework;

[0011] The adversarial loss and reconstruction loss are calculated for images reconstructed using features from different branches and different depths, and the network is trained by back propagation.

[0012] Among them, the local representation block:

[0013] LRB=Conv1(ReLU(ResBlock(x)))

[0014] Local feature map C output by LRB i-1 It is sent to the 1×1 convolutional layer and the gated linear unit to filter out effective features and convert them into the Transformer feature space.

[0015] The recursive expansion is as follows: the output of the previous recursive stage is used as the input of the next recursive stage, and multiple inferences are performed in the same module to train the module parameters.

[0016] Wherein, the multi-branch decoder is:

[0017] The missing modality image is reconstructed using the multi-branch features of the input modality provided by the fusion conversion unit, the missing image is reconstructed using the fusion branch global features, and the intermediate result is reconstructed using the local features in the intermediate recursive reasoning stage.

[0018] The hybrid gating module calculates adaptive fusion weights for the multi-branch synthetic image and obtains the final missing modality fusion result by weighting.

[0019] In a second aspect, a multimodal image synthesis device based on a recursive fusion transformation unit comprises: a processor and a memory, wherein the memory stores program instructions, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the methods described in the first aspect.

[0020] In a third aspect, a computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes any one of the methods described in the first aspect.

[0021] The beneficial effects of the technical solution provided by the present invention are:

[0022] 1. This paper uses the sensitivity of convolutional neural networks to local image features and the ability of ViT to capture long-distance dependencies to design a multimodal image synthesis method based on a recursive fusion transformation unit, so that the model can achieve dynamic compatibility with local and global information during the training process, not only obtaining local image features, but also taking into account the long-distance dependencies between the local features within the modality and between the modalities. Finally, leading results are obtained in the missing modality synthesis task, and a high-quality multimodal image synthesis model is realized;

[0023] 2. The present invention designs a feature conversion method, which adaptively selects effective features in the previous recursive process during training through a gated linear unit, and automatically converts the effective features into the Transformer feature space so as to add them to the old global features of the previous recursive process to further extract dependencies;

[0024] 3. The present invention reduces the number of parameters of the entire model and the dependence on the amount of training data by recursively expanding the fusion conversion unit, and ensures the accuracy of the deep features of the network by adding constraints to each recursive stage, thereby enhancing the quality of the synthesized image with a small number of parameters;

[0025] 4. The present invention designs a method for synthesizing missing modal images based on a hybrid gating module. A hybrid gating module is constructed for specific modality branches and multimodal fusion branches. The weight map of each branch is adaptively calculated through convolution operations and element-level Softmax normalization. The feature information of the multimodal branches is fully utilized, and multiple synthesis results are flexibly mixed in the image space to obtain the final missing modality synthesis result. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flowchart of a multimodal image synthesis method based on a recursive fusion conversion unit proposed by the present invention;

[0027] Figure 2 It is a schematic diagram of the fusion conversion unit module of the present invention. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0029] In order to solve the technical problems existing in the multimodal image synthesis method, the embodiment of the present invention proposes a multimodal image synthesis method based on a recursive fusion conversion unit. The method preliminarily extracts the local features of the input image through a multi-branch encoder. The method fully extracts the local feature information of the image and the long-distance dependency between the modalities by forming a fusion conversion unit and combining the respective advantages of a convolutional neural network and a Transformer network. The method constrains the synthesis results of each recursive stage by designing a recursive expansion method, thereby greatly reducing the model parameters and improving the model performance without reducing the model performance. The method synthesizes the reconstruction results of multiple branches and multiple recursive stages through a multi-branch decoder to guide the parameter update direction of the recursive process. The method adaptively mixes the synthetic images of a specific modality and a fusion modality in the image space by designing a hybrid gating module, thereby making full use of the different information of the multimodal image and improving the quality of the missing modality synthesis. The method forms a generator by connecting a fusion conversion unit and a hybrid gating module in series, and reaches the balance point of the best performance in the architecture of the generative adversarial network to obtain a high-quality missing modality synthesis result.

[0030] like Figure 1 As shown in the figure, a multimodal image synthesis method based on recursive fusion transformation unit consists of six components, and the functions of each part are as follows:

[0031] like Figure 2 As shown, in a multi-branch encoder component, an embodiment of the present invention converts multimodal input data from an image space to a feature space.

[0032] Furthermore, the CNN-based multi-branch encoder Encoder is used for a given M input modality X i ,i∈{1,2,…,M}, perform reasoning respectively and preliminarily extract local features of each modality for subsequent reasoning.

[0033]

[0034] then The input is converted into the form required by Transformer through primitive embedding, which serves as the initial information for global fusion. The formal expression of primitive embedding is: given an input image of size H×W×C, it is first divided into N non-overlapping primitives of size P×P×C, each primitive is flattened into a one-dimensional vector, and then converted into an embedding vector of dimension D through a convolutional mapping. In order to retain position information, each primitive embedding will be added with a corresponding position encoding PE. The whole process can be expressed as: F0=Conv(flatten(C0′))+PE, where F0 is the initial fusion input of global fusion, which contains the feature information unique to each input modality, and C0′ represents the initial features from different modalities. Splicing:

[0035]

[0036] In the fusion conversion unit component, the embodiment of the present invention uses a local representation block to extract local features of a specific input modality, and uses a feature conversion gate to filter effective features and convert feature shapes, which are sent to the global fusion block to extract long-distance dependencies between different modalities.

[0037] Furthermore, the CNN-based local representation block specifically includes: Each local representation block (LRB) consists of a ResBlock with two 3×3 convolutional layers, an activation function ReLU and a 1×1 convolutional layer. The formal expression of ResBlock is:

[0038] X′=ReLU(BatchNorm(Conv3(X)))

[0039] X″=I)atchNorm(Conv3(X′))

[0040] ResBlock(X)=X″+X

[0041] Among them, X represents the original input of the network, Conv3 represents the 3×3 convolution layer, and BatchNorm is the batch normalization layer. It prevents the internal covariate shift of the distribution by using the small batch statistics for normalization, fixes the distribution range and introduces learnable parameters to maintain the unique statistical characteristics of the data set and prevent the loss of learned information. ResBlock alleviates the problem of gradient disappearance in deep networks through residual connections, allowing the network to effectively extract high-level features by increasing the depth. Then ResBlock and the 1×1 convolution layer (Conv1) form a local representation block in the following form:

[0042] LRB=Conv1(ReLU(ResBlock(x)))

[0043] Furthermore, the feature conversion gate operation specifically includes: the local feature map C output by LRB i-1 It is sent to the 1×1 convolutional layer and gated linear unit to filter out effective features and convert them into the Transformer feature space for subsequent global fusion. The method for filtering effective features is:

[0044] S i =C i-1 +GLU(Conv1(C i-1 ))

[0045] Among them, C i-1 is the output of LRB at the i-1th recursive stage, S i For the selected feature map, GLU is defined as:

[0046]

[0047] Among them, W1, W2, b1, b2 are all learnable parameters. is element multiplication, σ is the Sigmoid function. Given the CNN feature map in LRB (H, W, C are the height, width, and number of channels of the feature map) and the Transformer embedding in the global fusion block (L, D are the number of feature embeddings and embedding dimensions) are inconsistent. The feature conversion gate uses two layers of 3×3 convolutional layers to convert the features of the CNN space into the Transformer feature space, and uses batch normalization for regularization.

[0048] CT i =GELU(BatchNorm(Conv3(S i )))

[0049] Furthermore, the Transformer-based Global Fusion Block (GFB) specifically includes: Multi-Head Self-Attention (MHSA) and Multilayer Perceptron (MLP) form the Transformer layer, and multiple Transformer layers are stacked to form the GFB:

[0050] x′=MHSA(LayerNorm(x))+x

[0051] TransformerLayer=MLP(LayerNorm(x′))+x′

[0052] GFB=TransformerLayer2(TransformerLayer1(x))

[0053] Among them, x is the input of GFB, MLP consists of multiple layers of linear mapping and nonlinear activation functions, and LayerNorm stands for layer normalization. Its principle is consistent with BatchNorm, but it is no longer based on a batch, but on a single sample. The mean and variance of the features in each channel are counted to perform distribution normalization and learnable parameter training. Compared with BatchNorm, LayerNorm is not sensitive to batch size and is more suitable for normalization of variable-length sequences in Transformer.

[0054] MHSA uses multiple independent self-attention heads (SA). Each head calculates the weighted sum of all elements in the input sequence. The results of different heads are combined and the output of MHSA is obtained after linear mapping:

[0055] MHSA(Q,K,V)=Concat(SA1,SA2,SA3,...,SA h )W O

[0056]

[0057] in, denote the query, key, and value computed by learnable linear projection, respectively. k is the dimension of K, W O Represents a learnable linear mapping matrix.

[0058] In the recursive expansion component, the embodiment of the present invention uses the output of the previous recursive stage as the input of the next recursive stage, and performs multiple inferences in the same module, so that the module parameters can be trained more fully.

[0059] Furthermore, in order to propagate local features in the recursive stage, LRB transforms the output C of the previous stage into i-1 The output C0 of the encoder is concatenated as input. The local feature C1 depends on C0, while C2 depends on C0 and C1. Similarly, the output of LRB in the i-th stage is formally expressed as:

[0060] C i =LRB(concat(C i-1 ,C0)),i∈{1,2,...,R}

[0061] Furthermore, GFB needs to fuse the modal local feature information from different LRBs and also retain the long-distance dependencies extracted in the past recursive stage. i T output by the last recursive GFBi-1 Perform element-wise sum as input:

[0062] T i =GFB(T i- 1+ΣCT i )

[0063] In the multi-branch decoder component, an embodiment of the present invention utilizes the multi-branch features of the input modality provided by the fusion conversion unit to reconstruct the missing modality image, utilizes the fusion branch global features to reconstruct the missing image, and utilizes the local features in the intermediate recursive reasoning stage to reconstruct the intermediate result.

[0064] Furthermore, the multi-branch decoder uses the final stage local features extracted by the local representation block for each input modality branch Synthesizing missing modality images

[0065]

[0066] Furthermore, for an inference process with R recursive stages, the decoder will reconstruct the feature maps of the M modes in each stage based on the local features of the corresponding recursive stage, and obtain M×(R-1) intermediate results And calculate the loss with the label image of the corresponding modality To supervise the direction of parameter updates in the recursive process:

[0067]

[0068] Among them, y i is the real labeled image data of a specific modality. This will then be back-propagated together with the other losses.

[0069] Furthermore, the multi-branch decoder also performs image synthesis on the fusion branch based on the global feature. For the fusion branch after R inference stages, the final fusion feature T R Since the decoder is a CNN architecture, T R Need to be restored to after the element expansion The primitive expansion is similar to the primitive embedding. The N dimension is first restored to the H×W two-dimensional space by reshaping, and then through convolution mapping, the D-dimensional vector is mapped to the C-dimensional channel to obtain the converted fusion feature Used to decode and generate images:

[0070]

[0071] In the hybrid gating module component, the embodiment of the present invention calculates adaptive fusion weights for the multi-branch synthetic image, and obtains the final missing modality fusion result by weighting.

[0072] Furthermore, for the M+1 branch images synthesized by the decoder, that is, M With 1 The hybrid gating module adaptively calculates the weight map of each branch to fully utilize the information of all input modal branches and global fusion branches in the image space to obtain accurate synthesis results:

[0073]

[0074] Among them, i∈{1, 2, ..., M+1}, the Softmax function normalizes the M+1 weight maps of each pixel.

[0075] Furthermore, according to the weight map learned by the gating module, multiple synthetic images are aggregated to obtain the final target image:

[0076]

[0077] in, is element-wise multiplication.

[0078] In the adversarial generative network component, the aforementioned components together constitute the generator, which promotes each other's performance by competing with the discriminator, and calculates the adversarial loss to guide the optimization direction.

[0079] Furthermore, the discriminator D is composed of multiple layers of convolution blocks as the part that dynamically competes with the generator G in the adversarial generative network architecture and promotes each other. Each convolution block includes a 3D convolution layer, a nonlinear activation function ReLU, and InstancenNorm (Instance Normalization). The discriminator D finally outputs a binary classification result through the Sigmoid function. The adversarial loss guides the entire dynamic adversarial and optimization process of the generative adversarial network. The adversarial loss of the generator G is:

[0080]

[0081] The adversarial loss of the discriminator is:

[0082]

[0083] Among them, X t True label images representing the missing modality.

[0084] Furthermore, during the training process, the generator based on the fusion conversion unit and the hybrid gating module will fully learn the modal feature information contained in the multimodal input to synthesize realistic images. The discriminator senses and classifies the synthesized images to assist the generator in adjusting the parameter update direction during the gradient back propagation process, supplement the high-frequency details in the synthesized images, and update its own parameters to improve the ability to discriminate images.

[0085] Example 1

[0086] An embodiment of the present invention provides a multimodal image synthesis method based on a recursive fusion conversion unit, the method comprising the following steps:

[0087] 101: The images of each input modality are converted from the image space to the feature space through the multi-branch encoder to obtain the initial local image features. At the same time, the local features of each modality are converted to the Transformer feature space through mutual splicing and primitive embedding operations as the initial fusion features.

[0088] Specifically, primitive embedding divides the H×W×C input image into N non-overlapping primitives of size P×P×C, each primitive is flattened into a one-dimensional vector, and then the C channels are converted into an embedding vector of dimension D through convolution mapping. The converted features are added with the corresponding position encoding PE to retain the position information.

[0089] 102: The fusion conversion unit learns local image features through the local representation block; through the feature conversion gate, it screens the effective features and converts the effective features into the Transformer feature space; through the global fusion block, it combines the effective features from multiple input modalities and the old global features in the Transformer feature space, and infers the new deep global features;

[0090] Specifically, the feature conversion gate filters the features of the local representation block through a 1×1 convolution layer and a gated linear unit, retains valid information, and transforms the feature form through a 3×3 convolution layer and batch normalization. The global fusion block is consistent with the ViT structure, concatenating the local feature information from multiple modal branches with the fusion output of the previous stage as the input of this stage, and capturing dependencies through a multi-head self-attention mechanism.

[0091] 103: The fusion conversion unit adopts a recursive expansion reasoning form to gradually extract local image features and global fusion features at different depths without increasing the number of parameters;

[0092] Specifically, in the process of multi-stage recursive expansion, the local representation block uses the local feature information output in the previous stage as the input of this stage to extract deep features. The global fusion block uses the output of the local representation block in the previous stage, transformed by the feature conversion gate, and concatenated with the output of the global fusion block in the previous stage as input to capture the intra-modal and inter-modal dependencies of local features.

[0093] 104: The multi-branch decoder uses the local image features obtained by the local representation blocks of different modality branches at different recursive stages to synthesize the intermediate results of the missing modality. The global features extracted by the global fusion block of the fusion branch will also be synthesized into the missing modality image through the decoder after primitive expansion.

[0094] Specifically, image expansion first reshapes the N-dimensional global features of the shape of N×D into the two-dimensional space of H×W, and then restores the D dimension to C channels through convolution operation. In this way, the global fusion features are converted back to the CNN feature space.

[0095] 105: The hybrid gating module further utilizes the multi-branch decoder to synthesize the missing modality images in each branch, and flexibly and dynamically fuses the synthesis results of different modalities with the results of the fusion branch in the fusion image space.

[0096] Specifically, the M specific input modality features extracted by the local representation block will be passed through the decoder to synthesize the missing modality image, and the fusion features learned by the global fusion block will also be passed through the decoder to reconstruct the missing modality. The resulting M+1 synthetic images are normalized using Softmax after residual connection, and pixel-level weights are assigned to each image to flexibly fuse modality information from different branches.

[0097] 106: The fusion conversion unit and the hybrid gating module are connected in series to form a generator. The discriminator and the generator form an adversarial generative network framework. The discriminator takes the missing modality real image data and the generator synthetic data as input, and outputs different classification results for subsequent calculation of adversarial loss.

[0098] 107: Calculate the adversarial loss and reconstruction loss values ​​for the fused reconstructed images of different modal branches and different recursive depths, sum them up and perform back propagation to train the network.

[0099] The obtained fused reconstructed image can calculate the adversarial loss and reconstruction loss of the corresponding modality. The reconstruction loss is in the form of L1 loss, including: the loss between the intermediate result of the recursive process and the true label image, and the loss between the final fusion result and the true label image.

[0100] In summary, the embodiment of the present invention gradually generates high-quality missing modality images through six parts: a multi-branch encoder, a fusion conversion unit, a recursive expansion, a multi-branch decoder, and a hybrid gating module; wherein the sensitivity of the convolutional neural network to local image features and the learning ability of the Transformer for long-distance dependencies between graphics elements are utilized to fully extract multimodal image information; a recursive expansion reasoning form is utilized (compared with stacking and connecting multiple modules of the same structure in series, the number of parameters is smaller, and the data set using the same sample size can be trained more fully to obtain better results), thereby realizing a low-parameter, high-performance feature extraction module; a hybrid gating module is used to calculate weights in the image space, and synthetic image results from different modalities are flexibly integrated, and multiple modules are fully trained together in the adversarial generative network framework, thereby obtaining high-quality missing modality images that are more in line with human visual perception.

[0101] Example 2

[0102] The embodiment of the present invention provides a reasoning method for a recursive expansion fusion conversion unit. The fusion conversion unit combines the advantages of convolutional neural networks and Transformers, and has good feature extraction capabilities. However, Transformers often have a large number of parameters, and a corresponding large-scale data set is required to find the global optimal solution in training. In order to reduce the dependence of Transformer on a large amount of data, the embodiment of the present invention introduces a recursive expansion reasoning method, which reduces the amount of network parameters by reducing the cascaded Transformer blocks, and ensures the effectiveness of network parameter training by repeatedly expanding six times. The method uses the output of the last reasoning of the local representation block in the fusion conversion unit as the input of the next reasoning of the local representation block, and the global fusion block uses the conversion results of the last global fusion block and the last local representation block output as the input of the next global fusion block to complete the recursive reasoning. In order to ensure the effective transmission of local features in the recursive process, the local features extracted by each modal branch are reconstructed using a decoder at each recursion, and the loss is calculated with the real label image to update the network parameters.

[0103] As shown in Table 1, the embodiments of the present invention have been tested for design effectiveness in three task scenarios. Among them, w / oRU means that the reasoning method of recursive expansion is not adopted, but six identical fusion conversion units are cascaded for calculation; w / oLr means that in each recursive expansion reasoning, the loss supervision network parameter optimization is no longer calculated for the intermediate results of each modality. It can be seen from the four evaluation indicators in the table that the algorithm design of recursive expansion has achieved better results than cascading multiple fusion conversion units, and the number of parameters has been sharply reduced from 137.6M to 39.51M, which greatly reduces the demand for the number of training samples; and the constraints on the recursive intermediate results effectively guide the iteration direction of the network parameters in the recursive process, so that the synthesis has more realistic, natural and high-quality results.

[0104] Table 1

[0105]

[0106] Example 3

[0107] The scheme in Example 1 is further introduced below in combination with specific examples and calculation formulas, as described below for details:

[0108] 1. Data Preparation

[0109] The embodiment of the present invention can be extended to the missing modality image synthesis task of any scene. The embodiment of the present invention verifies the effectiveness of the method on two public brain disease datasets and one indoor scene dataset. These include the Brain Tumor Segmentation 2020 dataset (BraTS2020), the Ischemic Stroke Lesion Segmentation 2015 dataset (ISLES2015), and the indoor scene dataset (Taskonomy).

[0110] BraTS2020 is a public brain tumor segmentation dataset with good calibration and alignment. It includes 369 training samples and 125 test samples. Each sample contains four modalities: T1, T1Gd, T2, and T2-FLAIR, as well as corresponding segmentation annotations.

[0111] ISLES2015 is a public ischemic stroke lesion segmentation dataset. It includes: subacute ischemic stroke lesion segmentation (SISS) and acute stroke outcome (SPES). This example selects the SISS part and divides 45 samples into 28 training samples and 17 test samples, where each sample contains: T1, T2, T2-FLAIR, DWI four modality data.

[0112] Taskonomy is a public indoor scene dataset that helps verify the generalization of the method of the present invention. Each sample in the dataset has more than ten modalities, including: RGB images, texture images, depth images, edge images, etc. This embodiment selects five of these modalities for experiments, including: depth images, shadow images, normal images, texture images, and RGB color images. The training set includes 1000 samples and the test set includes 500 samples.

[0113] 2. Network structure of multimodal image synthesis method based on recursive fusion conversion unit

[0114] The network structure of the multimodal image synthesis method based on the recursive fusion conversion unit in the embodiment of the present invention is as follows: Figure 1 As shown in the figure, a multimodal image synthesis method based on recursive fusion transformation unit consists of six components, and the functions of each part are as follows:

[0115] like Figure 2 As shown, in a multi-branch encoder component, an embodiment of the present invention converts multimodal input data from an image space to a feature space.

[0116] Furthermore, the CNN-based multi-branch encoder Encoder is used for a given M input modality X i ,i∈{1,2,…,M}, perform reasoning respectively and preliminarily extract local features of each modality for subsequent reasoning.

[0117]

[0118] then The input is converted into the form required by Transformer through primitive embedding, which serves as the initial information for global fusion. The formal expression of primitive embedding is: given an input image of size H×W×C, it is first divided into N non-overlapping primitives of size P×P×C, each primitive is flattened into a one-dimensional vector, and then converted into an embedding vector of dimension D through a convolutional mapping. In order to retain position information, each primitive embedding will be added with a corresponding position encoding PE. The whole process can be expressed as: F0=Conv(flatten(C0'))+PE, where F0 is the initial fusion input of global fusion, which contains the feature information unique to each input modality, and C0' represents the initial features from different modalities. Splicing:

[0119]

[0120] In the fusion conversion unit component, the embodiment of the present invention uses a local representation block to extract local features of a specific input modality, and uses a feature conversion gate to filter effective features and convert feature shapes, which are sent to the global fusion block to extract long-distance dependencies between different modalities.

[0121] Furthermore, the CNN-based local representation block specifically includes: Each local representation block (LRB) consists of a ResBlock with two 3×3 convolutional layers, an activation function ReLU and a 1×1 convolutional layer. The formal expression of ResBlock is:

[0122] X′=ReLU(BatchNorm(Conv3(X)))

[0123] X″=BatchNorm(Conv3(X′))

[0124] ResBlock(X)=X″+X

[0125] Among them, X represents the original input of the network, Conv3 represents the 3×3 convolution layer, and BatchNorm is the batch normalization layer. It prevents the internal covariate shift of the distribution by using the small batch statistics for normalization, fixes the distribution range and introduces learnable parameters to maintain the unique statistical characteristics of the data set and prevent the loss of learned information. ResBlock alleviates the problem of gradient disappearance in deep networks through residual connections, allowing the network to effectively extract high-level features by increasing the depth. Then ResBlock and the 1×1 convolution layer (Conv1) form a local representation block in the following form:

[0126] LRB=Conv1(ReLU(ResBlock(x)))

[0127] Furthermore, the feature conversion gate operation specifically includes: the local feature map C output by LRB i-1 It is sent to the 1×1 convolutional layer and gated linear unit to filter out effective features and convert them into the Transformer feature space for subsequent global fusion. The method for filtering effective features is:

[0128] S i =C i-1 +GLU(Conv1(C i-1 ))

[0129] Among them, C i-1 is the output of LRB at the i-1th recursive stage, S iFor the selected feature map, GLU is defined as:

[0130]

[0131] Among them, W1, W2, b1, b2 are all learnable parameters. is element multiplication, σ is the Sigmoid function. Given the CNN feature map in LRB (H, W, C are the height, width, and number of channels of the feature map) and the Transformer embedding in the global fusion block (L, D are the number of feature embeddings and embedding dimensions) are inconsistent. The feature conversion gate uses two layers of 3×3 convolutional layers to convert the features of the CNN space into the Transformer feature space, and uses batch normalization for regularization.

[0132] CT i =GELU(BatchNorm(Conv3(S i )))

[0133] Furthermore, the Transformer-based Global Fusion Block (GFB) specifically includes: Multi-Head Self-Attention (MHSA) and Multilayer Perceptron (MLP) form the Transformer layer, and multiple Transformer layers are stacked to form the GFB:

[0134] x′=MHSA(LayerNorm(x))+x

[0135] TransformerLayer=MLP(LayerNorm(x′))+x′

[0136] GFB=TransformerLayer2(TransformerLayer1(x))

[0137] Among them, x is the input of GFB, MLP consists of multiple layers of linear mapping and nonlinear activation functions, and LayerNorm stands for layer normalization. Its principle is consistent with BatchNorm, but it is no longer based on a batch, but on a single sample. It counts the mean and variance of the features in each channel, performs distribution normalization, and trains learnable parameters. Compared with BatchNorm, LayerNorm is not sensitive to batch size and is more suitable for normalization of variable-length sequences in Transformer.

[0138] MHSA uses multiple independent self-attention heads (SA). Each head calculates the weighted sum of all elements in the input sequence. The results of different heads are combined and the output of MHSA is obtained after linear mapping:

[0139] MHSA(Q,K,V)=Concat(SA1,SA2,SA3,...,SA h )W O

[0140]

[0141] in, denote the query, key, and value computed by learnable linear projection, respectively. k is the dimension of K, W O Represents a learnable linear mapping matrix.

[0142] In the recursive expansion component, the embodiment of the present invention uses the output of the previous recursive stage as the input of the next recursive stage, and performs multiple inferences in the same module, so that the module parameters can be trained more fully.

[0143] Furthermore, in order to propagate local features in the recursive stage, LRB transforms the output C of the previous stage into i-1 The output C0 of the encoder is concatenated as input. The local feature C1 depends on C0, while C2 depends on C0 and C1. Similarly, the output of LRB in the i-th stage is formally expressed as:

[0144] C i =LRB(concat(C i-1 ,C0)),i∈{1,2,...,R}

[0145] Furthermore, GFB needs to fuse the modal local feature information from different LRBs and also retain the long-distance dependencies extracted in the past recursive stage. i T output by the last recursive GFB i-1 Perform element-wise sum as input:

[0146] T i =GFB(T i-1 +ΣCT i )

[0147] In the multi-branch decoder component, an embodiment of the present invention utilizes the multi-branch features of the input modality provided by the fusion conversion unit to reconstruct the missing modality image, utilizes the fusion branch global features to reconstruct the missing image, and utilizes the local features in the intermediate recursive reasoning stage to reconstruct the intermediate result.

[0148] Furthermore, the multi-branch decoder uses the final stage local features extracted by the local representation block for each input modality branch Synthesizing missing modality images

[0149]

[0150] Furthermore, for an inference process with R recursive stages, the decoder will reconstruct the feature maps of the M modes in each stage based on the local features of the corresponding recursive stage, and obtain M×(R-1) intermediate results And calculate the loss with the label image of the corresponding modality to supervise the parameter update direction in the recursive process:

[0151]

[0152] Among them, y i is the real labeled image data of a specific modality. This will then be back-propagated together with the other losses.

[0153] Furthermore, the multi-branch decoder also performs image synthesis on the fusion branch based on the global feature. For the fusion branch after R inference stages, the final fusion feature T R Since the decoder is a CNN architecture, T R Need to be restored to after the element expansion The primitive expansion is similar to the primitive embedding. The N dimension is first restored to the H×W two-dimensional space by reshaping, and then through convolution mapping, the D-dimensional vector is mapped to the C-dimensional channel to obtain the converted fusion feature Used to decode and generate images

[0154]

[0155] In the hybrid gating module component, the embodiment of the present invention calculates adaptive fusion weights for the multi-branch synthetic image, and obtains the final missing modality fusion result by weighting.

[0156] Furthermore, for the M+1 branch images synthesized by the decoder, that is, M With 1 The hybrid gating module adaptively calculates the weight map of each branch to fully utilize the information of all input modal branches and global fusion branches in the image space to obtain accurate synthesis results:

[0157]

[0158] Among them, i∈{1, 2, ..., M+1}, the Softmax function normalizes the M+1 weight maps of each pixel.

[0159] Furthermore, according to the weight map learned by the gating module, multiple synthetic images are aggregated to obtain the final target image:

[0160]

[0161] in, is element-wise multiplication.

[0162] In the adversarial generative network component, the aforementioned components together constitute the generator, which promotes each other's performance by competing with the discriminator, and calculates the adversarial loss to guide the optimization direction.

[0163] Furthermore, the discriminator D is composed of multiple layers of convolution blocks as the part that dynamically competes with the generator G in the adversarial generative network architecture and promotes each other. Each convolution block includes a 3D convolution layer, a nonlinear activation function ReLU, and InstancenNorm (Instance Normalization). The discriminator D finally outputs a binary classification result through the Sigmoid function. The adversarial loss guides the entire dynamic adversarial and optimization process of the generative adversarial network. The adversarial loss of the generator G is:

[0164]

[0165] The adversarial loss of the discriminator is:

[0166]

[0167] Among them, X t True label images representing the missing modality.

[0168] Furthermore, during the training process, the generator based on the fusion conversion unit and the hybrid gating module will fully learn the modal feature information contained in the multimodal input to synthesize realistic images. The discriminator senses and classifies the synthesized images to assist the generator in adjusting the parameter update direction during the gradient back propagation process, supplement the high-frequency details in the synthesized images, and update its own parameters to improve the ability to discriminate images.

[0169] III. Evaluation Metrics and Protocols

[0170] In order to accurately evaluate the quality of the synthesized image, the embodiment of the present invention adopts four common image quality evaluation indicators, including: Peak Signal-to-Noise Ratio (PSNR), Feature Similarity Index Measurement (FSIM), Structural Similarity Index Measurement (SSIM), and Normalized Mean Squared Error (NMSE).

[0171] Among them, PSNR for a given real image y and a synthetic image Defined as:

[0172]

[0173] Where N is the number of pixels in the image. PSNR can objectively reflect the degree of image distortion. The larger the value, the less distortion and the better the quality.

[0174] Among them, FSIM is given a real image y and a synthetic image Defined as:

[0175]

[0176] S L (x) = [S PC (x)] α [S GM (x)] β

[0177]

[0178] Where α and β are usually set to 1. FSIM combines phase consistency and gradient amplitude, so it still has good evaluation capabilities when the image has more severe noise or blur degradation. The larger the value, the better the synthesis quality.

[0179] Among them, SSIM is given a real image y and a synthetic image Defined as:

[0180]

[0181] In the formula is the image mean, is the image variance, is the covariance, c1c2 is a constant to prevent division by zero errors. The larger the SSIM value, the more similar the synthetic image is to the real image in structure.

[0182] Among them, NMSE for a given real image y and synthetic image Defined as:

[0183]

[0184] The smaller the NMSE value, the smaller the error between the synthesized image and the true result, and the better the quality.

[0185] 4. Details of model usage

[0186] 1. Data supplement: The embodiment of the present invention uses different modalities in multiple data sets as input to verify the synthesis performance of missing modalities in different scenarios, and is divided into two types of tasks: single-modal input scenarios and multi-modal input scenarios. Specifically, for single-modal input scenarios, the BraTS2020 dataset and the ISLES2015 dataset both use the FLAIR modality as input, corresponding to the samples of synthesized T1, T2, T1Gd, and DWI modalities, respectively; the T1 modality is used as input, corresponding to the samples of synthesized FLAIR modality. The Taskonomy dataset uses the Shade modality to synthesize RGB and the Normal modality to synthesize RGB. For multi-modal input scenarios, the BraTS2020 dataset and the ISLES2015 dataset both use T2+FLAIR to synthesize T1, T1+FLAIR to synthesize T2 and T1Gd or DWI, and T1+T2 to synthesize FLAIR. The Taskonomy dataset uses Shade+Texture to synthesize RGB, and Normal+Depth to synthesize RGB. All input samples are unified to 224×224.

[0187] 2. Model optimization: The present invention example uses the Adam optimizer, with parameters β1=0.5, β2=0.999, and the learning rate is set to 2.0×10-4. To ensure sufficient training, the network parameters will be trained for 600 rounds.

[0188] 3. Hyperparameter setting: The number of recursive expansion layers in the embodiment of the present invention is R=6, and the trade-off ratio λ in the loss function is 100.

[0189] 4. Loss Setting: The embodiment of the present invention includes anti-loss (L adv ), reconstruction loss (L1), and recursive loss (L r). Adversarial loss and reconstruction loss are common loss constraints in generative adversarial networks, which are responsible for providing high-frequency and low-frequency information to the generator and guiding the direction of parameter optimization. In addition to the above two common loss functions, the embodiment of the present invention also designs a recursive loss. In each recursive expansion process of the fusion conversion unit, the L1 difference between the intermediate result generated by the decoder and the real label image is calculated to guide the transmission direction of local features in the recursive process. The total loss function of the network is L total =L adv +L1+L r .

[0190] In order to solve the technical problems existing in the multimodal image synthesis method, an embodiment of the present invention proposes a multimodal image synthesis method based on a recursive fusion conversion unit. The method forms a fusion conversion unit by fusing a convolutional neural network and a Transformer network structure, and fully extracts the local feature information of the image and the long-distance dependency between the modalities. The method realizes the adaptive screening of effective features in different recursive depths and the feature conversion between different network structures by designing a feature conversion gate. The method constrains the synthesis results of each recursive stage by designing a recursive expansion method of the fusion conversion unit, thereby greatly reducing the model parameters and improving the model performance without reducing the model performance. The method adaptively mixes the synthetic images of a specific modality and a fusion modality in the image space by designing a hybrid gating module, thereby making full use of the different information of the multimodal image and improving the quality of the missing modality synthesis.

[0191] The embodiments of the present invention all have the following four key innovations:

[0192] 1. A multimodal image synthesis method based on recursive fusion transformation unit is proposed

[0193] Technical effect: This method uses the sensitivity of convolutional neural networks to local image features and ViT's ability to capture long-distance dependencies to design a multimodal image synthesis method based on a recursive fusion transformation unit, which enables the model to achieve dynamic compatibility with local and global information during the training process. It not only obtains local image features, but also takes into account the long-distance dependencies between local features within and between modalities. Ultimately, it achieves leading results in the missing modality synthesis task and realizes a high-quality multimodal image synthesis model.

[0194] 2. A feature conversion method is proposed

[0195] Technical effect: This feature conversion method uses gated linear units to adaptively select effective features in the previous recursive process during training, and automatically converts the features of the convolutional neural network into a form suitable for ViT to further extract dependencies;

[0196] 3. A recursive reasoning method is proposed

[0197] Technical effect: By recursively expanding the fusion conversion unit, the overall number of model parameters is reduced, reducing the dependence on the amount of training data. By adding constraints to each recursive stage, the accuracy of the deep features of the network is guaranteed, thereby enhancing the quality of the synthesized image with a small number of parameters.

[0198] 4. A method for synthesizing missing modality images based on a hybrid gating module is proposed

[0199] Technical effect: This method constructs a hybrid gating module for specific modal branches and multimodal fusion branches. Through convolution operations and element-level Softmax normalization, the weight map of each branch is adaptively calculated, the feature information of the multimodal branches is fully utilized, and multiple synthesis results are flexibly mixed in the image space to obtain the final missing modality synthesis result.

[0200] In summary, the embodiments of the present invention provide a multimodal image synthesis method based on a recursive fusion conversion unit, design a feature conversion method, and realize the conversion and transmission of features between different structural networks; by recursively expanding the fusion conversion unit, the overall parameter amount of the model is reduced, and the quality of the synthesized image is enhanced under the premise of a small number of parameters; a method for synthesizing missing modal images based on a hybrid gating module is designed, so that multiple synthesis results can be flexibly mixed in the image space to obtain the final missing modality synthesis result.

[0201] Example 4

[0202] The method proposed in the embodiment of the present invention is compared with multiple technical methods on two brain medical datasets and an indoor scene dataset for single-modal input and multi-modal input.

[0203] 1. Single-mode input comparison

[0204] For the single-modal input scenario, the experimental results are shown in Table 2: The embodiment of the present invention evaluates the image synthesis performance on four quantitative indicators. The multimodal image synthesis method based on the recursive fusion conversion unit is superior to other popular image synthesis methods, proving the method's utilization of local and global features of the image, as well as its superior performance in missing modality image synthesis. In addition, the excellent performance on the two medical datasets shows that the method has stable synthesis performance, and the indicators on the indoor scene data also prove that the method has good generalization performance and can be migrated to different scenarios to complete tasks.

[0205] Table 2

[0206]

[0207] 2. Multimodal Input Comparison

[0208] For the multimodal input scenario, the experimental results are shown in Table 3: The embodiment of the present invention uses four quantitative indicators to evaluate the image synthesis performance. The multimodal image synthesis method based on the recursive fusion conversion unit has achieved leading synthesis performance among the popular multimodal image synthesis methods, and the evaluation indicators on the three data sets have achieved the best scores, which once again proves that this method effectively utilizes the image structure and high and low frequency features, and can synthesize real and natural high-quality missing modalities.

[0209] Table 3

[0210]

[0211]

[0212] A multimodal image synthesis device based on a recursive fusion conversion unit, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to execute the following method steps in embodiment 1:

[0213] The fusion conversion unit learns local image features through the local representation block, selects effective features from the local image features through the feature conversion gate, and converts the effective features to the Transformer feature space. The effective features are fused with the old global features through the global fusion block to infer the new global features. The fusion conversion unit adopts the recursive expansion reasoning form to extract local and global features of different depths.

[0214] The multi-branch decoder uses the local features of the intermediate depth of each branch to synthesize the missing modality image and calculates the loss to guide the optimization; the local and global features of the final stage are used to synthesize multiple missing modality reconstructed images; the hybrid gating module dynamically fuses multiple missing modality reconstructed images in the fused image space;

[0215] The fusion conversion unit, multi-branch decoder and hybrid gating module are connected in series to form a generator, and the generator and the discriminator form a generative adversarial network framework;

[0216] The adversarial loss and reconstruction loss are calculated for images reconstructed using features from different branches and different depths, and the network is trained by back propagation.

[0217] Among them, the local represents the block:

[0218] LRB=Conv1(ReLU(ResBlock(x)))

[0219] Local feature map C output by LRB i-1It is sent to the 1×1 convolutional layer and the gated linear unit to filter out effective features and convert them into the Transformer feature space.

[0220] The recursive expansion is as follows: the output of the previous recursive stage is used as the input of the next recursive stage, and multiple inferences are performed in the same module to train the module parameters.

[0221] Among them, the multi-branch decoder is:

[0222] The missing modality image is reconstructed using the multi-branch features of the input modality provided by the fusion conversion unit, the missing image is reconstructed using the fusion branch global features, and the intermediate result is reconstructed using the local features in the intermediate recursive reasoning stage.

[0223] Among them, the hybrid gating module calculates the adaptive fusion weights for the multi-branch synthetic images, and obtains the final missing modality fusion result by weighting.

[0224] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0225] The execution subjects of the above-mentioned processor and memory can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution subjects and are selected according to the needs of actual applications.

[0226] The data signal is transmitted between the memory and the processor via a bus, which is not described in detail in the embodiment of the present invention.

[0227] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiment.

[0228] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0229] It should be pointed out here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0230] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.

[0231] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium or a semiconductor medium, etc.

[0232] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0233] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0234] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal image synthesis method based on a recursive fusion transformation unit, characterized in that: The method comprises: The fusion conversion unit learns local image features through the local representation block, selects effective features from the local image features through the feature conversion gate, and converts the effective features to the Transformer feature space. The effective features are fused with the old global features through the global fusion block to infer the new global features. The fusion conversion unit adopts the recursive expansion reasoning form to extract local and global features of different depths. The multi-branch decoder uses the local features of the intermediate depth of each branch to synthesize the missing modality image and calculates the loss to guide the optimization; the local and global features of the final stage are used to synthesize multiple missing modality reconstructed images; the hybrid gating module dynamically fuses multiple missing modality reconstructed images in the fused image space; The fusion conversion unit, multi-branch decoder and hybrid gating module are connected in series to form a generator, and the generator and the discriminator form a generative adversarial network framework; The adversarial loss and reconstruction loss are calculated for images reconstructed using features from different branches and different depths, and the network is trained by back propagation.

2. The multimodal image synthesis method based on a recursive fusion conversion unit according to claim 1, characterized in that: The local representation block: LRB=Conv1(ReLU(ResBlock(x))) Local feature map C output by LRB i-1 It is sent to the 1×1 convolutional layer and the gated linear unit to filter out effective features and convert them into the Transformer feature space.

3. The multimodal image synthesis method based on a recursive fusion conversion unit according to claim 1, characterized in that: The recursive expansion is as follows: the output of the previous recursive stage is used as the input of the next recursive stage, and multiple inferences are performed in the same module to train the module parameters.

4. The multimodal image synthesis method based on a recursive fusion conversion unit according to claim 1, characterized in that: The multi-branch decoder is: The missing modality image is reconstructed using the multi-branch features of the input modality provided by the fusion conversion unit, the missing image is reconstructed using the fusion branch global features, and the intermediate result is reconstructed using the local features in the intermediate recursive reasoning stage.

5. The multimodal image synthesis method based on a recursive fusion conversion unit according to claim 1, characterized in that: The hybrid gating module calculates adaptive fusion weights for the multi-branch synthetic image and obtains the final missing modality fusion result by weighting.

6. A multimodal image synthesis device based on a recursive fusion conversion unit, characterized in that: The device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the methods of claims 1-5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 5.

Images