Wavelet transform enhanced hybrid fusion multi-modal mr composite ct system and method

Abstract

The application belongs to the technical field of medical image processing, and particularly relates to a wavelet transform enhanced hybrid fusion multi-modal MR synthesized CT system and method, which comprises an image preprocessing module, a hybrid fusion network training module and an image synthesis module. The hybrid fusion network training module comprises a hybrid fusion network, which comprises a T1 encoder, a T2 encoder, a hybrid fusion module and a decoder. The T1 encoder and the T2 encoder each comprise five residual convolution coding blocks connected in series. The hybrid fusion module comprises five hybrid fusion blocks, and five output features of different resolutions output by the five residual convolution coding blocks in the T1 encoder and the T2 encoder are respectively input into the five hybrid fusion blocks. The decoder comprises four decoding blocks connected in series and an output layer. The application can generate high-quality pseudo CT, has high synthesis precision, good structure detail reservation and high calculation efficiency, and is suitable for tumor radiotherapy, preoperative planning and the like.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of medical image processing, in particular to a wavelet transform enhanced hybrid fusion multi-modal MR synthesized CT system and method. BACKGROUND

[0002] Magnetic resonance (MR) and computed tomography (CT) are two widely used imaging techniques in radiotherapy treatment planning (RTP). MR images have excellent soft tissue contrast, which can provide clearer anatomical information for tumors, adjacent key structures and organs at risk (OARs), helping to accurately delineate tumor target regions and avoid key organs. CT images are widely used in radiotherapy dose calculation because they provide electron density information, and the core basis is the Hounsfield Units (HU) of the pixels in the image. In the actual radiotherapy process, in order to ensure the accuracy of dose calculation, CT and MR images usually need to be registered. However, the existing registration algorithms still have a spatial error of about 2-5 mm, which affects the accuracy of radiotherapy. At the same time, CT imaging relies on X-rays, which has a certain radiation risk, and the multi-modal image acquisition process is costly and time-consuming, and is prone to introduce motion artifacts. In view of the above problems, in recent years, a radiotherapy planning method using only MR images (MR-only) has been proposed, that is, a pseudo-CT image (Pseudo-CT) is synthesized from MR images to replace the real CT image for dose calculation. This method not only avoids multi-modal registration errors, but also significantly reduces radiation exposure and image acquisition costs, and has shown broad application prospects in radiotherapy clinics.

[0003] However, due to the difference in reflecting tissue physical properties between MR and CT, the MR synthesized CT task is highly challenging. Existing methods mainly include three categories based on tissue segmentation, atlas matching and deep learning. Among them, the deep learning method has become the current mainstream research direction due to its advantages such as no need for specific sequence input, ability to alleviate registration error influence, and automatic feature extraction. In clinical practice, multiple MR sequences such as T1-weighted, T2-weighted and T1-Flair are usually collected, each providing complementary anatomical and physiological information. However, most deep learning models currently use single sequence input or simple concatenation of multiple sequences, which makes it difficult to fully exploit and retain the specific information of each sequence. In addition, the bone tissue signal in MR images is weak, while the bone structure in CT images is high in intensity, and deep learning models often tend to learn low-frequency information, resulting in problems such as unclear bone structure expression, fuzzy edges and missing details in the synthesized CT images.

[0004] Therefore, it is urgent to provide an MR-CT synthesis system and method capable of effectively fusing complementary information of multi-sequence MR images while preserving feature differences of the multi-sequence MR images, improving the synthesis quality of bone tissue regions, maintaining edge definition and image details, and improving the accuracy and clinical usability of radiotherapy plans. SUMMARY

[0005] Based on this, the application provides a hybrid fusion multi-modal MR-CT synthesis system and method enhanced by wavelet transform to solve the technical defects of the prior art. The system and method can preserve specific information of multi-sequence MR images, utilize related information of the multi-sequence MR images, establish cross-modal correlation, improve the expression ability of bone tissue regions, and improve image edge blurring and structural loss, thereby preserving more high-quality details in the synthesized CT images and improving the overall synthesis effect and clinical applicability of the images.

[0006] In a first aspect, the embodiments of the application provide a hybrid fusion multi-modal MR-CT synthesis system enhanced by wavelet transform, comprising: an image preprocessing module, a hybrid fusion network training module, and an image synthesis module.

[0007] The hybrid fusion network training module comprises a hybrid fusion network, and the hybrid fusion network comprises a T1 encoder, a T2 encoder, a hybrid fusion module, and a decoder.

[0008] The T1 encoder and the T2 encoder have the same structure and each comprise five residual convolutional encoding blocks connected in series.

[0009] The hybrid fusion module comprises five hybrid fusion blocks corresponding to the five residual convolutional encoding blocks of the T1 encoder and the T2 encoder. The five output features of different resolutions output by the five residual convolutional encoding blocks of the T1 encoder and the T2 encoder are input into the corresponding hybrid fusion blocks.

[0010] The decoder comprises four decoding blocks connected in series and an output layer. The input feature of the first decoding block is the output feature of the fourth hybrid fusion block and the fifth hybrid fusion block, and the input features of the remaining decoding blocks are the output feature of the previous decoding block and the output feature of the hybrid fusion block of the same resolution. The output feature of the fourth decoding block is output through the output layer to obtain a synthesized CT image.

[0011] In a possible implementation, each residual convolutional encoding block comprises a residual convolutional block and a two-dimensional maximum pooling layer connected in series.

[0012] Each residual convolution block comprises a first residual connection and a first two-dimensional convolution layer, a first normalization layer, a first activation function layer, a first Dropout layer, a second two-dimensional convolution layer, a second normalization layer and a second activation function layer connected in sequence, one end of the first residual connection is connected to an input end of the first two-dimensional convolution layer, and the other end is connected to an output end of the second normalization layer, and the input feature of the residual convolution block and the output feature of the second normalization layer are added as the input feature of the second activation function layer;

[0013] The output feature of the residual convolution block is:

[0014]

[0015]

[0016]

[0017]

[0018]

[0019] wherein, X represents the input feature of the residual convolution block; Conv represents a two-dimensional convolution operation; BN represents a batch normalization operation; LeakyReLU represents a linear rectifier function with a leakage parameter, represents a negative half-axis slope of the linear rectifier function with a leakage parameter; represents a dropout operation.

[0020] In a possible implementation, the hybrid fusion block comprises a prior formula fusion module and a cross-attention fusion module connected in parallel, and the output features of the residual convolution encoding blocks in the T1 encoder and the residual convolution encoding blocks in the T2 encoder are input into the prior formula fusion module and the cross-attention fusion module respectively, and the output features of the prior formula fusion module and the cross-attention fusion module are spliced to obtain the output feature of the hybrid fusion block.

[0021] In a possible implementation, the prior formula fusion module performs element-wise addition, element-wise multiplication and element-wise maximum operation on the output features of the residual convolution encoding blocks in the T1 encoder and the residual convolution encoding blocks in the T2 encoder respectively, and splices three groups of results in the channel dimension; the output feature of the prior formula fusion module is:

[0022]

[0023] wherein, and respectively represent the output feature of the residual convolutional encoding block in the T1 encoder and the output feature of the residual convolutional encoding block in the T2 encoder; Concat represents a channel dimension splicing operation, and max represents an element-wise maximum value operation.

[0024] In a possible implementation, the cross-attention fusion module includes a T1 cross-attention fusion link and a T2 cross-attention fusion link in parallel, the T1 cross-attention fusion link and the T2 cross-attention fusion link have the same structure and each include a second residual connection and a series connection of an image block embedding module and a multi-head self-attention mechanism, and the second residual connection adds the input feature of the image block embedding module and the output feature of the multi-head self-attention mechanism to obtain the output feature of the cross-attention fusion link.

[0025] In a possible implementation, the image block embedding module is configured to output an image embedding feature , and a calculation formula of the image embedding feature is as follows:

[0026]

[0027]

[0028] wherein, represents the input feature of the image block embedding module; represents a division and flattening operation; represents an image embedding representation; represents a position embedding matrix.

[0029] In a possible implementation, the multi-head self-attention mechanism includes a shape reforming layer, a channel compression layer, a wavelet transform layer, a convolutional filtering layer, an inverse wavelet transform layer, and a multi-head self-attention module; the shape reforming layer, the channel compression layer, the wavelet transform layer, and the convolutional filtering layer are sequentially connected, the output of the convolutional filtering layer is divided into two branches and input into the multi-head self-attention module and the inverse wavelet transform layer respectively, and the output information of the multi-head self-attention module and the inverse wavelet transform layer is spliced and fused to obtain the output feature of the multi-head self-attention mechanism.

[0030] In a possible implementation, each decoding block includes a series connection of an up-sampling layer and a residual convolutional decoding block;

[0031] The residual convolutional decoding block includes a third residual connection and a fourth two-dimensional convolutional layer, a third normalization layer, a third activation function layer, a second Dropout layer, a fifth two-dimensional convolutional layer, a fourth normalization layer, and a fourth activation function layer connected in sequence; and an output feature of the residual convolutional decoding block is as follows:

[0032]

[0033]

[0034]

[0035]

[0036]

[0037] wherein, represents the input feature of the residual convolutional decoding block; Conv represents a two-dimensional convolution operation; BN represents a batch normalization operation; Dropout represents a dropout operation; ReLU is a linear rectification activation function, and the expression is as follows:

[0038] .

[0039] In a second aspect, the embodiments of the present application provide a hybrid fusion multi-modal MR synthesized CT method enhanced by wavelet transform, which is based on the hybrid fusion multi-modal MR synthesized CT system enhanced by wavelet transform in the first aspect, and comprises the following steps:

[0040] S1, preprocessing the obtained original MR T1 weighted image, MR T2 weighted image and CT image to obtain an MR T1 weighted image, an MR T2 weighted image and a labeled CT image;

[0041] S2, training the hybrid fusion network based on the MR T1 weighted image, the MR T2 weighted image and the labeled CT image to obtain a multi-modal MR synthesized CT model;

[0042] S3, inputting the actual MR T1 weighted image and MR T2 weighted image into the multi-modal MR synthesized CT model to obtain an actual synthesized CT image.

[0043] In a possible implementation, the step S2 is specifically as follows:

[0044] S21, inputting the MR T1 weighted image and the MR T2 weighted image into a T1 encoder and a T2 encoder respectively;

[0045] S22, the residual convolutional encoding blocks in the T1 encoder and the T2 encoder sequentially perform down-sampling on the input features, and output the obtained down-sampled features to a next residual convolutional encoding block and a hybrid fusion block with the same resolution respectively; the last residual convolutional encoding outputs the obtained down-sampled features to the last hybrid fusion block;

[0046] S23, each hybrid fusion block fuses the down-sampled features obtained by the residual convolutional encoding blocks of the T1 encoder and the T2 encoder with the same resolution to obtain fused features;

[0047] S24, the last fusion feature is upsampled by the first residual convolution decoding block, spliced with the second last fusion feature, and the spliced feature is decoded to obtain a decoded feature; subsequent residual convolution decoding blocks sequentially upsample the decoded feature output by the last residual convolution decoding block, splice the feature with the fusion feature of the same resolution, and decode the spliced feature to obtain a decoded feature; and the decoded feature output by the last decoding block is output to the output layer to obtain a synthetic CT image;

[0048] S25, based on the synthetic CT image and the label CT image, it is judged whether the termination condition is met, if the termination condition is met, the training is terminated, and the mixed fusion network corresponding to the optimal parameters in the training process is selected as the multi-modal MR synthetic CT model; otherwise, steps S21-S24 are repeated until the termination condition is met.

[0049] Compared with the prior art, the beneficial effects of the present application are:

[0050] (1) The present application proposes a hybrid fusion multi-modal MR synthetic CT network, which can effectively establish the cross-modal correlation between different modalities, while fully retaining the specificity information of each MR modality, thereby generating a high-precision CT image with complete structure and rich details.

[0051] (2) The present application designs a cross-attention fusion module, which performs cross-attention operation between two modalities, excavates potential complementary features, and strengthens the response to key structure areas through attention weight, thereby enhancing the fusion effect of multi-modal features.

[0052] (3) In order to reduce the computational complexity of the cross-attention mechanism, the present application introduces wavelet transform to optimize the original self-attention mechanism. By down-sampling the image embedding in the wavelet domain, the number of elements in the attention matrix is significantly reduced, and the computational burden is reduced. At the same time, due to the reversibility of wavelet transform, the feature information and spatial structure can still be preserved to the greatest extent.

[0053] (4) The present application adopts a multi-layer down-sampling convolution structure in the encoder part, so that the network can capture the multi-scale structural features of the image and enhance the modeling ability of different anatomical level information; in the decoder part, a symmetric multi-layer up-sampling structure is adopted to restore the spatial resolution layer by layer, effectively improving the clarity and edge fidelity of the generated image.

[0054] (5) In the loss function design aspect, the application constructs a hybrid loss function, which comprehensively considers pixel loss, adversarial loss and edge perception loss based on Laplacian pyramid. Among them, the pixel loss ensures the accuracy of gray scale restoration, the adversarial loss improves the image realism, and the edge perception loss guides the model to restore the bone structure and edge details from multiple frequency levels, effectively alleviating the problems of image blur and structure loss. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0056] Figure 1 The structure block diagram of the wavelet transform enhanced hybrid fusion multi-modal MR synthesized CT system provided by the embodiment of the present application is shown in the figure.

[0057] Figure 2 The structure diagram of the hybrid fusion network provided by the embodiment of the present application is shown in the figure, wherein F1 represents the output feature of the first hybrid fusion block, F2 represents the output feature of the second hybrid fusion block, F3 represents the output feature of the third hybrid fusion block, F4 represents the output feature of the fourth hybrid fusion block, and F5 represents the output feature of the fifth hybrid fusion block.

[0058] Figure 3 The structure diagram of the residual convolution block provided by the embodiment of the present application is shown in the figure.

[0059] Figure 4 The structure diagram of the hybrid fusion block provided by the embodiment of the present application is shown in the figure.

[0060] Figure 5 The structure diagram of the cross-attention fusion module provided by the embodiment of the present application is shown in the figure.

[0061] Figure 6 The structure diagram of the multi-head self-attention mechanism in the T1 cross-attention fusion link provided by the embodiment of the present application is shown in the figure.

[0062] Figure 7 The structure diagram of the residual convolution decoding block provided by the embodiment of the present application is shown in the figure.

[0063] Figure 8 The flowchart of the wavelet transform enhanced hybrid fusion multi-modal MR synthesized CT method provided by the embodiment of the present application is shown in the figure.

[0064] Figure 9A contrast schematic diagram of the MR T1 weighted image, the MR T2 weighted image, the labeled CT image and the synthesized CT image provided by the embodiment of the present application is shown in the figure, wherein MR TI represents the MR T1 weighted image, MR T2 represents the MR T2 weighted image, CT represents the labeled CT image, and sCT represents the synthesized CT image.

[0065] Figure 10 A structural schematic diagram of an electronic device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0066] In order to better understand the technical solutions of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0067] It should be clear that the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0068] The terms used in the embodiments of the present application are only for the purpose of describing the specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0069] It should be understood that the term "and / or" used herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0070] Referring to Figure 1 A structural block diagram of a wavelet transform enhanced hybrid fusion multi-modal MR synthesized CT system provided by the embodiment of the present application is shown in the figure. As Figure 1 shown, the system comprises an image preprocessing module 101, a hybrid fusion network training module 102 and an image synthesis module 103. Among them,

[0071] The image preprocessing module 101 is used for preprocessing the acquired original MR T1 weighted image, MR T2 weighted image and CT image to obtain the MR T1 weighted image, MR T2 weighted image and labeled CT image.

[0072] Specifically, the acquired original MR T1 weighted image and MR T2 weighted image are subjected to N4 bias field correction processing; the acquired CT image is subjected to threshold segmentation method to remove image artifacts; and all images are subjected to intensity normalization processing.

[0073] The hybrid fusion network training module 102 is configured to train the hybrid fusion network based on the MR T1 weighted image, the MR T2 weighted image and the label CT image, so as to obtain a multi-modal MR synthesized CT model.

[0074] The image synthesis module 103 is configured to input the actual MR T1 weighted image and the MR T2 weighted image into the multi-modal MR synthesized CT model, so as to obtain an actual synthesized CT image.

[0075] The hybrid fusion network training module comprises a hybrid fusion network and a loss evaluation module.

[0076] Referring to Figure 2 , a structure diagram of the hybrid fusion network provided by the embodiment of the present application is shown. As shown in Figure 2 , the hybrid fusion network comprises a T1 encoder, a T2 encoder, a hybrid fusion module and a decoder. Wherein,

[0077] The T1 encoder is configured to perform convolution feature extraction on the MR T1 weighted image, and obtain MR T1 multi-scale features at different resolutions through multi-layer down-sampling operations, and input the MR T1 multi-scale features into the hybrid fusion module.

[0078] The T2 encoder has the same structure as the T1 encoder, and is configured to perform convolution feature extraction on the MR T2 weighted image, and obtain MR T2 multi-scale features at different resolutions through multi-layer down-sampling operations, and input the MR T2 multi-scale features into the hybrid fusion module.

[0079] The hybrid fusion module is configured to receive the MR T1 multi-scale features and the MR T2 multi-scale features, calculate the cross-modal correlation between the two multi-scale features, and realize adaptive fusion of the features on the basis of preserving the modal specificity features, so as to obtain fusion features.

[0080] The decoder is configured to receive the fusion features, and generate a synthesized CT image through multi-layer convolution decoding operations.

[0081] The T1 encoder and the T2 encoder have the same structure, and each comprises five residual convolution encoding blocks connected in series, and each residual convolution encoding block comprises a residual convolution block and a two-dimensional maximum pooling layer connected in series.

[0082] Referring to Figure 3 , a structure diagram of the residual convolution block provided by the embodiment of the present application is shown. As shown in Figure 3As shown, each residual convolution block includes a first residual connection and sequentially connected first two-dimensional convolution layer, first normalization layer, first activation function layer (LeakyReLU), first Dropout layer, second two-dimensional convolution layer, second normalization layer and second activation function layer (LeakyReLU), one end of the first residual connection is connected to the input end of the first two-dimensional convolution layer, and the other end is connected to the output end of the second normalization layer, the input feature of the residual convolution block and the output feature of the second normalization layer are added as the input feature of the second activation function layer to enhance the feature transmission capability.

[0083] The output feature of the residual convolution block is:

[0084]

[0085]

[0086]

[0087]

[0088]

[0089] wherein, X represents the input feature of the residual convolution block; Conv represents a two-dimensional convolution operation; BN represents a batch normalization operation; LeakyReLU represents a linear rectifier function with a leakage parameter, represents the negative half-axis slope of the linear rectifier function with a leakage parameter, and in the embodiment the preferred value is 0.2; represents a dropout operation, which randomly discards some network neurons, and the dropout rate in the application is 0.3; represents the output feature of the residual convolution block.

[0090] Further, The calculation formula is:

[0091]

[0092] Further, the convolution kernel size of the first two-dimensional convolution layer is 3x3, the padding is 1, and the step is 1, which is used to double the channel number of the input feature.

[0093] Further, the convolution kernel size of the second two-dimensional convolution layer is 3x3, the padding is 1, and the step is 1, which is used to keep the channel number of the feature map unchanged.

[0094] The kernel size of the two-dimensional maximum pooling layer is 2x2, and the step is 2, which is used to downsample the output feature of the residual convolution block to obtain a low-resolution feature map of the next layer.

[0095] Referring to Figure 2 , the mixed fusion module includes five mixed fusion blocks corresponding to the five residual convolutional encoding blocks of the T1 encoder and the T2 encoder, and the output features of five different resolutions output by the five residual convolutional encoding blocks of the T1 encoder and the T2 encoder are respectively input into the corresponding mixed fusion blocks.

[0096] Referring to Figure 4 , a structural diagram of the mixed fusion block provided by the embodiment of the present application is shown in FIG. 3. As shown in FIG. 3, the mixed fusion block includes a prior formula fusion module and a cross-attention fusion module in parallel, and the output features of the T1 encoder and the T2 encoder are respectively input into the prior formula fusion module and the cross-attention fusion module, and the output features of the prior formula fusion module and the cross-attention fusion module are spliced to obtain the output features of the mixed fusion block. Figure 4

[0097] The prior formula fusion module is configured to fuse the output features of the T1 encoder and the T2 encoder at the element level, and specifically, the prior formula fusion module performs element-wise addition, element-wise multiplication and element-wise maximum value operation on the output features of the residual convolutional encoding blocks in the T1 encoder and the T2 encoder respectively, and splices the above three groups of results in the channel dimension. The output features of the prior formula fusion module are as follows:

[0098]

[0099] wherein, and represent the output features of the residual convolutional encoding blocks in the T1 encoder and the T2 encoder respectively; Concat represents the channel dimension splicing operation, and max represents the element-wise maximum value operation.

[0100] Referring to Figure 5 , a structural diagram of the cross-attention fusion module provided by the embodiment of the present application is shown in FIG. 4. As shown in FIG. 4, Figure 5

[0101] The cross-attention fusion module includes a T1 cross-attention fusion link and a T2 cross-attention fusion link in parallel, which respectively receive the output features of the residual convolutional encoding blocks in the T1 encoder and the T2 encoder, and the output features of the T1 cross-attention fusion link and the T2 cross-attention fusion link are spliced to obtain the output features of the cross-attention fusion module.

[0102] ​​​The T1 cross-attention fusion link and the T2 cross-attention fusion link have the same structure, and each includes a second residual connection and a serial connection of an image block embedding module and a multi-head self-attention mechanism, and the second residual connection adds the input feature of the image block embedding module and the output feature of the multi-head self-attention mechanism to obtain the output feature of the cross-attention fusion link.

[0103] The image block embedding module is configured to divide the input feature according to fixed-size image blocks, and flatten each image block to generate an embedding representation; and then add a position embedding code to the embedding representation of each image block to retain spatial position information to obtain an image embedding feature. .

[0104] Specifically, the image block embedding module divides and flattens the input feature according to fixed sizes to obtain an image embedding representation . The image embedding representation is added to a position embedding matrix of the same shape to generate an image embedding feature .

[0105] The input feature is divided and flattened according to fixed sizes specifically as follows: the input feature is divided according to image block sizes of , wherein represents the height of the image block, represents the width of the image block, h and w respectively represent the height and width of the input feature, H is the number of channels, the total number of image blocks obtained is W , and the embedding dimension of each image block after flattening is C . .

[0106] The calculation formula of the image embedding feature output by the image block embedding module is as follows: .

[0107]

[0108]

[0109] , wherein represents the input feature of the image block embedding module; represents the division and flattening operation; represents the image embedding representation; represents the position embedding matrix, which is initialized according to a normal distribution with a mean of 0 and a standard deviation of 0.02 in the embodiment, and is used to retain the position information of the image block.

[0110] Referring to​Figure 6 FIG. 1 is a structural diagram of a multi-head self-attention mechanism in a T1 cross-attention fusion link according to an embodiment of the present application. As shown in the figure, the multi-head self-attention mechanism includes a shape reforming layer, a channel compression layer, a wavelet transform layer, a convolution filtering layer, an inverse wavelet transform layer, and a multi-head self-attention module; the shape reforming layer, the channel compression layer, the wavelet transform layer, and the convolution filtering layer are connected in sequence, the output of the convolution filtering layer is divided into two branches, which are respectively input into the multi-head self-attention module and the inverse wavelet transform layer, and the output information of the multi-head self-attention module and the inverse wavelet transform layer is spliced and fused to obtain the output feature of the multi-head self-attention mechanism. Figure 6

[0111] Specifically, the shape reforming layer is configured to receive image embedding features output by an image block embedding module and adjust the image embedding features to adapt to the shape structure of wavelet transform. The channel compression layer is configured to perform channel compression on the image embedding features after shape reforming, so as to reduce the amount of calculation and retain key feature information. The wavelet transform layer is configured to perform wavelet transform on the compressed features, the wavelet transform adopts a Haar wavelet operator, and frequency domain features are extracted to realize decomposition of frequency domain information and retain edge structure information. The convolution filtering layer is configured to perform convolution operation on the frequency domain features after wavelet transform, so as to enhance local representation capability. The output features of the convolution filtering layer are divided into two paths: one branch of the output features generates keys and values after a linear transformation, and then inputs the keys and values into the multi-head self-attention module; the other branch of the output features inputs the output features into the inverse wavelet transform layer after wavelet transform, and the spatial domain features after recovery are used as residual information, which is spliced and fused with the output information of the multi-head self-attention module.

[0112] The output features of the multi-head self-attention mechanism in the T1 cross-attention fusion link and the output features of the multi-head self-attention mechanism in the T2 cross-attention fusion link are respectively as follows:

[0113]

[0114]

[0115] wherein, is a linear mapping calculation, is a channel splicing calculation, and denote the output features of the inverse wavelet transform layer and the multi-head self-attention module in the T1 cross-attention fusion link, respectively, and denote the output features of the inverse wavelet transform layer and the multi-head self-attention module in the T2 cross-attention fusion link, respectively.

[0116] ​​​​​The output features of the inverse wavelet transform layer in the T1 cross-attention fusion link and the T2 cross-attention fusion link and The calculation formula is:

[0117]

[0118]

[0119] wherein, and respectively represent the image embedding features output by the image block embedding module in the T1 cross-attention fusion link and the T2 cross-attention fusion link; represents a linear mapping calculation, multiplying the image embedding features by a matrix, which is a learnable parameter; represents a shape restoration calculation; Comp represents a channel compression calculation, which is a two-dimensional convolution calculation with a convolution kernel size of 1x1, a step size of 1, and padding of 0, compressing the channel number to one quarter of the original; represents a wavelet transform; Conv represents a two-dimensional convolution operation, which is a two-dimensional convolution calculation performed by a convolution filtering layer with a convolution kernel size of 3x3, a step size of 1, and padding of 1, to preliminarily enhance local features; represents an inverse wavelet transform.

[0120] The keys generated by the output features of the convolution filtering layer in the T1 cross-attention fusion link and the T2 cross-attention fusion link , values and keys , values are:

[0121]

[0122]

[0123] It should be noted that, and have exactly the same expression, and have exactly the same expression, so the calculation formulas of the two are combined in the above equations.

[0124] The multi-head self-attention module receives the keys and values generated by the output features of the convolution filtering layer in the fusion link it is located in and the query generated by the image embedding features output by the image block embedding module in the other fusion link, performs multi-head self-attention operation to realize effective information interaction and fusion between local and global features, thereby improving the precision and stability of cross-modal feature correlation modeling.

[0125] Specifically, the query , the key and the value all have shapes of , where denotes the total number of image patches, denotes the embedding dimension after flattening each image patch; and , , are split according to the number of attention heads in the multi-head self-attention module to obtain , , and , , each of which has a shape of . The self-attention is calculated for each group of .

[0126] The output features of the multi-head self-attention module in the T1 cross-attention fusion link and the T2 cross-attention fusion link are and , respectively.

[0127]

[0128]

[0129] where Concat denotes a channel dimension concatenation operation, denotes the output feature of the i-th head of the multi-head self-attention module in the T1 cross-attention fusion link, denotes the output feature of the i-th head of the multi-head self-attention module in the T2 cross-attention fusion link, , denotes the number of attention heads in the multi-head self-attention module; denotes a linear transformation matrix with a shape of , which is used to linearly transform the concatenation result of all attention heads. The above formula indicates that the output features of all attention heads are concatenated along the channel dimension and then multiplied by the linear transformation matrix

[0130] to obtain the output feature of the multi-head self-attention module .

[0131] and are calculated according to the following formula:

[0132] ​​

[0133]

[0134] where Softmax denotes a normalized weighting function, and denote the query of the th head, and , and denote the value of the th head, and , and denote the key of the th head, and transpose of the value of the th head, denotes a scaling factor.

[0135] The calculation formula of the query and

[0136]

[0137]

[0138] wherein, and denote the image embedding features output by the image block embedding module in the T1 cross-attention fusion link and the T2 cross-attention fusion link, respectively; is a linear mapping calculation, multiplying the image embedding features by a matrix, which is a learnable parameter.

[0139] On the basis of the above, refer to Figure 5 , the output features of the T1 cross-attention fusion link and the T2 cross-attention fusion link are respectively:

[0140]

[0141]

[0142] wherein, denotes the output feature of the T1 cross-attention fusion link, denotes the output feature of the T2 cross-attention fusion link, and denote the output feature of the residual convolutional encoding block in the T1 encoder and the output feature of the residual convolutional encoding block in the T2 encoder, respectively, and respectively represent the output features of the multi-head self-attention mechanism in the T1 cross-attention fusion link and the T2 cross-attention fusion link.

[0143] On the basis of the above, Figure 4 the output feature of the mixed fusion block is:

[0144]

[0145] wherein, the output feature of the T1 cross-attention fusion link, the output feature of the T2 cross-attention fusion link, the output feature of the prior formula fusion module, and Concat represents a channel dimension splicing operation.

[0146] On the basis of the above, Figure 2 the decoder comprises four decoding blocks and an output layer connected in sequence, the input feature of the first decoding block is the output feature of the fourth mixed fusion block and the fifth mixed fusion block, the input feature of the remaining decoding blocks is the output feature of the previous decoding block and the output feature of the mixed fusion block at the same resolution, and the output feature of the fourth decoding block is obtained through the output layer to obtain a synthetic CT image.

[0147] Each decoding block comprises a upsampling layer and a residual convolution decoding block connected in sequence.

[0148] The upsampling layer comprises a bilinear interpolation upsampling layer and a third two-dimensional convolution layer connected in sequence, the convolution kernel size of the third two-dimensional convolution layer is 1, the step is 1, and the padding is 0.

[0149] Referring to Figure 7 , a structural diagram of the residual convolution decoding block provided by the embodiment of the present application is shown. As shown in Figure 7 , the residual convolution decoding block comprises a third residual connection and a fourth two-dimensional convolution layer, a third normalization layer, a third activation function layer (ReLU layer), a second Dropout layer, a fifth two-dimensional convolution layer, a fourth normalization layer and a fourth activation function layer connected in sequence.

[0150] The convolution kernel size of the fourth two-dimensional convolution layer is 3, the step is 1, and the padding is 1, which is used to halve the channel number of the input feature; the third activation function layer adopts a ReLU function; the convolution kernel size of the fifth two-dimensional convolution layer is 3, the step is 1, and the padding is 1, which is used to keep the channel number unchanged; the fourth activation function layer adopts a ReLU function; the third residual connection is used to add the input feature of the fourth two-dimensional convolution layer and the output feature of the fourth normalization layer as the input feature of the fourth activation function layer.

[0151] the output feature of the residual convolution decoding block is:

[0152]

[0153]

[0154]

[0155]

[0156]

[0157] wherein, represents the input feature of the residual convolutional decoding block; Conv represents a two-dimensional convolution operation; BN represents a batch normalization operation; Dropout represents a dropout operation, which randomly discards some network neurons, and the dropout rate is 0.3; ReLU represents a linear rectification activation function, and the calculation formula is as follows:

[0158] .

[0159] The output layer is composed of a sixth two-dimensional convolutional layer and a fifth activation function layer in series, the convolution kernel size of the sixth two-dimensional convolutional layer is 1, the padding is 0, and the step is 0, and the fifth activation function layer uses a Tanh function.

[0160] Further, the hybrid fusion network training module further comprises a loss evaluation module, which is used for dynamically evaluating the performance in the training process of the hybrid fusion network. In the training process, the MR T1 weighted image, the MR T2 weighted image and the label CT image are proportionally divided into a training set and a validation set, the hybrid fusion network is supervised trained using the training set, in each training round, the loss function between the synthetic CT image generated by the hybrid fusion network and the corresponding label CT image is calculated on the training set, to drive the parameter update of the hybrid fusion network; at the same time, after each training round, the performance of the current hybrid fusion network is evaluated using the validation set, and the mean absolute error (MAE) thereof on the validation set is calculated; the system continuously monitors the change of the MAE index on the validation set, if the MAE on the validation set does not achieve obvious decline within continuous 50 training rounds, it is indicated that the model performance has no significant improvement, and it is determined that the model training has reached a stable state, the termination condition is met, and the training process is terminated; the hybrid fusion network parameters of the round corresponding to the minimum MAE index within the 50 training rounds are selected as the optimal parameters, and the hybrid fusion network corresponding to the optimal parameters is the multi-modal MR synthetic CT model; otherwise, the hybrid fusion network is continuously trained until the termination condition is met.

[0161] The application constructs a hybrid fusion multi-modal MR synthesized CT network to complete the task of MR synthesized CT. In the encoder part of the network, a multi-layer convolution downsampling structure is adopted, so that the network can extract multi-scale hierarchical feature information and enhance the perception ability of local and global structure of the image. Each layer of the encoding module includes convolution, normalization and activation function operations, which can gradually compress the spatial resolution of the input image and extract more abstract feature representation. In order to fully utilize the complementary structure and texture information in multi-modal MR images, the application introduces a hybrid fusion module, which combines two fusion strategies: prior formula fusion and cross attention fusion. The prior formula fusion fuses the features of different modalities at the element level, guiding the low-level complementarity of the features. The cross attention fusion establishes the attention relationship between different modalities at the image block level, learns the response weight of one modality to another modality, highlights the important information of the modality under the current semantic, effectively models the correlation between the cross modalities, avoids the redundancy and interference of the feature information, and retains the specificity of each modality. On the basis of the fused features, the decoder part adopts a multi-layer upsampling convolution structure, gradually restores the spatial resolution of the image, and restores the high-level abstract semantic to the pixel representation of the image level, so as to realize the generation of the final CT image. The network structure constructed by the application has end-to-end training ability, can fully mine the cross-modal structure information from multi-modal MR images, and effectively improves the structure fidelity and detail expressiveness of the synthesized CT image.

[0162] Embodiment 2

[0163] On the basis of the above embodiment, the present embodiment provides a hybrid fusion multi-modal MR synthesized CT method enhanced by wavelet transform, which is described with reference to Figure 8 The wavelet transform enhanced hybrid fusion multi-modal MR synthesized CT method provided by the embodiment of the application is shown in the flowchart. Figure 8 As shown in the figure, the method comprises the following steps:

[0164] Step S1, the obtained original MR T1 weighted image, MR T2 weighted image and CT image are preprocessed to obtain MR T1 weighted image, MR T2 weighted image and labeled CT image;

[0165] Step S2, training the hybrid fusion network based on the MR T1 weighted image, the MR T2 weighted image and the labeled CT image to obtain a multi-modal MR synthesized CT model;

[0166] Step S3, inputting the actual MR T1 weighted image and MR T2 weighted image into the multi-modal MR synthesized CT model to obtain an actual synthesized CT image.

[0167] Further, the step S2 is specifically:

[0168] Step S21, inputting the MR T1 weighted image and the MR T2 weighted image into a T1 encoder and a T2 encoder respectively;

[0169] Step S22, sequentially performing down-sampling on the input features by residual convolutional encoding blocks in the T1 encoder and the T2 encoder, and outputting the obtained down-sampled features to a next residual convolutional encoding block and a mixed fusion block with the same resolution respectively; the last residual convolutional encoding outputs the obtained down-sampled features to the last mixed fusion block;

[0170] Step S23, each mixed fusion block fuses the down-sampled features obtained by the residual convolutional encoding blocks of the T1 encoder and the T2 encoder with the same resolution to obtain fused features;

[0171] Step S24, performing up-sampling on the last fused feature by a first residual convolutional decoding block, splicing the up-sampled feature with the second last fused feature, and decoding the spliced feature to obtain a decoded feature; the subsequent residual convolutional decoding blocks sequentially perform up-sampling on the decoded features output by the previous residual convolutional decoding blocks, splice the up-sampled features with the fused features with the same resolution, and decode the spliced features to obtain decoded features; the decoded feature output by the last decoding block is output to an output layer to obtain a synthesized CT image;

[0172] Step S25, dynamically evaluating the performance of the mixed fusion network in the training process; specifically, in each training round, the loss function between the synthesized CT image generated by the mixed fusion network and the corresponding label CT image is calculated on the training set, which is used to drive the update of the model parameters; at the same time, after each training round, the performance of the current model is evaluated using the validation set, and the mean absolute error (MAE) on the validation set is calculated; this index is only used to evaluate the generalization ability of the model on the data not participating in the training, and is not used for model parameter update; the system continuously monitors the change of the MAE index on the validation set, if the MAE on the validation set does not achieve significant decline within 50 consecutive training rounds, it means that the performance of the model has no significant improvement, and the training of the model has reached a stable state, and the training process is terminated; at this time, the loss function has also converged and reached a relatively low level; saving the mixed fusion network parameters corresponding to the minimum MAE index as the optimal parameters, and the mixed fusion network corresponding to the optimal parameters is the multi-modal MR synthesized CT model; otherwise, repeat steps S21-S24 until the termination condition is met.

[0173] Further, the loss function described in the embodiment is a multi-component joint loss function, which comprehensively considers the pixel accuracy, structural consistency and perceptual authenticity of the image, and includes three parts of pixel loss, edge perception loss and adversarial loss, which are as follows:

[0174] First, pixel loss is used to measure the difference between the synthesized CT image and the labeled CT image at the pixel level. The pixel loss is calculated using the L1 loss function (mean absolute error), which is robust to outliers. The formula for calculating the pixel loss is as follows:

[0175]

[0176] in, Indicates pixel loss, This represents a composite CT image. This indicates a labeled CT image.

[0177] Furthermore, to enhance the edge detail representation capability of the synthesized CT images, this embodiment introduces an edge-aware loss based on the Laplacian pyramid structure. This edge-aware loss improves the model's ability to represent image structure and contours by extracting edge information from the image at multiple scales. For each layer of the Laplacian pyramid, the L1 error between the synthesized CT image and the labeled CT image at that scale is calculated, and then weighted and summed. The formula for calculating the edge-aware loss is as follows:

[0178]

[0179] in, This represents edge-sensing loss. The first of the Laplace pyramids i Edge images extracted from layers, Indicates the first i Weighting factors at the layer scale n This represents the number of levels in the Laplace Pyramid; in this embodiment, we take... n= 3.

[0180] Building upon the above, this embodiment further introduces adversarial loss, which employs a conditional generative adversarial network (GAN) design based on a pix2pix structure. The discriminator D uses a PatchGAN structure, dividing the image into multiple local regions for separate discrimination operations. The discrimination result of each patch is extracted through a convolutional network, outputting a probability map proportional to the input image size, used to determine whether each patch originates from a real image. The formula for calculating the adversarial loss is as follows:

[0181]

[0182] in, Indicating resistance to loss, Indicates to Seeking expectations, This represents a stitched image of the MR T1-weighted image and the MRT2-weighted image. Indicates discriminator D, The mixed fusion network is represented.

[0183] In summary, the loss function The calculation formula is:

[0184]

[0185] Wherein, , and respectively represent the weight coefficient of pixel loss, edge perception loss and adversarial loss, and in the embodiment, the values of , and are respectively set to 100, 50 and 1.

[0186] As can be seen from the above, in order to improve the generation quality and training stability of the multi-modal MR synthesized CT model, the application designs a hybrid loss function, which combines a variety of levels of supervision information to guide the model to learn from multiple scales from the pixel level to the structure level. The hybrid loss function includes: pixel loss, which uses L1 norm to calculate the error between the generated CT image and the real CT image at the pixel level, which helps to improve the overall gray scale consistency of the generated image; edge perception loss, based on the structure preserving loss function of the Laplacian Pyramid, the image is decomposed into multiple scale spatial frequency components, and the reconstruction error of different components is calculated respectively, so as to improve the fidelity of the image structure and edge, and prevent the network from producing blurred or structure distorted results in training; adversarial loss, introducing the discriminator module in the generative adversarial network architecture, by minimizing the difference between the generated image and the real image in the discriminator output, guiding the generated result to be more realistic and have high frequency details.

[0187] The loss function of the application can effectively improve the structure fidelity, edge clarity and overall perception quality of the synthesized CT image by using the joint optimization of multiple losses, thereby improving the reliability and practicality of the application in clinical application.

[0188] Figure 9 The contrast schematic diagram of the MR T1 weighted image, the MR T2 weighted image, the label CT image and the synthesized CT image provided by the embodiment of the application is shown in the figure, wherein MR TI represents the MR T1 weighted image, MR T2 represents the MR T2 weighted image, CT represents the label CT image, and sCT represents the synthesized CT image. As Figure 9As shown, it can be obviously seen that the synthetic CT image obtained by the system and method provided by the embodiments of the present disclosure is very similar to the label CT image. Meanwhile, the average of the three evaluation indexes of mean absolute error (MAE), structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) calculated on the test set can reach 27.12 (HU), 0.827 and 29.07 (dB), which proves the effectiveness of the system and method provided by the embodiments of the present disclosure.

[0189] Embodiment 3

[0190] Corresponding to the above-mentioned embodiments, the embodiments of the present application also provide an electronic device.

[0191] Referring to Figure 10 , a structural schematic diagram of an electronic device provided by the embodiments of the present application. As shown in the figure, Figure 10 The electronic device 1000 can include a processor 1001, a memory 1002 and a communication unit 1003. These components communicate through one or more buses, and those skilled in the art can understand that the electronic device structure shown in the figure does not constitute a limitation on the embodiments of the present application, which can be a bus type structure, or a star type structure, and can include more or fewer components than shown in the figure, or combine some components, or different component arrangements.

[0192] The communication unit 1003 is used to establish a communication channel, so that the electronic device can communicate with other devices.

[0193] The processor 1001 is the control center of the electronic device, which connects various parts of the electronic device through various interfaces and lines, and performs various functions of the electronic device and / or processes data by running or executing software programs and / or modules stored in the memory 1002 and calling data stored in the memory. The processor can be composed of integrated circuits (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs connected together. For example, the processor 1001 can only include a central processing unit (CPU). In the embodiments of the present application, the CPU can be a single operation core, or can include multiple operation cores.

[0194] The memory 1002 is configured to store the execution instructions of the processor 1001, and the memory 1002 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk or an optical disk.

[0195] When the execution instructions in the memory 1002 are executed by the processor 1001, the electronic device 1000 is enabled to perform part or all of the steps in the above method embodiments.

[0196] Corresponding to the above embodiments, the embodiments of the present application further provide a computer readable storage medium, wherein the computer readable storage medium can store a program, wherein the program can control the device where the computer readable storage medium is located to perform part or all of the steps in the above method embodiments when the program is running. In a specific implementation, the computer readable storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM) and the like.

[0197] Corresponding to the above embodiments, the embodiments of the present application further provide a computer program product, which contains executable instructions, when the executable instructions are executed on a computer, the computer executes part or all of the steps in the above method embodiments.

[0198] In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the cases of A alone, A and B together, and B alone. Wherein A and B can be singular or plural. The character " / " generally represents that the associated objects before and after it are in an "or" relationship. "At least one of the following" and the like expressions mean any combination of these items, including any combination of single or multiple items. For example, at least one of a, b and c can represent: a, b, c, a-b, a-c, b-c or a-b-c, wherein a, b and c can be single or multiple.

[0199] Those skilled in the art can clearly understand that the units and algorithm steps described in the embodiments disclosed herein can be realized by electronic hardware, computer software and a combination of the two. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0200] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0201] In several embodiments provided by the present application, any function realized in the form of a software function unit and sold or used as an independent product can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0202] The above description is merely a specific implementation of the present application. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered by the protection scope of the present application. The protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A hybrid fused multimodal MR synthesized CT system with wavelet transform enhancement, characterized by, include: Image preprocessing module, hybrid fusion network training module, image synthesis module; The hybrid fusion network training module includes a hybrid fusion network, which comprises a T1 encoder, a T2 encoder, a hybrid fusion module, and a decoder. The T1 encoder and the T2 encoder have the same structure, both consisting of five residual convolutional coding blocks connected in series. The hybrid fusion module includes five hybrid fusion blocks, which correspond to the five residual convolutional coding blocks of the T1 encoder and the T2 encoder, respectively. The five output features with different resolutions output by the five residual convolutional coding blocks of the T1 encoder and the T2 encoder are respectively input into the corresponding hybrid fusion blocks. The hybrid fusion block includes a priori formula fusion module and a cross-attention fusion module connected in parallel. The output features of the residual convolutional coding block in the T1 encoder and the residual convolutional coding block in the T2 encoder are respectively input into the priori formula fusion module and the cross-attention fusion module. The output features of the priori formula fusion module and the cross-attention fusion module are concatenated to obtain the output features of the hybrid fusion block. The prior formula fusion module performs element-by-element addition, element-by-element multiplication, and element-by-element maximum value operation on the output features of the residual convolutional encoding block in the T1 encoder and the residual convolutional encoding block in the T2 encoder respectively, and splices the three groups of results in the channel dimension; the output feature of the prior formula fusion module is: ​ ， wherein, and respectively represent the output feature of the residual convolutional encoding block in the T1 encoder and the output feature of the residual convolutional encoding block in the T2 encoder; Concat represents a channel dimension concatenation operation, and max represents an element-wise maximum operation. The cross-attention fusion module includes a parallel T1 cross-attention fusion link and a T2 cross-attention fusion link. The T1 cross-attention fusion link and the T2 cross-attention fusion link have the same structure, both including a second residual connection and a series image patch embedding module and a multi-head self-attention mechanism. The second residual connection adds the input features of the image patch embedding module and the output features of the multi-head self-attention mechanism to obtain the output features of the cross-attention fusion link. The image block embedding module is configured to output image embedding features , the image embedding features The calculation formula is: ， ， wherein, represents an input feature of the image patch embedding module; represents a partitioning and flattening operation; represents an image embedding representation; represents a positional embedding matrix; The multi-head self-attention mechanism includes a shape reshaping layer, a channel compression layer, a wavelet transform layer, a convolutional filtering layer, an inverse wavelet transform layer, and a multi-head self-attention module. The shape reshaping layer, channel compression layer, wavelet transform layer, and convolutional filtering layer are connected in sequence. The output of the convolutional filtering layer is divided into two branches, which are respectively input to the multi-head self-attention module and the inverse wavelet transform layer. The output information of the multi-head self-attention module and the inverse wavelet transform layer are spliced ​​and fused to obtain the output features of the multi-head self-attention mechanism. The decoder includes four decoding blocks and an output layer connected in series. The input features of the first decoding block are the output features of the fourth and fifth hybrid fusion blocks. The input features of the remaining decoding blocks are the output features of the previous decoding block and the output features of the hybrid fusion block at the same resolution. The output features of the fourth decoding block are processed by the output layer to obtain a synthesized CT image.

2. The hybrid fused multi-modal MR synthesized CT system with wavelet transform enhancement of claim 1, wherein, Each of the residual convolutional coding blocks comprises a residual convolutional block and a two-dimensional max pooling layer connected in series. Each residual convolutional block includes a first residual connection and a first two-dimensional convolutional layer, a first normalization layer, a first activation function layer, a first dropout layer, a second two-dimensional convolutional layer, a second normalization layer, and a second activation function layer connected in sequence. One end of the first residual connection is connected to the input of the first two-dimensional convolutional layer, and the other end is connected to the output of the second normalization layer. The input features of the residual convolutional block are added to the output features of the second normalization layer as the input features of the second activation function layer. Output features of a residual convolutional block is: ， ， ， ， ， wherein, X represents the input features of the residual convolutional block; Conv denotes a two-dimensional convolution operation; BN denotes a batch normalization operation; LeakyReLU denotes a linear rectifier function with a leaky parameter, denotes the slope of the negative half-axis of the linear rectifier function with a leaky parameter; denotes a dropout operation.

3. The hybrid fused multi-modal MR synthesized CT system with wavelet transform enhancement of claim 1, wherein, Each decoding block consists of a cascaded upsampling layer and a residual convolutional decoding block; The residual convolutional decoding block comprises a third residual connection and a fourth two-dimensional convolutional layer, a third normalization layer, a third activation function layer, a second Dropout layer, a fifth two-dimensional convolutional layer, a fourth normalization layer and a fourth activation function layer connected in sequence; the output feature of the residual convolutional decoding block is is: ， ， ， ， ， wherein, denotes the input feature of the residual convolutional decoding block; Conv denotes a two-dimensional convolution operation; BN denotes a batch normalization operation; Dropout represents a dropout operation; ReLU is a linear rectification activation function, and the expression is as follows: 。 4. A hybrid fused multimodal MR synthesized CT method with wavelet transform enhancement, characterized in that, The method is based on the wavelet transform enhanced hybrid fusion multi-modal MR synthesized CT system of claim 1, comprising: S1, pre-processing the obtained original MR T1 weighted image, MR T2 weighted image and CT image to obtain MR T1 weighted image, MR T2 weighted image and labeled CT image; S2, training the hybrid fusion network based on MR T1 weighted image, MR T2 weighted image and labeled CT image to obtain a multi-modal MR synthesized CT model; S3, inputting the actual MR T1 weighted image and MR T2 weighted image into the multi-modal MR synthesized CT model to obtain an actual synthesized CT image.

5. The hybrid fused multi-modal MR synthetic CT method with wavelet transform enhancement as claimed in claim 4, wherein, Step S2 is specifically: S21, inputting the MR T1 weighted image and the MR T2 weighted image into the T1 encoder and the T2 encoder respectively; S22, the residual convolutional coding blocks in the T1 encoder and the T2 encoder sequentially perform down-sampling on the input features, and output the obtained down-sampled features to the next residual convolutional coding block and the hybrid fusion block with the same resolution respectively; the last residual convolutional coding outputs the obtained down-sampled features to the last hybrid fusion block; S23, each hybrid fusion block fuses the down-sampled features obtained by the residual convolutional coding blocks of the T1 encoder and the T2 encoder with the same resolution to obtain fused features; S24, after the last fused feature is up-sampled by the first residual convolutional decoding block, it is spliced with the second last fused feature, and the spliced feature is decoded to obtain a decoded feature; the subsequent residual convolutional decoding blocks sequentially up-sample the decoded features output by the previous residual convolutional decoding blocks, splice them with the fused features with the same resolution, and decode the spliced features to obtain decoded features; The decoded feature output by the last decoding block is obtained by the output layer to obtain a synthesized CT image; S25, based on the synthesized CT image and the labeled CT image, it is judged whether the termination condition is met, if the termination condition is met, the training is terminated, and the hybrid fusion network corresponding to the optimal parameters in the training process is selected as the multi-modal MR synthesized CT model; Otherwise, repeat steps S21-S24 until the termination condition is met.

Images