Method, device and equipment for synthesizing CT image based on MR image and medium

Abstract

The application discloses a CT image synthesis method and device based on MR images, equipment and medium, the application obtains an image synthesis model by inputting a first MR image and a first CT image into a network model for training; a global block embedding module is used for global feature extraction on the first MR image, so that the global feature extraction result is more global; an encoding module is used for local feature extraction on the first MR image, an RSC self-attention module is used for first feature processing according to the global feature extraction result and the local feature extraction result, which can expand the receptive field of the features, a decoding module is used for decoding processing on the first feature processing result to generate a second CT image; a discriminator is used for generating a discrimination result according to the first CT image and the second CT image for training, and the finally obtained target CT image is more detailed and clear, and the application can be widely applied to the field of image processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a CT image synthesis method and device based on MR images, equipment and medium. BACKGROUND

[0002] The existing treatment of head and neck cancer and other malignant tumors can adopt a radiotherapy plan, in which a computed tomography (CT) of a patient's lesion is needed to be taken for radiotherapy dose planning and tumor positioning, which can be called a gold standard. At present, magnetic resonance imaging (MRI, referred to as MR) has been increasingly used as an auxiliary means of CT for radiotherapy planning due to its superior soft tissue contrast to CT, for segmenting tumor tissue and healthy organs. However, due to the simultaneous shooting of MR and CT images of the patient, it will bring a great economic burden to the patient and is low in efficiency, so the academic field has begun to explore the synthesis of corresponding CT images from MR images.

[0003] Traditional synthesis methods today are mainly based on feature matching, such as segmentation-based, image block-based, atlas-based, sparse coding-based models, but all have certain defects, for example, the segmentation-based method, due to the same CT value of each type of tissue, will cause the loss of blurred details of the image; the sparse coding-based model, due to the optimization of sparse coding at all image positions, the computational cost of the algorithm is often very high, so the traditional method often needs high data quality and large computational overhead, and the synthesized image has the problems of blurring and serious loss of details, which needs to be solved. SUMMARY

[0004] In view of this, in order to solve at least one of the above technical problems, the purpose of the present application is to provide a CT image synthesis method and device based on MR images, equipment and medium.

[0005] The technical solution adopted by the embodiments of the present application is:

[0006] The CT image synthesis method based on MR images comprises:

[0007] Obtaining training data; the training data comprises a first MR image and a first CT image of the same target object;

[0008] The first MR image and the first CT image are input into a network model for training to obtain an image synthesis model; the network model comprises a CST generator and a discriminator, the CST generator comprises an encoding module, a decoding module, an RSC self-attention module and a global block embedding module; the global block embedding module is used for global feature extraction on the first MR image, the encoding module is used for local feature extraction on the first MR image, the RSC self-attention module is used for first feature processing according to the global feature extraction result and the local feature extraction result, and the decoding module is used for decoding processing on the first feature processing result to generate a second CT image; the discriminator is used for generating a discrimination result according to the first CT image and the second CT image for training.

[0009] The MR image to be synthesized is input into the image synthesis model to obtain a target CT image.

[0010] Further, the training data comprises:

[0011] An original MR image and an original CT image of a same target object are obtained;

[0012] The original MR image is subjected to an offset field operation and then registered with the original CT image to obtain a registered MR image and a registered CT image;

[0013] The registered CT image is subjected to a preset range assignment processing;

[0014] The registered MR image and the registered CT image subjected to the assignment processing are subjected to normalization processing;

[0015] An image of a preset size is cut from the registered MR image subjected to the normalization processing to obtain a first MR image, and an image of a preset size is cut from the registered CT image subjected to the normalization processing to obtain a first CT image.

[0016] Further, the first MR image and the first CT image are input into a network model for training to obtain an image synthesis model, comprising:

[0017] The first MR image is subjected to local feature extraction by the encoding module;

[0018] The first MR image is subjected to global feature extraction by the global block embedding module;

[0019] The local feature extraction result and the global feature extraction result are fused, and the fusion result is subjected to first feature processing by the RSC self-attention module;

[0020] The first feature processing result is subjected to decoding processing by the decoding module to generate a second CT image;

[0021] inputting the first CT image and the second CT image into the discriminator to obtain a discrimination result;

[0022] training a network model according to the discrimination result and a preset loss function to obtain an image synthesis model.

[0023] Further, the encoding module comprises at least a first CNN module, a first RSC self-attention module and a second CNN module; the local feature extraction of the first MR image by the encoding module comprises:

[0024] performing at least one first feature extraction on the first MR image by the first CNN module to obtain first feature information;

[0025] performing second feature processing on the first feature information by the first RSC self-attention module;

[0026] performing second feature extraction on the second feature processing result by the second CNN module to obtain a local feature extraction result.

[0027] Further, the global block embedding module comprises a block embedding module and a second RSC self-attention module; the global feature extraction of the first MR image by the global block embedding module comprises:

[0028] performing second feature extraction on the first MR image by the block embedding module to obtain second feature information;

[0029] performing third feature processing on the second feature information by the second RSC self-attention module to obtain a global feature extraction result.

[0030] Further, the first feature processing comprises at least one feature sub-processing, and the RSC self-attention module comprises at least one third RSC self-attention module; the first feature processing of the fusion result by the RSC self-attention module comprises:

[0031] performing the feature sub-processing on the fusion result by the third RSC self-attention module, and the feature sub-processing specifically comprises:

[0032] performing first separable convolution on the fusion result, performing first dilated convolution on the first separable convolution result, performing second separable convolution on the first dilated convolution result, and performing second dilated convolution on the second separable convolution result;

[0033] performing first layer normalization according to the second dilated convolution result, and performing window multi-head attention on the first layer normalization result;

[0034] performing first addition on the window multi-head attention result and the second dilated convolution result, performing second layer normalization on the first addition result, and performing processing of the first multi-layer perception on the second layer normalization result;

[0035] performing second addition on the processing result of the first multi-layer perception and the first addition result, performing third layer normalization on the second addition result, and performing moving window multi-head attention on the third layer normalization result;

[0036] performing third addition on the moving window multi-head attention result and the second addition result, performing fourth layer normalization on the third addition result, and performing processing of the second multi-layer perception on the fourth layer normalization result;

[0037] performing fourth addition on the processing result of the second multi-layer perception, the third addition result and the second dilated convolution result to obtain a feature sub-processing result.

[0038] Further, the training of the network model according to the discrimination result and the preset loss function to obtain the image synthesis model comprises:

[0039] determining an adversarial loss value of the discriminator according to the discrimination result and the adversarial loss function;

[0040] determining a mean absolute error value according to the first CT image, the second CT image and the mean absolute error loss function;

[0041] determining a perceptual loss value according to the first CT image, the second CT image and the perceptual loss function;

[0042] adjusting network parameters of the network model according to the adversarial loss value, the mean absolute error value and the perceptual loss value to obtain the image synthesis model.

[0043] The embodiment of the present application further provides a CT image synthesis device based on MR images, comprising:

[0044] an acquisition module configured to acquire training data; the training data comprising a first MR image and a first CT image of a same target object;

[0045] The training module is configured to input the first MR image and the first CT image into a network model to train the network model to obtain an image synthesis model; the network model comprises a CST generator and a discriminator, the CST generator comprises an encoding module, a decoding module, an RSC self-attention module, and a global block embedding module; the global block embedding module is configured to perform global feature extraction on the first MR image, the encoding module is configured to perform local feature extraction on the first MR image, the RSC self-attention module is configured to perform first feature processing according to the global feature extraction result and the local feature extraction result, and the decoding module is configured to perform decoding processing on the first feature processing result to generate a second CT image; the discriminator is configured to generate a discrimination result according to the first CT image and the second CT image to train the network model.

[0046] The synthesis module is configured to input a to-be-synthesized MR image into the image synthesis model to obtain a target CT image.

[0047] The electronic device comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the method.

[0048] The computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by a processor to implement the method.

[0049] The method comprises the following steps: obtaining training data, the training data comprising a first MR image and a first CT image of a same target object; inputting the first MR image and the first CT image into a network model to train the network model to obtain an image synthesis model; the network model comprising a CST generator and a discriminator, the CST generator comprising an encoding module, a decoding module, an RSC self-attention module, and a global block embedding module; the global block embedding module being configured to perform global feature extraction on the first MR image, the encoding module being configured to perform local feature extraction on the first MR image, the RSC self-attention module being configured to perform first feature processing according to the global feature extraction result and the local feature extraction result, the decoding module being configured to perform decoding processing on the first feature processing result to generate a second CT image; the discriminator being configured to generate a discrimination result according to the first CT image and the second CT image to train the network model; inputting a to-be-synthesized MR image into the image synthesis model to obtain a target CT image. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 The step flowchart diagram of the CT image synthesis method based on MR image of the present application;

[0051] Figure 2 The structure diagram of the network model of the specific embodiment of the present application;

[0052] Figure 3 The structure diagram of the third RSC self-attention module of the specific embodiment of the present application;

[0053] Figure 4 The diagram of generating the discrimination result of the specific embodiment of the present application;

[0054] Fig. 5(a) is a diagram of the MR image to be synthesized in the specific embodiment, Fig. 5(b) is a diagram of the real CT corresponding to the MR image to be synthesized in the specific embodiment, Fig. 5(c) is a diagram of the target CT image in the specific embodiment, and Fig. 5(d) is a diagram of the output difference. DETAILED DESCRIPTION

[0055] In order to enable persons skilled in the art to better understand the present application scheme, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should be within the scope of protection of the present application.

[0056] The terms "first", "second", "third", and "fourth" and the like in the specification and claims of the present application and the drawings are used to distinguish different objects, and are not used to describe a particular order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0057] In this document, the term "embodiment" means that the specific features, structures or characteristics described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor is it independent or alternative to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0058] Explanation of terms related to the embodiments of the present application:

[0059] Transformer: A neural network based on self-attention mechanism, capable of capturing global context information, is one of the main network frameworks in deep learning.

[0060] Swin transformer: A neural network based on hierarchical local window self-attention mechanism.

[0061] Convolutional neural network: CNN, a kind of feedforward neural network, is widely used in image processing and speech recognition, and is one of the main network frameworks in deep learning.

[0062] Generative adversarial network: GAN for short, learns through the method of mutual game between two neural networks (generator and discriminator), and its output needs to imitate the real samples in the training set as much as possible.

[0063] Patch embedding: Block embedding, similar to word embedding in natural language processing, divides the image into image blocks, and embeds a vector for each image block. The length of the vector represents the encoding length of the image.

[0064] Registration: In the medical field, it refers to using a spatial change (or a series of spatial changes) to make the corresponding points of one medical image consistent with those of another medical image in space. This consistency means that the same anatomical point on the human body has the same spatial position in the two matching images.

[0065] Instance normalization: Also known as contrast normalization, normalization is used to change the values of numerical columns in the data set to the same scale when the features in the data have different ranges, which can speed up optimization. The implementation object of instance normalization is the entire batch of images, not a single image.

[0066] Context information: Generally understood as being aware of and able to apply some or all information that can affect the objects in the scene and image, in the image field it is mainly understood as the relationship between pixels and surrounding pixels.

[0067] Residual connection: Also known as skip connection, is the core method in residual neural network to alleviate the phenomenon of gradient vanishing and gradient explosion in deep neural network. Its output is the linear superposition of input and nonlinear change of input.

[0068] MLP: Multi-layer perceptron, a kind of feedforward neural network, maps a set of input vectors to a set of output vectors.

[0069] Tanh: A kind of activation function of neural network, which maps the input to the interval of -1 to 1.

[0070] GELU: is an activation function of neural network.

[0071] ReLU: is an activation function of neural network.

[0072] Discrete wavelet transform: is the scale and translation of basic wavelet is discretized, commonly used in image processing two wavelets as wavelet transform function, can be on different scales to signal decomposition, and the choice of different scales can be determined according to different target.

[0073] As Figure 1 shown, the embodiment of the application provides a CT image synthesis method based on MR image, comprising steps S100-S300:

[0074] S100, obtaining training data.

[0075] In the embodiment of the application, the training data includes a first MR image and a first CT image with the same target object, that is, the first MR image and the first CT image are images obtained by photographing the same target object, for example, including but not limited to a part of the object, such as hand, leg or neck, etc. It should be noted that MR is the abbreviation of MRI, that is, magnetic resonance imaging (Magnetic Resonance Imaging, MRI), and CT refers to computer tomography.

[0076] Optionally, step S100 comprises steps S110-S150:

[0077] S110, obtaining an original MR image and an original CT image with the same target object.

[0078] It should be noted that the original MR image and the original CT image are images before preprocessing, and the number of original MR images and original CT images is several.

[0079] S120, after the original MR image is subjected to the shift field operation, the original CT image is registered to obtain a registered MR image and a registered CT image.

[0080] Optionally, the original MR image and the original CT image need to be preprocessed, specifically: the original MR image is subjected to the N4 shift field operation, and the Elastix method is used to register the original CT image, and the length and width of the image are adjusted to 512 resolution to obtain the registered MR image and the registered CT image.

[0081] S130, performing preset range assignment processing on the registered CT image.

[0082] Optionally, the registered MR images and the registered CT images are classified into a training set, a verification set and a test set according to a preset ratio. The registered CT images in the training set are subjected to preset range assignment processing, and the preset range can be adjusted according to actual conditions. For example, the CT values of the registered CT images are assigned as -1000 when the CT values are lower than -1000, and the CT values are assigned as 2500 when the CT values are higher than 2500, and the preset range is set as -1000-2500.

[0083] S140, normalizing the registered MR images and the registered CT images subjected to the assignment processing.

[0084] In the embodiment of the application, the values of the registered MR images and the registered CT images subjected to the assignment processing are all normalized to -1 to 1.

[0085] S150, cutting a preset size of images from the registered MR images subjected to the normalization processing to obtain a first MR image, and cutting a preset size of images from the registered CT images subjected to the normalization processing to obtain a first CT image.

[0086] Specifically, a center point is selected from the registered MR images subjected to the normalization processing, and a region of a certain size, for example, a region of 8x256x256, is determined with the center point as the center, and then 8x256x256 size images are cut at the corresponding positions in the registered MR images subjected to the normalization processing and the registered CT images subjected to the normalization processing, respectively, to obtain the first MR image and the first CT image.

[0087] S200, inputting the first MR image and the first CT image into a network model for training to obtain an image synthesis model.

[0088] Optionally, the network model comprises a CST generator and a discriminator, the basic framework of the CST generator is a UNet structure, and the CST generator is a deep learning model combining a Swin transformer and a CNN network; in the embodiment of the application, the CST generator comprises an encoding module, a decoding module, an RSC self-attention module and a global block embedding module. It should be noted that by setting the CNN-Swin-Transformer generator, the computational complexity of the Transformer and the image size are in a linear relationship through the mechanism of local window self-attention, which reduces the computational overhead.

[0089] Optionally, step S200 comprises steps S210-S260:

[0090] S210, extracting local features of the first MR image by the encoding module.

[0091] Optionally, the encoding module includes at least one first CNN module, a first RSC self-attention module, and a second CNN module.

[0092] In this embodiment of the invention, step S210 includes steps S2101-S2103:

[0093] S2101. The first MR image is subjected to at least one first feature extraction through the first CNN module to obtain the first feature information.

[0094] like Figure 2 As shown, it should be noted that the number of first CNN modules can be set as needed. In this embodiment of the invention, three first CNN modules are set between the first RSC self-attention module 104 and the input first MR image, namely the first CNN module 101, the first CNN module 102 and the first CNN module 103. The first feature extraction process performed by each first CNN module is the same. The first CNN module 101 performs first feature extraction on the input of the first CNN module 101, the first CNN module 102 performs first feature extraction on the input of the first CNN module 102 (such as the output of the first CNN module 101), and the first CNN module 103 performs first feature extraction on the input of the first CNN module 103 (such as the output of the first CNN module 102). That is, the first feature extraction is performed three times on the first MR image, so that the first CNN module 103 outputs the first feature information.

[0095] Optionally, each first feature extraction includes sequential convolution, instance normalization, and ReLU activation function. The principle is to map the number of channels of the input (e.g., the first MR image) to a high-dimensional space and output a feature map with a certain size, which may include the number of channels, layer thickness, height, width, etc. of the feature map.

[0096] S2102, The first feature information is processed by the first RSC self-attention module to perform second feature processing.

[0097] Specifically, the first feature information output by the first CNN module 103 is input into the first RSC self-attention module 104 to perform second feature processing on the first feature information.

[0098] S2103. The second feature is extracted from the second feature processing result by the second CNN module to obtain the local feature extraction result.

[0099] It should be noted that the process of the second feature extraction of the second CNN module 105 is similar to the first feature extraction. Optionally, the kernel size, padding and channel number of the first CNN module 101 are set to 3x5x5, 1x2x2 and 16; the convolution kernel and padding of the first CNN module 102, the first CNN module 103 and the second CNN module 105 are set to 3x3x3, 1x1x1, and the channel number is set to the channel number of the previous CNN module (which can be the second CNN module or the first CNN module), for example, the channel number of the first CNN module 103 is twice the channel number of the first CNN module 102. In addition, the convolution step size of the first CNN module 102 is set to 2x2x2, and the convolution step size of the first CNN module 103 and the second CNN module 105 is set to 1x2x2, so as to avoid the case that the image information is damaged due to excessive downsampling in the dimension D because the number of layers D of the input image is relatively thin.

[0100] In the embodiment of the application, the image is downsampled by controlling the size of the convolution step size of the first CNN module 102, the first CNN module 103 and the second CNN module 105, the number of channels is doubled, and after three convolutions with downsampling properties, the depth local feature information of the image, i.e. the local feature extraction result, can be obtained. For example, the local feature extraction result output by the second CNN module 105 is the feature map where R is a real number, C is the channel number of the first MR image, H is the height of the first MR image, W is the width of the first MR image, and D is the (number of layers) thickness of the first MR image.

[0101] S220, performing global feature extraction on the first MR image by a global patch embedding module.

[0102] As shown in Figure 2 Optionally, the global patch embedding module 201 constitutes a global patch embedding branch, which is used for performing global feature extraction on the first MR image. Specifically, the global patch embedding module 201 comprises a patch embedding module 202 and a second RSC self-attention module 203.

[0103] Optionally, step S220 comprises steps S2201-S2202:

[0104] S2201, performing second feature extraction on the first MR image by the patch embedding module to obtain second feature information.

[0105] In the embodiment of the application, it is assumed that C is the channel number of the first MR image, H is the height of the first MR image, W is the width of the first MR image, and D is the (number of layers) thickness of the first MR image, and the first MR image F is in R C×H×W×DThe second feature extraction includes but is not limited to convolution processing, dimension flattening, transposition, layer normalization and recovery processing:

[0106] The convolution processing obtains a convolution image block F conv The formula is as follows:

[0107]

[0108] Wherein, Conv(·) represents convolution operation.

[0109] Then, F conv is subjected to dimension conversion into one dimension (dimension flattening) and transposition, layer normalization is performed on the channels of C, and a layer normalization result F Pemb is obtained. The formula is as follows:

[0110] F Pemb = Layernorm(transpose(flatten(F conv )))

[0111] Wherein, Layernorm represents layer normalization, transpose represents transposition, and flatten represents dimension flattening. Finally, the dimensions of the features in the normalization result F Pemb are subjected to recovery processing to restore the original input block embedding module size, and the second feature information is obtained and output.

[0112] S2202, the second feature information is subjected to third feature processing by the second RSC self-attention module, and a global feature extraction result is obtained.

[0113] As shown in Figure 2 , in the embodiment of the application, in order to increase the information interaction between the features of the block embedding module (i.e., the Patch embedding module 202) and the globality, a second RSC self-attention module 203 is introduced to the output second feature information to perform third feature processing on the second feature information.

[0114] S230, the local feature extraction result and the global feature extraction result are fused, and the fusion result is subjected to first feature processing by the RSC self-attention module.

[0115] Specifically, the global feature extraction result representing the features based on the image block level is fused with the local feature extraction result representing the features based on the pixel level, and the fusion includes but is not limited to addition, and then the fusion result is subjected to first feature processing through an RSC self-attention module. It should be noted that the RSC self-attention module includes at least one third RSC self-attention module, and the RSC self-attention module is a residual Swin-CNN (RSC) transformer module. The number of the third RSC self-attention module can be adjusted according to actual conditions, and is not specifically limited; each third RSC self-attention module performs one feature sub-processing on the input, and therefore the first feature processing includes at least one feature sub-processing.

[0116] In the embodiment of the application, the RSC self-attention module is introduced, the Swin transformer module is optimized using separable convolution and dilated convolution, the receptive field of the features is expanded, and the CNN is combined to retain the local details of the CNN while enhancing the globality of the features.

[0117] Exemplarily, taking the third RSC self-attention module 300 including six third RSC self-attention modules as an example, the third RSC self-attention modules are sequentially connected, the structures of the third RSC self-attention modules are the same, and the structures of the third RSC self-attention module, the second RSC self-attention module and the first RSC self-attention module are the same, and the process of the feature sub-processing is the same as the second feature processing and the third feature processing.

[0118] As shown in Figure 3 Optionally, taking one of the third RSC self-attention modules as an example, the third RSC self-attention module includes a first unit 301, a first Swin transformer module (including a second unit 302 and a third unit 303), and a second Swin transformer module (including a fourth unit 304 and a fifth unit 305), the first unit 301, the second unit 302, the third unit 303, the fourth unit 304 and the fifth unit 305 are connected by a residual connection, and the feature sub-processing included in the first feature processing in step S230 specifically includes S2301-S2306:

[0119] S2301, performing first separable convolution on the fusion result, performing first dilated convolution on the first separable convolution result, performing second separable convolution on the first dilated convolution result, and performing second dilated convolution on the second separable convolution result.

[0120] Specifically, the first unit 301 performs first separable convolution on the fusion result, performs first dilated convolution on the first separable convolution result, performs second separable convolution on the first dilated convolution result, and performs second dilated convolution on the second separable convolution result and outputs.

[0121] It should be noted that the separable convolution is a shared separable convolution, and the ordinary convolution can process each channel respectively, and the ordinary convolution only has a filter and a feature map same as the number of channels, while the shared separable convolution shares weights between filters, which is equivalent to only having one filter, so that the calculation amount of the convolution process is obviously reduced. Alternatively, the dilated rate used by the first dilated convolution and the first dilated convolution is 1x2x2, which can effectively expand the receptive field of the extracted features.

[0122] In the embodiment of the application, by introducing the combination of separable convolution and dilated convolution, the receptive field of the input image is expanded, so that the image features input into the second unit 302 subsequently have stronger globality, and the quality of the finally synthesized CT image is improved.

[0123] S2302, performing first layer normalization according to the second dilated convolution result, and performing window multi-head attention on the first layer normalization result.

[0124] Specifically, the input of the second unit 302 is the result of adding the second dilated convolution result and the original input of the second unit 301, and the second unit 302 performs first layer normalization on the result, performs window multi-head attention on the first layer normalization result, and outputs.

[0125] Suppose the input of the second unit 302 is C s is a channel, D s is a dimension (thickness), H s is a height, and W s is a width. The input s is flattened and transposed in the dimension D s after the dimension, to obtain the first layer normalization result wherein L=D s ×H s ×W . The window multi-head attention WMSA includes setting a three-dimensional local window and calculating a multi-head self-attention MSA, that is, the input is first divided into an image block of a three-dimensional local window size, and then the multi-head self-attention is calculated. Specifically: according to the size S d ×S h ×S w of the three-dimensional local window (dimension, height, and width), the size of X S is adjusted to where N win represents the number of three-dimensional local windows, S win = S d × S h × S w represents the size of the window, C s represents the encoding length of each three-dimensional local window, the size of the three-dimensional local window includes but is not limited to being set to 2x4x4, and the number of heads (three-dimensional local windows) is 8. In addition, the calculation formula of the multi-head self-attention MSA is as follows:

[0126] MSA(Q, K, V) = Concat(head1, …, head i )W0

[0127]

[0128] where Q, K, and V are queries, keys, and values, respectively; head i represents the self-attention of the i-th head, Concat is a concatenation function, W0is a weight matrix of multi-head attention, and W i Q represents the weight matrix corresponding to Q, W i K represents the weight matrix corresponding to K, W i V represents the weight matrix corresponding to V, Attention() is a self-attention function, softmax(·) represents an activation function of a neural network, D k is the dimension size of K, and K T is the transpose of K.

[0129] S2303, first addition is performed according to the window multi-head attention result and the second dilated convolution result, second layer normalization is performed on the first addition result, and the second layer normalization result is processed by the first multi-layer perceptron.

[0130] Specifically, the input of the third unit 303 is the first addition result after the first addition is performed according to the window multi-head attention result and the second dilated convolution result, the third unit 303 performs second layer normalization on the first addition result, and the second layer normalization result is processed by the first multi-layer perceptron.

[0131] Specifically, the formula of the first multi-layer perceptron MLP is as follows:

[0132] MLP(X) = GELU(XW1 + b1)W2 + b2

[0133] Wherein, X is the second layer normalization result, GELU(·) is an activation function of the neural network, W1 and W2 are weights of two full connection layers in the first multi-layer perception (MLP) respectively, b1 and b2 are bias terms corresponding to the two full connection layers respectively.

[0134] S2304, the processing result of the first multi-layer perception is secondly added with the first addition result, the second addition result is thirdly normalized, and the third normalization result is processed by the moving window multi-head attention.

[0135] Specifically, the input of the fourth unit 304 is the second addition result after the processing result of the first multi-layer perception is secondly added with the first addition result, the fourth unit 304 thirdly normalizes the second addition result, and processes the third normalization result by the moving window multi-head attention. It should be noted that the moving window multi-head attention can be moved and can interact information between windows.

[0136] S2305, the moving window multi-head attention result is thirdly added with the second addition result, the third addition result is fourthly normalized, and the fourth normalization result is processed by the second multi-layer perception.

[0137] Specifically, the input of the fifth unit 305 is the third addition result after the moving window multi-head attention result is thirdly added with the second addition result, the fifth unit 305 fourthly normalizes the third addition result, and processes the fourth normalization result by the second multi-layer perception.

[0138] It should be noted that the second Swin transformer module is similar to the first Swin transformer module and will not be repeated, and the difference is that the multi-head attention in the fourth unit 304 is the moving window multi-head attention S.

[0139] S2306, the fourth addition is performed according to the processing result of the second multi-layer perception, the third addition result and the second dilated convolution result, and the feature sub-processing result is obtained.

[0140] Specifically, the fourth addition is performed according to the processing result of the second multi-layer perception, the third addition result and the second dilated convolution result, and the feature sub-processing result is obtained and outputted, for example, the fourth addition is performed on the processing result of the second multi-layer perception, the third addition result and the second dilated convolution result to obtain the feature sub-processing result and output it.

[0141] It can be understood that when the third RSC self-attention module 300 has only one third RSC self-attention module, the first feature processing result output by the third RSC self-attention module is the feature sub-processing result, and when the third RSC self-attention module 300 has six third RSC self-attention modules, the feature sub-processing result output by the previous third RSC self-attention module is taken as the input of the subsequent third RSC self-attention module to perform another feature sub-processing, until the output of the last third RSC self-attention module is obtained as the final first feature processing result.

[0142] Therefore, the flow of the Swin transformer module is as follows:

[0143]

[0144]

[0145]

[0146] wherein, l represents the number of layers corresponding to the Swin transformer module (for example, the second unit 302 and the third unit 303 are the first layer, and the fourth unit 304 and the fifth unit 305 are the second layer), represents the input of the lth layer, represents the output of the multi-head attention, WMSA(·) represents the window multi-head self-attention, SWMSA(·) represents the moving window multi-head self-attention, Layernorm represents the layer normalization (such as first layer normalization, second layer normalization, third layer normalization, and fourth layer normalization), and MLP represents the first multi-layer perceptron or the second multi-layer perceptron. It should be noted that in SWMSA, the window is moved to the left and downward by half of the size of the three-dimensional window, and the self-attention is recalculated according to the new window information, thereby effectively improving the information interaction between the windows.

[0147] S240, decoding the first feature processing result by the decoding module to generate a second CT image.

[0148] Optionally, the decoding module 400 comprises a plurality of decoder CNN modules 401 connected in sequence and corresponding in number to the first CNN module, each decoder CNN module 401 is also provided with (up-sampling + convolution + instance normalization + ReLU activation function) to form a symmetrical structure with the first CNN module. It should be noted that the decoding process is to input the first feature processing result output by the RSC self-attention module into the first decoder CNN module 401 of the decoding module 400, and each decoder CNN module 401 is different from the first CNN module in that the up-sampling is realized by using the nearest neighbor interpolation method before convolution, and the corresponding first CNN module is connected by a jump to splice the feature maps of the same size in the first CNN module, so that the image gradually recovers to the original input size, and then each decoder CNN module 401 is activated by using the Tanh activation function, so that the last decoder CNN module 401 outputs the synthesized CT image to obtain the second CT image.

[0149] S250, input the first CT image and the second CT image into the discriminator to obtain a discrimination result.

[0150] Optionally, the second CT image synthesized by the CST generator and the first CT image are input into the Markov discriminator for judgment to obtain a discrimination result, for example, the real discrimination result is 1 and the false synthesis is 0.

[0151] As shown in Figure 4 , specifically, before inputting into the discriminator, the first CT image and the second CT image can also be input into a conversion module for discrete wavelet conversion and convolution conversion, respectively. The input image (such as the first CT image or the second CT image) is decomposed in the horizontal and vertical directions by Haar discrete wavelet to obtain 4 feature maps with resolution reduced to half of the original, and then convolution conversion is performed by using a convolution module: a convolution kernel with a convolution step of 2 is used to obtain 1 feature map with resolution reduced to half of the original, and 5 feature maps obtained by the two conversion methods are spliced as inputs of the Markov discriminator. Then, the Markov discriminator divides the input image into a plurality of N×N image blocks for judgment, which can effectively model the image as a Markov random field, so that the discrimination result can be obtained according to the input image. Through the above method, the discrete wavelet variation and the convolution conversion are used on the input on the basis of the existing Markov discriminator, which enhances the detail information of the image and is beneficial to improve the discrimination ability of the image. Figure 1 Figure 1 S260, training the network model according to the discrimination result and a preset loss function to obtain an image synthesis model.

[0152] S260, training the network model according to the discrimination result and a preset loss function to obtain an image synthesis model.

[0153] ​Optionally, the preset loss function includes but is not limited to an adversarial loss function, a mean absolute error loss function, and a perceptual loss function, and step S260 includes steps S2601-S2603.

[0154] S2601, determining an adversarial loss value of the discriminator according to the discrimination result and the adversarial loss function.

[0155] Specifically, the adversarial loss value L adv is calculated according to the discrimination result and the adversarial loss function.

[0156] S2602, determining a mean absolute error value according to the first CT image, the second CT image, and the mean absolute error loss function.

[0157] Specifically, the mean absolute error value L L1 is determined according to the first CT image, the second CT image, and the mean absolute error loss function (i.e., the L1 loss function).

[0158] S2603, determining a perceptual loss value according to the first CT image, the second CT image, and the perceptual loss function.

[0159] Specifically, the perceptual loss value L P is determined according to the first CT image, the second CT image, and the perceptual loss function. It should be noted that the perceptual loss function adopts a pre-trained VGG19 perceptual loss model, the VGG19 perceptual loss model convolves the first CT image and the second CT image to obtain first feature and second feature, and then calculates the difference according to the first feature and the second feature to obtain the perceptual loss value L P .

[0160] S2604, adjusting network parameters of the network model according to the adversarial loss value, the mean absolute error value, and the perceptual loss value to obtain an image synthesis model.

[0161] The adversarial loss value, the mean absolute error value, and the perceptual loss value constitute a total loss function L total as shown in the following formula:

[0162]

[0163] Where G represents the CST generator, D represents the discriminator, min is the minimization generator loss, max is the maximization discriminator loss, L adv represents the adversarial loss value, L L1 represents the mean absolute error value, and L PThe representative perceptual loss value is specifically obtained by alternately training the CST generator and the discriminator, updating the network parameters, and reaching a target range of the total loss function value to obtain the image synthesis model. Optionally, the initial learning rate used in the training process is 0.0002, the Adma optimizer is used to train 150 batches.

[0164] Optionally, the CT image synthesis method based on the MR image according to the embodiment of the present application further comprises a step S270:

[0165] S270, model test saving.

[0166] For example, the index used in the test and verification process is the mean square error, the validation set is tested in each batch, the mean square error of the second CT image and the first CT image is calculated, and the network parameters corresponding to the minimum mean square error are saved as the model result. Finally, the model result is sent to the test set for index evaluation to obtain the final model result, that is, the image synthesis model.

[0167] Optionally, the network parameters include but are not limited to data processing (or preprocessing) related parameters, training process and training related parameters, or network related parameters. For example, the data processing (or preprocessing) related parameters include but are not limited to the parameters of enriching the database (enrich data), the parameters of data generalization processing (feature normalization and scaling), and the parameters of BN processing (batch normalization); the training process and training related parameters include but are not limited to the training momentum, the learning rate, the attenuation function, the weight initialization, and the regularization related method; the network related parameters include but are not limited to the selection parameters of the classifier, the number of neurons, the number of filters, and the number of network layers.

[0168] S300, inputting the to-be-synthesized MR image into the image synthesis model to obtain a target CT image.

[0169] As shown in the figure, Fig. 5(a) is a to-be-synthesized MR image, Fig. 5(b) is a real CT corresponding to the to-be-synthesized MR image, the to-be-synthesized MR image is input into the image synthesis model, and a target CT image shown in Fig. 5(c) is obtained. It can be seen that the difference between the target CT image and the real CT is very small. Specifically, after inputting the to-be-synthesized MR images of 43 patients in the test set into the image synthesis model, the final mean square error MAE is 69.02, the structural similarity SSIM is 0.759, and the peak signal-to-noise ratio PSNR is 27.78. The output difference is shown in Fig. 5(d), which is the output difference result of the target CT image in Fig. 5(c) minus the real CT in Fig. 5(b). It can be seen that the overall difference is small.

[0170] In addition, the CT image synthesis method based on the MR image has high synthesis quality and rich synthesis details, and the synthesis of air and bone parts is more prominent than that of the prior art.

[0171] The embodiment of the present application also provides a CT image synthesis device based on an MR image, comprising:

[0172] The acquisition module is configured to acquire training data, wherein the training data comprises a first MR image and a first CT image of a same target object.

[0173] The training module is configured to input the first MR image and the first CT image into a network model to perform training, and obtain an image synthesis model, wherein the network model comprises a CST generator and a discriminator, the CST generator comprises an encoding module, a decoding module, an RSC self-attention module and a global block embedding module, the global block embedding module is configured to perform global feature extraction on the first MR image, the encoding module is configured to perform local feature extraction on the first MR image, the RSC self-attention module is configured to perform first feature processing according to the global feature extraction result and the local feature extraction result, and the decoding module is configured to perform decoding processing on the first feature processing result to generate a second CT image, and the discriminator is configured to generate a discrimination result according to the first CT image and the second CT image to perform training.

[0174] The synthesis module is configured to input a to-be-synthesized MR image into the image synthesis model to obtain a target CT image.

[0175] The contents in the above method embodiments are applicable to the device embodiments, the device embodiments specifically realize the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0176] The embodiment of the present application also provides an electronic device, which comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, and the processor loads and executes the at least one instruction, the at least one program, the code set or the instruction set to realize the CT image synthesis method based on the MR image of the foregoing embodiments. The electronic device of the embodiment of the present application includes but is not limited to any smart terminal such as a mobile phone, a tablet computer, a computer and a vehicle-mounted computer.

[0177] The contents in the above method embodiments are applicable to the device embodiments, the device embodiments specifically realize the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0178] The embodiment of the present application also provides a computer readable storage medium, and the computer readable storage medium stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded by a processor and executed to implement the MR image based CT image synthesis method of the foregoing embodiment.

[0179] The embodiment of the present application also provides a computer program product or a computer program, and the computer program product or the computer program includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the MR image based CT image synthesis method of the foregoing embodiment.

[0180] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is to be construed to cover the recurrence of a process, procedure, composition, element or step in different forms and permutations of the application unless there is a clear indication to the contrary. It is further understood that the use of relational terms such as "comprising", "having", "including", and "containing" are to be construed in a non-exclusively manner, i.e. these terms are to be construed to cover a process, method, system, product, or device including additional steps, elements, or components not expressly recited or only implicitly by a sentence, clause or phrase.

[0181] It should be understood that, in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases of only A, only B and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" or the like means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b and c can be single or multiple.

[0182] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the described apparatus embodiments are merely schematic. The division of the units is merely logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0183] The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially or substantially, or all or part of the technical solutions, can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions used to cause a computer device (such as a personal computer, a server, or a network device) to perform all or part of the steps of the methods in the embodiments of the present application. The aforementioned storage medium includes: various memories (such as a read-only memory, a random access memory, and a flash memory), a disk, and the like.

[0184] The above, the above embodiments are merely used to describe the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of CT image synthesis based on MR images, characterized in that, The method comprises the following steps: acquiring training data; the training data comprises a first MR image and a first CT image of a same target object; inputting the first MR image and the first CT image into a network model for training to obtain an image synthesis model; the network model comprises a CST generator and a discriminator; the CST generator comprises an encoding module, a decoding module, an RSC self-attention module, and a global block embedding module; the global block embedding module is configured to extract global features of the first MR image; the encoding module is configured to extract local features of the first MR image; the RSC self-attention module is configured to perform first feature processing on the global feature extraction result and the local feature extraction result; the decoding module is configured to perform decoding processing on the first feature processing result to generate a second CT image; the discriminator is configured to generate a discrimination result according to the first CT image and the second CT image for training; inputting a to-be-synthesized MR image into the image synthesis model to obtain a target CT image; the step of inputting the first MR image and the first CT image into the network model for training to obtain the image synthesis model comprises the following steps: extracting local features of the first MR image by the encoding module; extracting global features of the first MR image by the global block embedding module; fusing the local feature extraction result and the global feature extraction result, and performing first feature processing on the fusion result by the RSC self-attention module; performing decoding processing on the first feature processing result by the decoding module to generate a second CT image; inputting the first CT image and the second CT image into the discriminator to obtain a discrimination result; training the network model according to the discrimination result and a preset loss function to obtain the image synthesis model; the encoding module comprises at least one first CNN module, a first RSC self-attention module, and a second CNN module; the step of extracting local features of the first MR image by the encoding module comprises the following steps: extracting first feature information of the first MR image at least once by the first CNN module; performing second feature processing on the first feature information by the first RSC self-attention module; extracting local feature extraction result by performing second feature extraction on the second feature processing result by the second CNN module; the global block embedding module comprises a block embedding module and a second RSC self-attention module; the step of extracting global features of the first MR image by the global block embedding module comprises the following steps: extracting second feature information of the first MR image by the block embedding module; obtaining global feature extraction result by performing third feature processing on the second feature information by the second RSC self-attention module.

2. The CT image synthesis method based on MR images according to claim 1, characterized in that: the step of acquiring training data comprises the following steps: acquiring an original MR image and an original CT image of a same target object; performing offset field operation on the original MR image and performing registration with the original CT image to obtain a registered MR image and a registered CT image; Perform assignment processing of a preset range on the registered CT image; Perform normalization processing on the registered MR image and the registered CT image after the assignment processing; Obtain a first MR image by cropping a preset size of image from the registered MR image after the normalization processing, and obtain a first CT image by cropping a preset size of image from the registered CT image after the normalization processing.

3. The CT image synthesis method based on MR images according to claim 1, characterized by: The first feature processing includes at least one feature sub-processing, and the RSC self-attention module includes at least one third RSC self-attention module; the first feature processing of the fusion result by the RSC self-attention module includes: The feature sub-processing of the fusion result by the third RSC self-attention module includes: Performing first separable convolution on the fusion result, performing first dilated convolution on the first separable convolution result, performing second separable convolution on the first dilated convolution result, and performing second dilated convolution on the second separable convolution result; Performing first layer normalization according to the second dilated convolution result, and performing window multi-head attention on the first layer normalization result; Performing first addition according to the window multi-head attention result and the second dilated convolution result, performing second layer normalization on the first addition result, and performing processing of the first multi-layer perception on the second layer normalization result; Performing second addition on the processing result of the first multi-layer perception and the first addition result, performing third layer normalization on the second addition result, and performing moving window multi-head attention on the third layer normalization result; Performing third addition on the moving window multi-head attention result and the second addition result, performing fourth layer normalization on the third addition result, and performing processing of the second multi-layer perception on the fourth layer normalization result; Performing fourth addition according to the processing result of the second multi-layer perception, the third addition result and the second dilated convolution result to obtain the feature sub-processing result.

4. The CT image synthesis method based on MR images according to Claim 1, characterized by: The training of the network model according to the discrimination result and the preset loss function to obtain the image synthesis model includes: Determining the adversarial loss value of the discriminator according to the discrimination result and the adversarial loss function; Determining the mean absolute error value according to the first CT image, the second CT image and the mean absolute error loss function; Determining the perceptual loss value according to the first CT image, the second CT image and the perceptual loss function; Adjusting the network parameters of the network model according to the adversarial loss value, the mean absolute error value and the perceptual loss value to obtain the image synthesis model.

5. An apparatus for implementing the MR image-based CT image synthesis method according to any one of claims 1 to 4, characterized by It includes: An acquisition module is configured to acquire training data; The training data includes a first MR image and a first CT image of a same target object; The first MR image and the first CT image are of a same target object. The training module is configured to input the first MR image and the first CT image into a network model to train the network model to obtain an image synthesis model; the network model comprises a CST generator and a discriminator; the CST generator comprises an encoding module, a decoding module, an RSC self-attention module, and a global block embedding module; the global block embedding module is configured to perform global feature extraction on the first MR image; the encoding module is configured to perform local feature extraction on the first MR image; the RSC self-attention module is configured to perform first feature processing according to the global feature extraction result and the local feature extraction result; the decoding module is configured to perform decoding processing on the first feature processing result to generate a second CT image; and the discriminator is configured to generate a discrimination result according to the first CT image and the second CT image to perform training. The synthesis module is configured to input a to-be-synthesized MR image into the image synthesis model to obtain a target CT image.

6. An electronic device, comprising: The electronic device comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set, or an instruction set, which are loaded and executed by the processor to implement the method according to any one of claims 1-4.

7. A computer readable storage medium characterized in that, The storage medium stores at least one instruction, at least one program, a code set, or an instruction set, which are loaded and executed by the processor to implement the method according to any one of claims 1-4.

Images