Cross-semantic perception based abdominal tumor contrast magnetic resonance image generation method

Abstract

The application discloses a kind of based on cross semantic perception's abdominal tumor contrast magnetic resonance image generation method, the generator based on feature enhancement and multiscale attention is introduced in the present application, and the expression ability of texture details and edge information is enhanced by refining feature map;In order to let discriminator can pay more attention to the spatial and semantic features of image, the discriminator based on cross semantic perception is designed, and the semantic features of real contrast image are introduced in the discriminator as additional guide information.Further, the reversibility of conversion is ensured by using cycle consistency loss, the semantic difference of image is learned by using perception loss, more attention is paid to the visual quality of image, and it is closer to the subjective perception of human eye to image quality, so as to solve the problem of pixel point misalignment between generated image and real image.

Description

Technical Field

[0001] This invention belongs to the field of magnetic resonance MRI imaging technology, specifically relating to a method for generating abdominal tumor contrast magnetic resonance images based on cross-semantic perception. Background Technology

[0002] MRI contrast-enhanced imaging is a technique that involves intravenously injecting a contrast agent followed by magnetic resonance imaging (MRI). While standard MRI already provides very clear images of anatomical structures, the contrast agent, once introduced into the body, alters the magnetic field environment within the tissue, thus shortening the tissue's T1 relaxation time. By observing the changes in signal intensity before and after contrast agent injection, physicians can more clearly observe the physiological and pathological functional state of human tissues, particularly blood perfusion and angiogenesis. This enables diagnostic, differential, and treatment decisions in critical areas such as oncology, neurology, and cardiovascular diseases.

[0003] However, traditional contrast imaging techniques rely heavily on intravenous or arterial injection of contrast agents, which not only increases the financial burden on patients but may also cause potential adverse reactions, especially for patients with renal insufficiency, potentially increasing the stress on the kidneys and exacerbating kidney damage. Therefore, generating high-quality contrast images using deep learning methods without the use of contrast agents has significant value and meaning for clinical diagnosis.

[0004] In recent years, deep learning-based medical image generation methods have made significant progress in image generation tasks of multiple modalities such as ultrasound, CT, and MRI. However, existing deep learning-based magnetic resonance MRI contrast image generation methods still face the following challenges: (1) Scarcity of high-quality registration data. Plain and contrast images of patients acquired at the same time point need to be registered to ensure pixel-level alignment. However, in actual clinical practice, even slight movement of the patient during the scanning process can lead to image misalignment, causing significant noise to the model's learning. In this case, the contrast images generated by traditional generation methods may exhibit artifacts or blurring, making them unsuitable for the diverse and complex task requirements in clinical events. Moreover, high-quality medical image data (such as multi-center, multi-device, large-scale datasets) is difficult to obtain, costly, and involves privacy issues. (2) Insufficient expression of heterogeneity. Tumors exhibit high heterogeneity in terms of cell density, blood vessel distribution, and necrotic areas, which are key diagnostic criteria. However, when deep learning models learn the average statistical characteristics of training data, they tend to generate the most common and ordinary tumor appearance, thus ignoring the heterogeneity of real tumors. (3) Severe loss of tumor edge details. For malignant tumors, their diagnostic invasive growth features (such as spiculation or lobulated structures) are difficult to accurately reconstruct during the tumor generation process, resulting in overly smooth or blurred tumor boundaries. This lack of morphological features means that the final contrast-enhanced images cannot capture the detailed features of the target region, directly affecting the assessment of its invasiveness. Therefore, constructing an image generator capable of accurately generating contrast-enhanced images of abdominal tumors has significant clinical importance and practical value. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a method for generating abdominal tumor contrast-enhanced magnetic resonance images based on cross-semantic perception. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] This invention provides a method for generating contrast-enhanced magnetic resonance images of abdominal tumors based on cross-semantic perception, the method comprising:

[0007] Obtain a realistic flat scan image;

[0008] The real plain scan images are processed using a pre-trained imaging generation model based on multi-scale attention and cross-semantic perception to output contrast-enhanced magnetic resonance images; wherein,

[0009] The imaging generation model based on multi-scale attention and cross-semantic perception includes:

[0010] Generator based on feature enhancement and multi-scale attention G Generators based on feature enhancement and multi-scale attentionF Discriminator based on cross-semantic awareness and a discriminator based on cross-semantic awareness ;

[0011] Generator based on feature enhancement and multi-scale attention:

[0012] Both feature enhancement and multi-scale attention generators include: an array of residual convolutional blocks that incorporates a feature enhancement module and a skip connection structure that incorporates multi-scale attention feature fusion;

[0013] The generator G Based on the feature enhancement module and multi-scale attention feature fusion, the contrast image I2 is output in a forward loop for the plain scan image I1, and the contrast image I2 is output in a backward loop for the unenhanced image I3;

[0014] The generator F Based on the feature enhancement module and multi-scale attention feature fusion, the unenhanced image I3 is output in a backward loop for the contrast image I2, and the unenhanced image I3 is output in a forward loop for the real contrast image I4;

[0015] Discriminator based on cross-semantic awareness:

[0016] Both discriminators based on cross-semantic awareness introduce corresponding cross-attention modules, each of which includes a linear layer and several attention heads;

[0017] The discriminator Based on the cross-attention module, the plain scan image I1 and the unenhanced image I3 are processed to output the plain scan image discrimination result and the discriminator. Adversarial loss for plain scan image I1 and unenhanced image I3, respectively;

[0018] The discriminator Based on the cross-attention module, the real contrast image I4 and the contrast image I2 are processed to output the contrast image discrimination result and the discriminator. Adversarial loss for contrast-enhanced image I2 and real contrast-enhanced image I4, respectively;

[0019] The loss function of the imaging generation model based on multi-scale attention and cross-semantic perception during training includes:

[0020] The loss function is based on a discriminator with cross-semantic perception and a generator based on feature enhancement and multi-scale attention; the loss function of the generator based on feature enhancement and multi-scale attention consists of flat scan adversarial loss, angiography adversarial loss, cycle consistency loss and perceptual loss.

[0021] The beneficial effects of this invention are:

[0022] The solution provided in this invention refines the feature map by introducing a generator based on feature enhancement and multi-scale attention, thereby enhancing the ability to express texture details and edge information. To enable the discriminator based on cross-semantic perception to pay more attention to the spatial and semantic features of the image, a discriminator based on cross-semantic perception is designed, incorporating the semantic features of real contrast images as additional guiding information. Furthermore, cycle consistency loss is used to ensure the reversibility of the transformation, and perceptual loss is used to learn the semantic differences of the image, focusing more on the visual quality of the image and more closely resembling the subjective perception of image quality by the human eye, thus solving the problem of pixel misalignment between the generated image and the real image. The abdominal tumor contrast MRI image generation method proposed in this invention can generate relatively high-quality abdominal tumor contrast images while simultaneously focusing on the detailed features of the tumor. This has significant clinical implications and practical value. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating the steps of a method for generating abdominal tumor contrast magnetic resonance images based on cross-semantic perception, as provided in an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram illustrating the principle of the contrast image generation model based on multi-scale attention and cross-semantic perception in the method for generating abdominal tumor contrast magnetic resonance images based on cross-semantic perception provided in an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram illustrating the principle of the generator based on feature enhancement and multi-scale attention in the angiography image generation model based on multi-scale attention and cross-semantic perception provided in the embodiments of the present invention.

[0026] Figure 4 This is a schematic diagram illustrating the principle of the feature enhancement module in the generator based on feature enhancement and multi-scale attention provided in an embodiment of the present invention.

[0027] Figure 5 This is a schematic diagram illustrating the principle of the multi-scale attention feature fusion module in the generator based on feature enhancement and multi-scale attention provided in this embodiment of the invention.

[0028] Figure 6 This is a schematic diagram illustrating the principle of the discriminator based on cross-semantic perception in the angiography image generation model based on multi-scale attention and cross-semantic perception provided in the embodiments of the present invention.

[0029] Figure 7 This is a visualization of the results of a cross-semantic perception-based method for generating contrast-enhanced magnetic resonance images of abdominal tumors, as provided in this embodiment of the invention, compared to contrast-enhanced images generated by existing technologies.

[0030] Figure 8 The image shows the subtraction visualization results of an abdominal tumor contrast magnetic resonance imaging image generation method based on cross-semantic perception provided in this embodiment of the invention, compared with the contrast images generated by existing technologies. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0032] This invention provides a method for generating abdominal tumor contrast-enhanced magnetic resonance images based on cross-semantic perception, such as... Figure 1 As shown, it may include the following steps:

[0033] S1, acquire the actual flat scan image;

[0034] S2 utilizes a pre-trained contrast-enhanced image generation model based on multi-scale attention and cross-semantic perception to process real plain scan images and output contrast-enhanced magnetic resonance images; among which...

[0035] Contrast image generation models based on multi-scale attention and cross-semantic perception can include:

[0036] Generator based on feature enhancement and multi-scale attention G Generators based on feature enhancement and multi-scale attention F Discriminator based on cross-semantic awareness and a discriminator based on cross-semantic awareness ;

[0037] Generator based on feature enhancement and multi-scale attention:

[0038] Both feature enhancement and multi-scale attention generators include: an array of residual convolutional blocks that incorporates a feature enhancement module and a skip connection structure that incorporates multi-scale attention feature fusion;

[0039] generator G Based on the feature enhancement module and multi-scale attention feature fusion, the contrast image I2 is output in a forward loop for the plain scan image I1, and the contrast image I2 is output in a backward loop for the unenhanced image I3;

[0040] generator F Based on the feature enhancement module and multi-scale attention feature fusion, the unenhanced image I3 is output in a backward loop for the contrast image I2, and the unenhanced image I3 is output in a forward loop for the real contrast image I4;

[0041] Discriminator based on cross-semantic awareness:

[0042] Both discriminators based on cross-semantic awareness introduce corresponding cross-attention modules, each of which includes a linear layer and several attention heads;

[0043] Discriminator Based on the cross-attention module, the plain scan image I1 and the unenhanced image I3 are processed to output the plain scan image discrimination result and the discriminator. Adversarial loss for plain scan image I1 and unenhanced image I3, respectively;

[0044] Discriminator Based on the cross-attention module, the real contrast image I4 and the contrast image I2 are processed to output the contrast image discrimination result and the discriminator. Adversarial loss for contrast-enhanced image I2 and real contrast-enhanced image I4, respectively;

[0045] The loss function during the training process of the angiography image generation model based on multi-scale attention and cross-semantic perception includes:

[0046] The loss function is based on the discriminator with cross-semantic perception and the loss function is based on the generator with feature enhancement and multi-scale attention. The loss function of the generator with feature enhancement and multi-scale attention consists of flat scan adversarial loss, angiography adversarial loss, cycle consistency loss and perceptual loss.

[0047] To improve the image quality of generated contrast images while simultaneously focusing on detailed features in the target region, this invention proposes a method for generating abdominal tumor contrast-enhanced magnetic resonance images based on cross-semantic awareness. The overall generation framework of this method is based on the traditional CycleGAN structure, and incorporates a generator based on feature enhancement and multi-scale attention to refine the feature map, enhancing its ability to express texture details and edge information. Furthermore, to enable the discriminator to pay closer attention to the spatial and semantic features of the image, this invention introduces a discriminator based on cross-semantic awareness, incorporating the semantic features of the actual contrast image as additional guiding information. Figure 2This paper presents the overall framework of SAMT-GAN, an angiographic image generation model based on multi-scale attention and cross-semantic perception. It shows that it consists of two generators based on feature enhancement and multi-scale attention, and two discriminators based on cross-semantic perception, utilizing cycle consistency loss to ensure the reversibility of the transformation. Furthermore, due to the misalignment between flat and angiographic images, the generated image may also have pixel misalignment with the real image. This embodiment of the invention introduces perceptual loss to learn the semantic differences between the images. Compared with the traditional mean squared error loss function, perceptual loss focuses more on the visual quality of the image and is closer to the subjective perception of image quality by the human eye. By calculating the differences between two images using a pre-trained neural network, high-level features of the image can be extracted.

[0048] The generator based on feature enhancement and multi-scale attention in the SAMT-GAN angiography image generation model proposed in this embodiment of the invention, such as... Figure 3 As shown, it may include:

[0049] The encoder, residual convolutional block array, and decoder are connected in sequence, along with a skip connection structure located between the encoder and decoder; among them,

[0050] The residual convolutional block array consists of nine sequentially connected residual convolutional blocks, with feature enhancement modules introduced in the first and sixth residual convolutional blocks;

[0051] The skip connection structure includes a first multi-scale attention feature fusion module and a second multi-scale attention feature fusion module.

[0052] Understandably, this generator based on feature enhancement and multi-scale attention employs a U-shaped structure with an encoder and a decoder. It extracts image features through a series of downsampling modules, gradually restores the image to its original size through a series of upsampling modules, and then fuses shallow and deep semantic features through a skip connection structure, thereby improving the quality of the generated image.

[0053] from Figure 3As can be seen, nine sequentially connected residual convolutional blocks are set between the encoder and decoder. In this embodiment of the invention, a Feature Enhancement Module (FEM) is introduced in the first and sixth residual convolutional blocks to enhance the expression of local and global features, prevent the loss of deep information, enable the network to effectively learn more complex features, and improve the quality of image generation. At the same time, a Multi-Scale Attention Feature Fusion Module (MAFM) is introduced in the skip connection structure to fuse features of different scales. By allocating attention weights, the network pays more attention to the generation of regions, improves the model's attention to key features, and thus makes the generated image clearer.

[0054] Specifically, feature enhancement modules, such as Figure 4 As shown, it may include:

[0055] The following components are connected in sequence: downsampling unit, position encoding unit, first normalization layer, multi-head self-attention unit, second normalization layer, feedforward layer, residual connection unit, upsampling unit, and convolutional block.

[0056] The feature enhancement module first downsamples the input feature map twice, reducing the image size to 1 / 4 of the input image. Then, the downsampled image is flattened into a 2D sequence, and positional encoding is added to each location within the sequence. This sequence is then fed into a Multi-Head Self-Attention (MSA) unit. The output of the MSA unit is added to the flattened 2D sequence, and the transformed sequence is obtained through Layer Normalization (LN), a FeedForward layer, and residual connections. Finally, the transformed sequence is resized to a specified size, then upsampled twice to restore it to the input image size and concatenated with the input image along the channel dimension. A 1×1 convolutional layer halves the channels, and the final output of the feature enhancement module is obtained through two stacked 3×3 convolutional blocks and residual connections. The Multi-Head Self-Attention (MSA) unit, as shown below... Figure 4As shown in the lower left section, the attention layer has three inputs: Q, K, and V. Q represents the information to be focused on, K represents the features of each element in the sequence, and V represents the actual information contained within. Q, K, and V are obtained by performing three different linear transformations on the input sequence. These transformed vectors are then divided into several heads. Each head first calculates the dot product of Q and K to obtain the attention weight, and then performs the dot product with V to obtain the result for that head. Finally, the results of multiple heads are concatenated and passed to a linear layer to obtain the final output of the multi-head self-attention module. The Feature Enhancement Module (FEM) models global dependencies in the downsampled low-dimensional feature space, overcoming the limitation of traditional CNNs that can only handle local features. FEM utilizes CNNs for local feature extraction while introducing MSA to model long-distance dependencies, fully mining information from both global and local features and improving the representational power of features.

[0057] The first multi-scale attention feature fusion module is set between the encoder and decoder at the original resolution, and the second multi-scale attention feature fusion module is set between the encoder and decoder downsampled to the second resolution.

[0058] The working process of the multi-scale attention feature fusion module may include:

[0059] The multi-scale attention feature fusion module integrates the feature maps corresponding to the encoding layer. Feature maps corresponding to the decoding layer Add them together to output a superimposed feature map. ;

[0060] For overlay feature maps Perform different convolutions to obtain feature maps of different scales. , , ;

[0061] The feature map is obtained by adding feature maps of different scales. For feature maps Global average pooling and channel mapping are performed, and then feature maps are obtained through three convolutions. Feature map Feature map ;

[0062] feature map Feature map and feature map The corresponding weight feature maps are obtained by applying the Sigmoid activation function respectively. , , ;

[0063] Weighted feature map , , Feature maps of different scales , , Multiplying the corresponding values ​​yields a weighted result at three different scales. , , ;

[0064] Weighted results of three different scales , , The summation serves as the output of the multi-scale attention feature fusion module.

[0065] Understandably, a schematic diagram of the Multi-Scale Attention Feature Fusion (MAFM) module is shown below. Figure 5 As shown, the input to this multi-scale attention feature fusion module is the feature map output from a certain layer of the encoder. Feature maps output from the corresponding layer in the decoder The superimposed feature map is obtained by adding the two feature maps together. For overlay feature maps Three feature maps of different scales were obtained by performing 1×1 convolution, 3×3 convolution, and 5×5 convolution respectively. , , Add these three feature maps together to obtain the feature map. Regarding this feature map Global average pooling is used to obtain the feature map. Then, a 1×1 convolution is used to map the channels to 1 / 16 of their original size, and then three 1×1 convolutions are used to restore the channels to their original size. C The feature maps were obtained respectively. Feature map Feature map Then, the weighted feature maps between 0 and 1 are obtained by applying the Sigmoid activation function. , , Then calculate the weighted feature map , , Compare with feature maps of three different scales. , , Multiplying the corresponding values ​​yields a weighted result at three different scales. , , Finally, the weighted results of these three different scales were combined. , , The feature maps of the final output of MAFM are obtained by adding them together. This achieves the effect of enhancing useful channels and suppressing features of unimportant channels.

[0066] Traditional discriminators primarily rely on pixel-level information, tending to learn low-level texture information and thus neglecting global structure. Furthermore, they treat the entire image indiscriminately, potentially neglecting key regions and leading to information loss in lesion areas. To enable the discriminator to focus more on high-level semantic features and the feature changes in lesion areas, thereby ensuring consistency in lesion shape, size, and edge information between plain and contrast images, this invention proposes a cross-semantic perception-based discriminator, SAD, whose specific structure is as follows: Figure 6 As shown, it can be seen that by introducing the semantic information of the image as a discrimination condition, it enhances the discriminator's ability to distinguish between real and generated images.

[0067] Discriminator based on cross-semantic awareness For example, the working process of a discriminator based on cross-semantic awareness can include:

[0068] The discriminator based on cross-semantic awareness extracts the semantic information of the real angiographic image I4 through the intermediate feature layer of the pre-trained CLIP model, and then outputs the first 2D sequence through the linear layer.

[0069] The contrast image I2 is subjected to four consecutive downsampling operations and layer normalization processing to output the second 2D sequence;

[0070] Cross-attention is applied to the first 2D sequence and the second 2D sequence, and then a fused feature map is output through layer normalization, feedforward layer and one upsampling operation.

[0071] The feature maps obtained by performing three downsampling operations on the fused feature map and the imaging image I2 are stitched together along the channel dimension, and then processed by a convolutional layer to output a Patch block with 1 channel.

[0072] The Patch block is evaluated, and the evaluation result is output.

[0073] Understandably, the input of the cross-semantic awareness discriminator SAD consists of contrast image I2 and real contrast image I4. The semantic information of the real contrast image I4 is extracted through the intermediate feature layer of the pre-trained CLIP model, and then flattened into a first 2D sequence. The contrast image I2 undergoes four consecutive downsampling operations to reduce its image size to 1 / 16 of the original size. Then, the downsampled image is flattened into a second 2D sequence. Cross-attention is applied to the first 2D sequence and the second 2D sequence. After layer normalization (LN), feed forward, and one upsampling operation, the image is restored to 1 / 8 of the input image size. Then, it is concatenated and integrated with the feature map obtained after three downsampling operations on the input image in the channel dimension. After processing by a 5×5 convolutional layer, a patch block with 1 channel is obtained. Finally, the obtained patch block is discriminated to obtain the discrimination result. In cross-attention, the features extracted from the angiographic image I2 are used as query Q, and the semantic features extracted from the real angiographic image I4 by the pre-trained CLIP model are used as key K and value V.

[0074] In the training process of the angiography image generation model based on multi-scale attention and cross-semantic perception, the loss function of the discriminator based on cross-semantic perception is determined by the discriminator based on cross-semantic perception. Adversarial loss for flat scan image I1 and unenhanced image I3 and a discriminator based on cross-semantic awareness. The adversarial loss is composed of the real contrast image I4 and the contrast image I2.

[0075] The loss function of the generator based on feature enhancement and multi-scale attention consists of flat scan adversarial loss, angiography adversarial loss, cycle consistency loss and perceptual loss;

[0076] The cycle consistency loss was confirmed using plain scan image I1, contrast image I2, real contrast image I4, and unenhanced image I3.

[0077] Perceptual loss was confirmed using plain scan image I1, unenhanced image I3, real contrast image I4, and contrast image I2.

[0078] The cycle consistency loss is determined using the plain scan image I1, the unenhanced image I3, the true contrast image I4, and the contrast image I2. The unenhanced image I3 can be obtained by further transforming the contrast image I2 generated from the plain scan image I1. The contrast image I2 can be obtained by further transforming the unenhanced image I3 generated from the true contrast image I4. Understandably, the cycle consistency loss... L cycle1 Cycle consistency loss can be confirmed by the plain scan image I1 and the unenhanced image I3.L cycle2 The perceptual loss can be confirmed by the real contrast-enhanced image I4 and the contrast-enhanced image I2. The perceptual loss can be confirmed by the plain scan image I1, the unenhanced image I3, the real contrast-enhanced image I4, and the contrast-enhanced image I2.

[0079] Cyclic consistency loss ensures that the generated image maintains consistency with the original image in its core structure, avoiding the loss of key medical features. Perceptual loss can calculate the differences in feature layers using pre-trained models such as VGG networks, ensuring that the generated contrast images are semantically closer to the real images.

[0080] The loss function of the discriminator based on cross-semantic awareness is as follows:

[0081] ;

[0082] ;

[0083] ;

[0084] in, x Represents a flat scan image. y Represents an imaging image. Represents a plain scan image X A data domain-based discriminator based on cross-semantic awareness. Representation of imaging images Y A data domain-based discriminator based on cross-semantic awareness. G A generator based on feature enhancement and multi-scale attention representing the direction from plain scan image to contrast image. F A generator based on feature enhancement and multi-scale attention that represents the direction from the contrast image to the plain scan image. X The data field represents the data field containing the plain scan image. Y The data field represents the data field where the imaging image is located.

[0085] The loss function of the generator based on feature enhancement and multi-scale attention is as follows:

[0086] ;

[0087] ;

[0088] ;

[0089] ;

[0090] in, This represents the cycle consistency loss, which is calculated from... X Data domain to Y Data domain to XThe mean absolute error between the image obtained from the data domain and the image from the original X data domain. The weights representing the cycle consistency loss Indicates perceived loss. The weights represent the perceptual loss, which are used to make the model pay more attention to semantic information. The ratio that can be set Slightly larger Indicates in Y The average is taken from the data samples in the data domain. express Y The true distribution of data samples in the data domain. express Y Randomly sample a data point from the real data in the data domain. y , Denotes the norm of L1, This means that image features are extracted through the VGG network. The contrast image I2 and the real contrast image I4 are respectively sent to the VGG network for feature extraction, and then the mean square error between them is calculated.

[0091] The training process for a pre-trained angiography image generation model based on multi-scale attention and cross-semantic perception may include:

[0092] Obtain a magnetic resonance imaging (MRI) contrast-enhanced image dataset of rectal cancer patients. The MRI contrast-enhanced image dataset includes real plain scan and contrast-enhanced paired data. Divide the MRI contrast-enhanced image dataset into a training sample dataset and a test sample dataset according to a preset ratio.

[0093] Set the training parameters and input the training sample dataset into the angiography image generation model based on multi-scale attention and cross-semantic perception;

[0094] Based on the generator's loss function, the generator, which combines feature enhancement and multi-scale attention, is trained using the training sample dataset to obtain the trained generator parameters W. C ;

[0095] Based on the discriminator's loss function, the cross-semantic awareness-based discriminator is trained using the training sample dataset to obtain the trained discriminator parameters W. D ;

[0096] Alternately update the generator based on feature enhancement and multi-scale attention and the discriminator based on cross-semantic perception until a preset number of iterations is reached, and output the updated generator parameters and updated discriminator parameters to obtain the trained angiography image generation model based on multi-scale attention and cross-semantic perception.

[0097] The trained contrast image generation model based on multi-scale attention and cross-semantic perception was tested using a test sample dataset to obtain an evaluation index for the contrast image generation model. If the evaluation index meets the preset requirements, the trained contrast image generation model based on multi-scale attention and cross-semantic perception is used as a pre-trained contrast image generation model based on multi-scale attention and cross-semantic perception. If the evaluation index does not meet the preset requirements, the trained contrast image generation model based on multi-scale attention and cross-semantic perception is trained again until the evaluation index meets the preset requirements, so as to obtain a pre-trained contrast image generation model based on multi-scale attention and cross-semantic perception.

[0098] Specifically, the training process of the pre-trained angiography image generation model based on multi-scale attention and cross-semantic perception can include:

[0099] S00, Data Preprocessing, may include:

[0100] S001, Obtain the magnetic resonance imaging (MRI) contrast data dataset of rectal cancer patients, including paired data of plain scan and actual contrast images;

[0101] S002, unify the size of the images in the tumor patient dataset, that is, downsample the MRI images and labeled images to a size of 256×256;

[0102] S003, divide the rectal tumor patient dataset into a 6:1 ratio to obtain the training sample set S1 and the test sample set S2.

[0103] S01, Constructing a generator based on feature enhancement and multi-scale attention can include:

[0104] S011, Building an encoder for extracting multi-scale features may include:

[0105] S0111, a 7×7×7 convolutional kernel with a stride of 1, an InstanceNorm normalization layer, and a LeakyReLU activation layer are connected in series to form Conv block C1;

[0106] S0112, a 3×3×3 convolutional kernel, an InstanceNorm normalization layer, and a LeakyReLU activation layer are connected in series to form Conv block C2;

[0107] S0113, an encoder is formed by concatenating one Conv block C1, two Conv blocks C2, and two downsampling layers.

[0108] S012, constructing a multi-head self-attention unit, may include:

[0109] S0121, construct three linear layers to perform linear transformations on the input sequence. These correspond to three inputs: Q represents the information we need to focus on, K represents the features of each element, and V represents the actual information contained within.

[0110] S0122, the transformed vector is divided into 8 attention heads, which are then connected in series with the linear layer;

[0111] S0123, perform a dot product operation on the query matrix Q and the key matrix K to determine the scaling factor, scale the dot product result to obtain the attention weight, and then perform a dot product with the value matrix V to obtain the result of the attention head;

[0112] S0124 concatenates the results of 8 attention heads and connects them in a linear layer to form a multi-head self-attention unit.

[0113] S013, Building a feature enhancement module may include:

[0114] S0131 connects two downsampling layers, reducing the image size to 1 / 4 of the input image;

[0115] S0132, flatten the downsampled image into a 2D sequence, and add a position code to each position in the sequence;

[0116] S0133, perform layer normalization on the sequence, and feed the transformed sequence into the constructed multi-head self-attention unit;

[0117] S0134, the connection layer normalization, feedforward layer and residual are used to transform the sequence;

[0118] S0135, reshape the transformed sequence to a size of 64×64;

[0119] S0136 connects two upsampling layers to restore the input image size to 256×256;

[0120] S0137, the input is concatenated as a feature map to the feature channel of the upsampled output, and a channel convolution with a kernel size of 1×1 is connected in series to extract the required features and halve the number of channels;

[0121] S0138, the downsampling layer, the constructed multi-head self-attention unit, the upsampling layer and two Conv blocks C2 are sequentially connected in series to form a feature enhancement module.

[0122] S014, Construct a multi-scale attention feature fusion module, which may include:

[0123] S0141, after fusing the feature maps of the encoder and decoder with the same resolution, the data is then input into the decoder;

[0124] S0142 is connected to 1×1 convolution, 3×3 convolution and 5×5 convolution respectively;

[0125] S0143, connected to a global averaging layer with the same resolution, and then concatenated with a convolution and a convolution with channels reduced to 1 / 16 of the original;

[0126] S0144, three convolutions are connected in parallel, and then concatenated with the sigmoid activation function, a 1×1 convolution, a 3×3 convolution, and a 5×5 convolution, respectively;

[0127] S0145 cascades three convolutional layers at different scales of original resolution, an average pooling layer, three C×1×1 convolutional layers, and three convolutional layers at different scales of original resolution to form a multi-scale attention feature fusion module.

[0128] S015, two Conv blocks C2, one Conv block C1 and two upsampling layers are concatenated in sequence to form a decoder.

[0129] S016, Constructing residual convolutions can include:

[0130] S0161, two 3×3×3 convolutional kernels, an InstanceNorm normalization layer, and a LeakyReLU activation layer are concatenated to form a residual convolutional block;

[0131] S0162, construct 9 residual convolutional blocks between the encoder and decoder;

[0132] S0163 introduces the constructed feature enhancement module FEM in the 1st and 6th residual convolutional blocks.

[0133] S017, the encoder, residual convolutional block with added feature enhancement module and decoder are connected in series, and a skip connection structure of the constructed multi-scale attention feature fusion module is added between the encoder and decoder of the same resolution to form a generator based on feature enhancement and multi-scale attention.

[0134] S02, Constructing a discriminator based on cross-semantic awareness may include:

[0135] S021, Building a cross-attention module may include:

[0136] S0211, construct three linear layers to perform linear transformations on the input sequence. These correspond to three inputs: Q represents the information of interest, obtained from the generated image; K represents the features of each element; and V represents the actual information contained within, obtained from the real image.

[0137] S0212, divide the transformed vector into 8 attention heads;

[0138] S0213, perform a dot product operation on the query matrix Q and the key matrix K to determine the scaling factor, scale the dot product result to obtain the attention weight, and then perform a dot product with the value matrix V to obtain the result of the attention head;

[0139] S0214 concatenates the results of the eight attention heads and connects them in a linear layer to form a cross-attention module.

[0140] S022, Processing the real contrast image I4 may include:

[0141] The intermediate layer of the pre-trained CLIP model is concatenated with a prior layer to extract semantic information, flatten it into a 2D sequence, and then reshape it.

[0142] S023, Processing the contrast image I2 may include:

[0143] S0231 connects four downsampling layers, reducing the image size to 1 / 16 of the input image;

[0144] S0232, flatten the downsampled image into a 2D sequence and reshape it;

[0145] S0233, perform layer normalization on the sequence.

[0146] S024, the features processed from the real contrast image I4 and the image processed from the contrast image I2 are fed into the constructed cross-attention module for cross-attention.

[0147] S025, the connection layer is normalized, the feedforward network and the upsampling layer are used to restore it to 1 / 8 of the input image size.

[0148] S026, the upsampled and restored image is concatenated with the feature map after three downsampling steps in the processing of the generated angiographic image, and then connected to a 5×5 convolution to obtain a patch block with 1 channel. The patch block is then judged to obtain the true and false judgment results.

[0149] S027. Based on the constructed generator based on feature enhancement and multi-scale attention and the discriminator based on cross-semantic perception, an angiography image generation model based on multi-scale attention and cross-semantic perception is obtained.

[0150] S03, training the angiography image generation model based on multi-scale attention and cross-semantic perception may include:

[0151] S031, set the total number of iterations to 200, the batch size to 8, randomly initialize the network parameters, and use the Adam optimizer for optimization. Set the initial learning rate of the generator based on feature enhancement and multi-scale attention and the discriminator based on cross-semantic awareness to 1e-4. Additionally, λ is the weight of the cycle consistency loss, with a value of 10; α is the weight of the perceptual loss, with a value of 20.

[0152] S032, the sum of the adversarial losses of the two discriminators based on cross-semantic perception is set to constitute the loss function of the discriminator based on cross-semantic perception.

[0153] S033 uses the MRI images from the training set S1 as input to the SMAT-GAN imaging generation model based on multi-scale attention and cross-semantic perception to obtain the corresponding tumor imaging results.

[0154] S034, updating the discriminator based on cross-semantic awareness may include:

[0155] Input real contrast images I4 and I2, calculate the loss value of the discriminator based on cross-semantic awareness, backpropagate, and use the Adam optimizer to update the parameters W of the cross-semantic awareness discriminator of the SMAT-GAN network. D .

[0156] S035, define the loss function of the generator based on feature enhancement and multi-scale attention by setting adversarial loss, cycle consistency loss and perceptual loss.

[0157] S036, updating the generator based on feature enhancement and multi-scale attention may include:

[0158] Input the original flat scan image, calculate the loss value of the generator based on feature enhancement and multi-scale attention, backpropagate, and use the Adam optimizer to update the generator parameters W of the SMAT-GAN network based on feature enhancement and multi-scale attention. C Obtain the model's evaluation metrics: PSNR, SSIM, and MAE.

[0159] S037, Repeat steps S034-S036, alternately updating the discriminator based on cross-semantic awareness and the generator based on feature enhancement and multi-scale attention until the set number of iterations is reached, and output the updated generator parameters based on feature enhancement and multi-scale attention and the discriminator parameters based on cross-semantic awareness.

[0160] S038. Repeat steps S034-S037 until all data in the training sample set S1 has been trained, completing one iteration of training.

[0161] S039, Repeat step S038 until the set number of iterations is reached, stop training, and select the model parameters with the best imaging generation evaluation index from all evaluation metrics as the model parameters of the pre-trained imaging generation model based on multi-scale attention and cross-semantic perception.

[0162] S0310, the pre-trained angiography image generation model based on multi-scale attention and cross-semantic perception is tested using the test sample set S2 to obtain the evaluation index of the pre-trained angiography image generation model based on multi-scale attention and cross-semantic perception.

[0163] The pre-trained angiography image generation model based on multi-scale attention and cross-semantic perception can directly generate angiography from flat scan images to output angiography magnetic resonance images.

[0164] To evaluate the robustness and generalization of the contrast image generation model based on multi-scale attention and cross-semantic perception proposed in this embodiment of the invention, comparative and ablation experiments were conducted using a dataset for verification. The dataset is an MRI contrast image dataset of rectal tumors. The detailed information and partitioning of this dataset will be described below. The MRI contrast image dataset of rectal tumors comes from the radiology department of a hospital, using contrast image data from 368 patients. Among them, 53 contrast images have tumor labels, and 315 contrast images do not. Since the generation method proposed in this embodiment of the invention is based on 2D images for training and testing, 2D slices were selected from the 3D images in this dataset to form the 2D image dataset of rectal cancer contrast images. A total of 5610 2D images were processed. The acquisition equipment used included a 1.5 Tesla scanner (Signa, GEM Medical Systems) and a 3.0 Tesla scanner (Magnetom Verio, Siemens), with five image resolutions available: 512×512, 256×208, 256×184, 256×192, and 288×288. The intra-slice resolution ranged from 0.6836mm×0.6836mm to 1.508mm×1.508mm, and the inter-slice thickness ranged from 2.5mm to 3.5mm. The 512×512 resolution image comprised the largest proportion of the data, which made the tumor region more clearly visible in the contrast images.

[0165] Due to the significant variation in image size within the dataset, all images were uniformly downsampled to 256×256 pixels. This experiment only included training and test sets; a validation set was not included. The data was divided into training and test sets based on patients. The training set consisted of 315 patients and 4778 2D images, while the test set consisted of 53 patients and 832 2D images. All data in the test set included pixel-level tumor segmentation labels to facilitate subsequent similarity calculations and visualization of tumor regions.

[0166] The experimental setup and evaluation metrics in this embodiment of the invention are as follows:

[0167] This invention was trained on a server running a Linux operating system. The server was equipped with an Intel Core 17-12700KF CPU and an NVIDIA GeForce RTX 3090 GPU with 24GB of video memory. PyTorch was used as the deep learning framework, and Python was used as the development language. The Adam optimizer was used in the experiment, with a batch size of 8. The total number of iterations was set to 200 when experimenting on the rectal tumor contrast imaging dataset. The initial learning rate of both the generator based on feature enhancement and multi-scale attention and the discriminator based on cross-semantic perception was set to 1e-4. Additionally, λ was the weight of the cycle consistency loss, with a value of 10, and α was the weight of the perceptual loss, with a value of 20. Evaluation metrics: On the rectal tumor contrast imaging dataset, the plain scan and contrast images of the tumor are almost perfectly aligned. Therefore, in this experiment, Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Mean Absolute Error (MAE) were used as evaluation metrics for the quality of the generated results. The specific calculation formulas are as follows:

[0168] ;

[0169] ;

[0170] Where L is a constant representing the maximum pixel value of the image, MSE represents the mean square error, and m and n are the height and width of the image, respectively. and These are the original image and the generated image, respectively. The lower the MSE value, the higher the quality of the image being evaluated; the higher the PSNR value, the better the quality of the image being evaluated.

[0171] ;

[0172] in, and It is a constant, and this constant is used to prevent the denominator from being 0. and These are the mean values ​​of the original image and the generated image, respectively. and These are the variances of the original image and the generated image, respectively. and These are the covariances of the original image and the generated image, respectively. SSIM is used to measure the similarity between two images in terms of structural features, brightness, and contrast. Its value ranges from [-1, 1]. The closer the value is to 1, the higher the similarity between the two images.

[0173] ;

[0174] MAE is the mean absolute error. The lower the value, the smaller the deviation between the generated image and the real image, and the closer the quality of the generated image is to the real image.

[0175] The comparative experimental results and analysis are as follows:

[0176] To demonstrate the superiority of the SAMT-GAN proposed in this embodiment, the model was trained and tested on a rectal tumor contrast imaging dataset. Seven generative algorithms—Pix2Pix, CycleGAN, TransUNet, ResViT, SAGAN, SegGuided-Diffusion, and RRDB-SeD—were compared in comparative experiments. Pix2Pix and CycleGAN are classic generative models and will not be elaborated further. TransUNet, ResViT, and SAGAN utilize attention mechanisms to enhance feature learning in key regions. SegGuided-Diffusion uses segmentation masks as guiding information to ensure the model follows the lesion morphology of the image during generation. RRDB-SeD introduces semantic information to improve global and local detail consistency. To ensure fairness, all comparison methods used the same data augmentation strategy during training, including horizontal and vertical flipping and random angle rotation.

[0177] To demonstrate the robustness and generalization of the SAMT-GAN proposed in this embodiment, additional model training and testing were conducted on a rectal tumor contrast dataset. Seven generation algorithms were used for comparative experiments, and the generation performance of the model was comprehensively evaluated using PSNR, SSIM, and MAE metrics.

[0178] Tables 1 and 2 show a comparison of the generation performance of the method proposed in the embodiments of the present invention with other generation methods in whole images and local tumor regions.

[0179] Table 1. Comparison of SAMT-GAN and other generation methods on whole-image generation results.

[0180]

[0181] Table 2. Comparison of SAMT-GAN and other generation methods in generating results on local tumor regions.

[0182]

[0183] As shown in Tables 1 and 2, compared with other generation methods, SAMT-GAN achieved the best PSNR (25.5151), SSIM (0.8330), and MAE (34.3376) on the whole image, and the best PSNR (17.2970), SSIM (0.4121), and MAE (149.9701) on the local tumor region. Compared with Pix2Pix, SAMT-GAN increased PSNR and SSIM by 0.8019 and 0.0358 respectively on the whole image, and reduced MAE by 3.8248. On the local tumor region, it increased PSNR and SSIM by 1.4644 and 0.0879 respectively, and reduced MAE by 38.8678. Among all the comparison methods, RRDB-SeD performs better in generating images across the entire image. Compared to RRDB-SeD, SAMT-GAN increases PSNR and SSIM by 0.3205 and 0.0103 respectively, and reduces MAE by 1.9935. This indicates that SAMT-GAN can force the discriminator based on cross-semantic awareness to pay more attention to high-level semantic features, making the generated image closer to the real image in both global structure and local details. Among all the comparison methods, SAGAN performs better in generating images in local tumor regions. Compared to SAGAN, SAMT-GAN increases PSNR and SSIM by 0.6681 and 0.0231 respectively, and reduces MAE by 22.4174. This indicates that SAMT-GAN enables the network to generate images more effectively in the target region, paying more attention to key regional features, thus making the generated image clearer. In summary, the comparative experiments on the results of generating images across the entire image and local tumor regions demonstrate the effectiveness and superiority of the SAMT-GAN proposed in this embodiment.

[0184] To more clearly compare the generation effects of different models, Figure 7 This image presents a visualization of the results of a cross-semantic perception-based method for generating contrast-enhanced MRI images of abdominal tumors, compared to existing techniques. The first row shows the plain scan image of the rectal tumor, the second row shows the contrast-enhanced image of the rectal tumor, and the last row shows the contrast-enhanced image predicted by SAMT-GAN. The tumor region is marked with a red box in the image. Figure 7As shown in columns 1, 3, and 6, the tumor region in the contrast images generated by SAMT-GAN is brighter and has higher contrast with the surrounding tissue compared to tumor regions generated by other models. The tumors generated by other models are relatively dark and have low differentiation from the surrounding tissue, especially in column 6, where the tumors in the contrast images generated by CycleGAN and TransUNet are almost undetectable. As shown in column 2, the internal tumor regions in the contrast images generated by Pix2Pix and SAGAN are relatively blurry, while the tumor details in the SAMT-GAN image are closer to the tumor structure in the real contrast image. Since directly observing the contrast images generated by different methods does not allow for a direct visual observation of the contrast changes in the tumor region, in... Figure 8 The text further illustrates the subtraction visualization results of abdominal tumor contrast-enhanced MRI images generated by a cross-semantic perception-based method and existing techniques. Brighter tumor areas indicate greater intensity changes between the contrast-enhanced and plain scan images. As shown in columns 1 and 2, only SAGAN clearly shows tumor changes; other methods do not produce particularly noticeable changes. However, the tumor morphology in SAGAN differs significantly from the actual subtraction image, while SANT-GAN's subtraction image is closer to the actual one. In columns 3 and 4, the tumor changes in other comparison methods are extremely subtle, while SAMT-GAN's subtraction image clearly shows intensity changes. In columns 5 and 6, although some methods show brightness changes in the tumor area, the morphology differs significantly from the actual subtraction image. SAMT-GAN's subtraction image is significantly closer to the morphology and structure of the tumor in the actual subtraction image. In summary, the comparison of visualization results from the two methods further illustrates the superiority of the SAMT-GAN proposed in this embodiment of the invention in the task of generating contrast images.

[0185] Ablation Experiment Results and Analysis

[0186] To demonstrate the effectiveness of the modules FEM, MAFM, and SAD proposed in this embodiment, ablation experiments were conducted on a rectal tumor angiography dataset. Since the plain scan images and angiography images in the rectal tumor dataset are almost perfectly aligned at the pixel level, a CycleGAN-based framework was not used; instead, a Pix2Pix framework was employed, using only a single discriminator and generator. The FEM and MAFM modules were introduced into the generator, while SAD was introduced into the discriminator. The specific generator and discriminator frameworks remained unchanged. The carotid plaque angiography dataset was not used for ablation experiments due to pixel misalignment issues. Tables 3 and 4 show the ablation experimental results on the entire image and on local tumor regions, respectively.

[0187] Table 3 shows the ablation experiment results on the entire image.

[0188]

[0189] Table 4. Ablation experimental results in local tumor areas.

[0190]

[0191] The first row represents the baseline Pix2Pix, and the second row represents Pix2Pix+FEM (Model 1). Compared to the baseline Pix2Pix, adding FEM to the generator improved the SSIM from 0.7972 to 0.8041 across the entire image, and increased PSNR and SSIM by 0.3523 and 0.0508 respectively in local tumor regions, while decreasing MAE by 7.9543. The third row represents Pix2Pix+FEM+MAFM (Model 2). Adding MAFM to the second row improved the SSIM from 0.8041 to 0.8221 across the entire image, and increased PSNR and SSIM by 0.2821 and 0.0237 respectively in local tumor regions, while decreasing MAE by 8.977. 7; The fourth row represents Pix2Pix+FEM+SAD (Model 3). After adding SAD in the second row, the SSIM on the entire image increased from 0.8041 to 0.8131, and in the local tumor region, PSNR and SSIM increased by 0.6983 and 0.0242 respectively, while MAE decreased by 12.0539. The fifth row represents Pix2Pix+FEM+MAFM+SAD, which is the SAMT-GAN proposed in this embodiment. After adding MAFM and SAD to the second row, PSNR and SSIM increased by 0.8503 and 0.0289 on the entire image, and in the local tumor region, PSNR and SSIM increased by 1.1121 and 0.0371 respectively, while MAE decreased by 27.9135. This demonstrates the effectiveness and transferability of the modules FEM, MAFM, and SAD proposed in this embodiment for contrast image generation tasks.

[0192] Tables 3 and 4 clearly show that the introduction of the MAFM module significantly improves the SSIM (Score on the entire image and in local tumor regions). In the entire image, the SSIM increases from 0.8041 to 0.8221; in the local tumor region, it increases from 0.3750 to 0.3987. The addition of the MAFM module also clearly shows that the heatmap focuses more on the tumor region, exhibiting a larger activation value. This further demonstrates the effectiveness of the MAFM module, which can fuse features at different scales, allowing the network to focus more on generating the target region, thereby improving the model's generation performance. Tables 3 and 4 also clearly show that the introduction of the SAD discriminator significantly improves the SSIM (Score on the entire image and in local tumor regions). In the entire image, the SSIM increases from 0.8041 to 0.8131; in the local tumor region, it increases from 0.3750 to 0.3992. To verify whether the semantic information introduced by the SAD discriminator proposed in this embodiment of the invention can improve the discriminator's discrimination performance, the test set consisted of 832 2D plain scan images. Each image had two categories: generated contrast images and real contrast images. These 832 sets of real and fake contrast images were input into PatchGAN and SAD, respectively. The features output by PatchGAN were relatively chaotic, while the features output by SAD were relatively clustered. This demonstrates that, guided by semantic features, the proposed SAD discriminator can perform finer-grained discrimination. To further verify whether the module proposed in this embodiment of the invention promotes the training of the discriminator and the generator, the loss curves of the discriminator based on cross-semantic perception and the generator based on feature enhancement and multi-scale attention in the method for generating abdominal tumor contrast magnetic resonance images based on cross-semantic perception are compared with those of the discriminator and generator in the existing Pix2Pix. The discriminator loss curve of Pix2Pix shows frequent sharp rises and falls, especially with significant fluctuations in the latter half of the training. This may be because the generator suddenly generates more deceptive samples at certain stages, causing the discriminator to have difficulty judging for a while, thus increasing the loss. The generator loss decreases more slowly and with slight fluctuations, while the discriminator loss is lower, which may result in the discriminator being slightly stronger than the generator. The loss curve of the SAMT-GAN discriminator based on cross-semantic awareness is smooth overall, without significant oscillations, and continuously decreases with minimal fluctuations. This indicates that the discriminator based on cross-semantic awareness is stably learning to distinguish between real and fake samples. There is no situation where the discriminator based on cross-semantic awareness excessively suppresses the generator based on feature enhancement and multi-scale attention, or the discriminator based on cross-semantic awareness fails. The loss curve of the generator based on feature enhancement and multi-scale attention is smooth with very small fluctuations, showing a good convergence trend. Therefore, it is evident that the module proposed in this embodiment of the invention promotes the training process of both the discriminator and the generator.

[0193] Downstream mission experimental results and analysis

[0194] This invention validates, through downstream tasks, that the generated contrast-enhanced tumor images can improve the accuracy of tumor segmentation. Experimental analysis is conducted on a rectal cancer contrast-enhanced dataset. First, 59 patients with segmentation labels are divided into a training set, a test set, and fine-tuning data. The training set contains 40 cases, the test set contains 9 cases, and the fine-tuning data contains 10 cases. To demonstrate that the generated contrast images can be used for training medical image segmentation, it is assumed that the fine-tuning data does not contain contrast images, only plain scan images. However, all comparison methods can generate contrast images from plain scan images, so these generated contrast images can be used as fine-tuning data. First, the classic UNet is used to train a segmentation model using rectal tumor contrast images. Then, based on the plain scan images of rectal tumors, images with tumor contrast enhancement are synthesized using different generation models. These contrast images are then fine-tuned on the trained segmentation model. Finally, the fine-tuned model is used to predict the segmentation results on the test set, and the segmentation performance is evaluated using Dice, IoU, SEN, PRE, and HD metrics.

[0195] Table 5 compares the segmentation results of angiographic images generated by SAMT-GAN and other generative methods in downstream segmentation tasks.

[0196]

[0197] Table 5 shows the segmentation results of SAMT-GAN and other generative methods on contrast-enhanced images. The first row represents the segmentation results obtained by training directly on plain scan images, with a segmentation Dice of only 53.35% and an IoU of only 40.95%. This shows that it is difficult to segment relatively complete tumor regions from unenhanced images. The second row represents the segmentation results obtained by training on contrast-enhanced images, with a segmentation Dice of 74.17% and an IoU of 60.52%. This indicates that contrast enhancement can effectively improve tumor segmentation performance. The third row represents the segmentation results after fine-tuning using real contrast-enhanced data. In addition, rows 4 to 10 are the segmentation results obtained after fine-tuning the generation method compared with the previous one. The last row is the segmentation result obtained after fine-tuning the generation method SAMT-GAN proposed in this embodiment. The image generated by SAMT-GAN has the highest Dice (69.82%), IoU (55.33%), SEN (79.58%), PRE (67.85%), and HD (4.96). Compared with the segmentation results of the flat scan image, Dice is improved by 16.47% and IoU is improved by 14.38%. Compared with Pix2Pix

[50] , the segmentation Dice of the angiography image generated by SAMT-GAN is improved by 5.34% and IoU is improved by 5.93%. Compared with SAGAN, the segmentation Dice of the angiography image generated by SAMT-GAN is improved by 7.36% and IoU is improved by 6.14%, indicating that SAMT-GAN is better able to notice the multi-scale features of the target region. Compared to RRDB-SeD, SAMT-GAN improved the segmentation Dice of the contrast-enhanced images by 10.78% and the IoU by 9.92%, indicating that SAMT-GAN can extract richer semantic features and generate more realistic images. This demonstrates that the generated contrast-enhanced images provide richer contrast information about the tumor region, enabling the segmentation network to more accurately identify and locate the tumor area, thereby improving segmentation performance. These experimental results further validate the application value of generative models in medical image analysis and provide a feasible solution for contrast-enhanced medical image segmentation.

[0198] To address issues such as high noise and artifacts in images, pixel misalignment between plain and contrast images, and scarce relevant data, this invention first introduces a generator based on feature enhancement and multi-scale attention. A Feature Enhancement Module (FEM) is introduced between the encoder and decoder of a traditional generator, enhancing the representation of local and global features and enabling the network to effectively learn more complex features. Simultaneously, a Multi-Scale Attention Feature Fusion Module (MAFM) is added to the skip connection structure to integrate information from different scales, improving the model's focus on key features. Furthermore, to enable the discriminator to pay more attention to the spatial and semantic features of the image, a Semantic Aware Discriminator (SAD) based on a cross-attention mechanism is introduced, incorporating semantic features of real contrast images as additional guiding information. Finally, extensive comparative experiments on a rectal tumor contrast dataset demonstrate that the proposed SAMT-GAN method generates more realistic contrast images compared to other existing generation methods. Detailed ablation experiments were also conducted on the rectal tumor contrast dataset, showcasing the changes in heatmap visualization results before and after the introduction of the MAFM module, proving the effectiveness of the proposed FEM, MAFM, and SAD. Finally, a downstream segmentation task was used to verify that the generated contrast-enhanced tumor images can improve the accuracy of tumor segmentation. Among all the generation methods, SAMT-GAN generated the contrast-enhanced images with the highest tumor segmentation accuracy after model fine-tuning, further demonstrating the superiority and effectiveness of SAMT-GAN in contrast-enhanced image generation tasks.

[0199] It should be noted that, in the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0200] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for generating contrast-enhanced magnetic resonance images of abdominal tumors based on cross-semantic perception, characterized in that, include: Obtain a realistic flat scan image; The real plain scan images are processed using a pre-trained imaging generation model based on multi-scale attention and cross-semantic perception to output contrast-enhanced magnetic resonance images; wherein, The imaging generation model based on multi-scale attention and cross-semantic perception includes: Generator based on feature enhancement and multi-scale attention G Generators based on feature enhancement and multi-scale attention F Discriminator based on cross-semantic awareness and a discriminator based on cross-semantic awareness ; Generator based on feature enhancement and multi-scale attention: Both feature enhancement and multi-scale attention generators include: an array of residual convolutional blocks that incorporates a feature enhancement module and a skip connection structure that incorporates multi-scale attention feature fusion; The generator G Based on the feature enhancement module and multi-scale attention feature fusion, the contrast image I2 is output in a forward loop for the plain scan image I1, and the contrast image I2 is output in a backward loop for the unenhanced image I3; The generator F Based on the feature enhancement module and multi-scale attention feature fusion, the unenhanced image I3 is output in a backward loop for the contrast image I2, and the unenhanced image I3 is output in a forward loop for the real contrast image I4; The generator based on feature enhancement and multi-scale attention also includes: The encoder, residual convolutional block array, and decoder are connected in sequence, along with a skip connection structure located between the encoder and decoder; among them, The residual convolutional block array includes nine sequentially connected residual convolutional blocks, with feature enhancement modules introduced in the first and sixth residual convolutional blocks; The skip connection structure includes a first multi-scale attention feature fusion module and a second multi-scale attention feature fusion module; Discriminator based on cross-semantic awareness: Both discriminators based on cross-semantic awareness introduce corresponding cross-attention modules, each of which includes a linear layer and several attention heads; The discriminator Based on the cross-attention module, the plain scan image I1 and the unenhanced image I3 are processed to output the plain scan image discrimination result and the discriminator. Adversarial loss for plain scan image I1 and unenhanced image I3, respectively; The discriminator Based on the cross-attention module, the real contrast image I4 and the contrast image I2 are processed to output the contrast image discrimination result and the discriminator. Adversarial loss for contrast-enhanced image I2 and real contrast-enhanced image I4, respectively; The loss function of the imaging generation model based on multi-scale attention and cross-semantic perception during training includes: The loss function is based on a discriminator with cross-semantic perception and a generator based on feature enhancement and multi-scale attention; the loss function of the generator based on feature enhancement and multi-scale attention consists of flat scan adversarial loss, angiography adversarial loss, cycle consistency loss and perceptual loss.

2. The method for generating abdominal tumor contrast-enhanced magnetic resonance images based on cross-semantic perception according to claim 1, characterized in that, The training process of the pre-trained angiography image generation model based on multi-scale attention and cross-semantic perception includes: A magnetic resonance imaging (MRI) contrast-enhanced image dataset of rectal cancer patients is obtained, which includes real plain scan and contrast-enhanced paired data; the MRI contrast-enhanced image dataset is divided into a training sample dataset and a test sample dataset according to a preset ratio; Set the training parameters and input the training sample dataset into the angiography image generation model based on multi-scale attention and cross-semantic perception; Based on the generator's loss function, the generator, which combines feature enhancement and multi-scale attention, is trained using the training sample dataset to obtain the trained generator parameters W. C ; Based on the discriminator's loss function, the cross-semantic awareness-based discriminator is trained using the training sample dataset to obtain the trained discriminator parameters W. D ; Alternately update the generator based on feature enhancement and multi-scale attention and the discriminator based on cross-semantic perception until a preset number of iterations is reached, and output the updated generator parameters and updated discriminator parameters to obtain the trained angiography image generation model based on multi-scale attention and cross-semantic perception. The trained contrast image generation model based on multi-scale attention and cross-semantic perception was tested using a test sample dataset to obtain an evaluation index for the contrast image generation model. If the evaluation index meets the preset requirements, the trained contrast image generation model based on multi-scale attention and cross-semantic perception is used as a pre-trained contrast image generation model based on multi-scale attention and cross-semantic perception. If the evaluation index does not meet the preset requirements, the trained contrast image generation model based on multi-scale attention and cross-semantic perception is trained again until the evaluation index meets the preset requirements, so as to obtain a pre-trained contrast image generation model based on multi-scale attention and cross-semantic perception.

3. The method for generating abdominal tumor contrast-enhanced magnetic resonance images based on cross-semantic perception according to claim 1, characterized in that, The feature enhancement module includes: The following components are connected in sequence: downsampling unit, position encoding unit, first normalization layer, multi-head self-attention unit, second normalization layer, feedforward layer, residual connection unit, upsampling unit, and convolutional block.

4. The method for generating abdominal tumor contrast-enhanced magnetic resonance images based on cross-semantic perception according to claim 1, characterized in that, The first multi-scale attention feature fusion module is located between the encoder and decoder at the original resolution, and the second multi-scale attention feature fusion module is located between the encoder and decoder downsampled to the second resolution.

5. The method for generating abdominal tumor contrast-enhanced magnetic resonance images based on cross-semantic perception according to claim 1, characterized in that, The working process of the multi-scale attention feature fusion module includes: The multi-scale attention feature fusion module will integrate the feature maps corresponding to the encoding layer. Feature maps corresponding to the decoding layer Add them together to output a superimposed feature map. ; For the superimposed feature map Perform different convolutions to obtain feature maps of different scales. , , ; The feature map is obtained by adding feature maps of different scales. For the feature map Global average pooling and channel mapping are performed, and then feature maps are obtained through three convolutions. Feature map Feature map ; The feature map Feature map and feature map The corresponding weight feature maps are obtained by applying the Sigmoid activation function respectively. , , ; The weight feature map , , Feature maps of different scales , , Multiplying the corresponding values ​​yields a weighted result at three different scales. , , ; Weighted results of three different scales , , The summation serves as the output of the multi-scale attention feature fusion module.

6. The method for generating abdominal tumor contrast-enhanced magnetic resonance images based on cross-semantic perception according to claim 1, characterized in that, The working process of the discriminator based on cross-semantic awareness includes: The discriminator based on cross-semantic perception extracts the semantic information of the real angiography image I4 through the intermediate feature layer of the pre-trained CLIP model, and then outputs the first 2D sequence through the linear layer. The contrast image I2 is subjected to four consecutive downsampling operations and layer normalization processing to output the second 2D sequence; Cross-attention is applied to the first 2D sequence and the second 2D sequence, and then a fused feature map is output through layer normalization, feedforward layer and one upsampling operation. The feature maps obtained by performing three downsampling operations on the fused feature map and the imaging image I2 are spliced ​​and integrated in the channel dimension, and then processed by a convolutional layer to output a Patch block with 1 channel. The patch block is evaluated, and the evaluation result is output.

7. The method for generating abdominal tumor contrast-enhanced magnetic resonance images based on cross-semantic perception according to claim 1, characterized in that, The loss function of the discriminator based on cross-semantic awareness is determined by the discriminator based on cross-semantic awareness. Adversarial loss for flat scan image I1 and unenhanced image I3, and a discriminator based on cross-semantic awareness. The adversarial loss is composed of the real contrast image I4 and the contrast image I2.

8. The method for generating abdominal tumor contrast-enhanced magnetic resonance images based on cross-semantic perception according to claim 1, characterized in that, The loss function of the generator based on feature enhancement and multi-scale attention consists of flat scan adversarial loss, angiography adversarial loss, cycle consistency loss and perceptual loss; The cycle consistency loss is confirmed by plain scan image I1, contrast image I2, real contrast image I4, and unenhanced image I3; The perceived loss was confirmed by the plain scan image I1, the unenhanced image I3, the real contrast image I4, and the contrast image I2.

Images