A medical image cross-modality generation method and device based on wavelet high-frequency enhancement

Abstract

The application discloses a medical image cross-modal generation method and device based on wavelet high-frequency enhancement, and has the following effects: common features and unique features of multi-modal medical images are effectively learned by using multi-scale local-global features and high-frequency texture detail information, and accurate and fine target modal images are obtained. The generation of missing modal images can effectively cope with the limitation of medical conditions and reduce the cost of obtaining multi-modal images, and has great application value; the CMMB block of the global branch encoder aggregates global information and multi-scale local features, fully learns information of different anatomical structures and muscle textures, and generates fine edge texture and tissue details of the target modal image; the RSTB block of the high-frequency branch encoder extracts high-frequency features from the input modal and is aggregated with the global branch to promote the texture fidelity of the generated image and approach the real image; and the CAG mechanism of the decoder promotes the feature interaction between the decoder and the encoder, removes redundant information, and generates efficient images.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision, and particularly relates to a medical image cross-modality generation method and device based on wavelet high-frequency enhancement. BACKGROUND

[0002] In the field of modern medicine, computed tomography (CT) is widely used due to its high spatial resolution and precise electron density information, while magnetic resonance (MR) imaging is known for its excellent soft tissue contrast, especially multi-sequence MR, which can provide complementary anatomical and pathological information. Therefore, in clinical diagnosis, combining multiple MR sequences or fusing MR and CT images can help obtain more comprehensive anatomical details. However, in clinical practice, due to imaging equipment and imaging time limitations, there may be missing sequences when acquiring multi-sequence MR images; at the same time, registration errors when combining MR and CT can introduce spatial uncertainty, and ionizing radiation in the CT imaging process may pose a safety risk to patients. Therefore, generating missing sequences using existing MR sequences or generating CT images from MR images has become an efficient and safe alternative.

[0003] Medical image cross-modality generation is a core technology in medical image analysis and intelligent diagnosis and treatment systems. Its purpose is to generate missing or other related modality medical images from an existing medical image, which can provide multiple modalities of medical images, effectively cope with the limitations of imaging equipment, and improve the accuracy of diagnosis and treatment planning.

[0004] Although traditional machine learning methods can achieve medical image cross-modality generation, such as methods based on tissue segmentation and atlas, these methods are usually difficult to generate detailed textures due to the nonlinear relationship between tissue contrast in different modalities. The rise of deep learning has driven rapid development in this field, and numerous advanced methods have been proposed, achieving efficient and accurate cross-modality image generation. ResViT combines the local learning ability of CNN and the sensitivity of Transformer to global information, and proposes a CNN-Transformer hybrid network to achieve accurate image cross-modality generation. Hi-Net proposes a cross-modality generation network based on autoencoder, which learns the common characteristics of existing modalities and further learns the unique features of missing modalities by fusing them. MTTNet proposes a dual-branch multi-scale CNN-Transformer hybrid network to capture global spatial information between different anatomical structures. Although these methods have achieved cross-modality generation to some extent, they all ignore the inherent characteristics of deep neural networks that prefer to learn low-frequency information, and the lack of high-frequency information will lead to inaccurate tissue boundaries and fine structures, affecting the performance of the generated images.

[0005] Therefore, how to design a more accurate medical image cross-modal generation method is a technical problem to be solved. SUMMARY

[0006] Therefore, it is necessary to provide a medical image cross-modal generation method and device based on wavelet high-frequency enhancement in view of the problems of the prior art.

[0007] In a first aspect, the embodiments of the present application disclose a medical image cross-modal generation method based on wavelet high-frequency enhancement, comprising the following steps:

[0008] S1: constructing a medical image cross-modal generation network based on a Transformer-Mamba hybrid network, wherein the medical image cross-modal generation network comprises a generator and a discriminator composed of a global branch encoder, a high-frequency branch encoder and a decoder;

[0009] S2: performing feature extraction and down-sampling on an original input medical image based on the global branch encoder to obtain global information and multi-scale local information;

[0010] S3: performing wavelet transform on the original input medical image based on the high-frequency branch encoder, modeling and down-sampling the wavelet high-frequency component to obtain high-frequency detail information;

[0011] S4: performing feature aggregation on the high-frequency detail information and the global information in multiple spatial dimensions to obtain aggregated features with high-frequency details;

[0012] S5: repeating steps S2-S4 multiple times along the depth direction of the medical image cross-modal generation network to obtain first output features of the encoder layer;

[0013] S6: performing redundant information screening and splicing on the first output features based on the gating mechanism of the decoder, and performing up-sampling through deconvolution to obtain second output features of the decoder;

[0014] S7: repeating step S6 along the depth direction of the medical image cross-modal generation network to obtain a target modality image;

[0015] S8: performing result calculation on the target modality image based on a loss function, and optimizing the medical image cross-modal generation network.

[0016] Preferably, step S2 comprises:

[0017] S21: performing feature extraction on the original input medical image based on a convolution-Mamba hybrid block of the global branch encoder to obtain initial multi-scale local information and initial global information;

[0018] S22: adaptively aggregating and average-pooling down-sampling the initial multi-scale local information and initial global information to obtain multi-scale global information and local information.

[0019] Preferably, the convolution-Mamba hybrid block comprises a Mamba branch for obtaining the global information and a multi-kernel hole convolution block branch for obtaining the multi-scale local information.

[0020] Preferably, step S3 comprises:

[0021] S31: performing wavelet transform on the original input medical image based on the high-frequency branch encoder, and learning high-frequency subbands of wavelet components by using a residual Swin-Transformer block to obtain high-frequency modeling features;

[0022] S32: average-pooling down-sampling the high-frequency modeling features to obtain high-frequency detail information.

[0023] Preferably, step S4 comprises:

[0024] adding the high-frequency detail information to the global information at each encoder layer and performing feature aggregation to obtain aggregated features with high-frequency details.

[0025] Preferably, step S6 comprises:

[0026] S61: performing redundancy information screening on current layer output features of a current layer encoder based on a cross-attention gating mechanism of the decoder and splicing previous layer output features of a previous layer decoder to obtain first output features;

[0027] S62: up-sampling the screened and spliced first output features by deconvolution to obtain second output features of the decoder.

[0028] Preferably, step S8 comprises:

[0029] S81: calculating a first loss between the target modality image and the real target modality image based on an L1 loss function;

[0030] S82: calculating a second loss between the target modality image and the real target modality image based on an adversarial loss function;

[0031] S83: fusing the first loss and the second loss to obtain an overall loss;

[0032] S84: optimizing the medical image cross-modality generation network based on the overall loss.

[0033] In a second aspect, the embodiments of the present application disclose a medical image cross-modality generation device based on wavelet high-frequency enhancement, comprising:

[0034] a network construction unit configured to construct a medical image cross-modality generation network based on a Transformer-Mamba hybrid network, wherein the medical image cross-modality generation network comprises a generator and a discriminator constituted by a global branch encoder, a high-frequency branch encoder and a decoder;

[0035] a feature extraction unit configured to perform feature extraction on an original input medical image based on the global branch encoder and perform down-sampling to obtain global information and multi-scale local information;

[0036] a high-frequency modeling unit configured to perform wavelet transform on the original input medical image based on the high-frequency branch encoder, model and down-sample wavelet high-frequency components to obtain high-frequency detail information;

[0037] a feature aggregation unit configured to perform feature aggregation on the high-frequency detail information and the global information in multiple spatial dimensions to obtain aggregated features with high-frequency details;

[0038] a deep search unit configured to repeatedly input the feature extraction unit, the high-frequency modeling unit and the feature aggregation unit multiple times along a depth direction of the medical image cross-modality generation network to obtain first output features of an encoder layer;

[0039] a decoding processing unit configured to perform redundant information screening and splicing on the first output features based on a gating mechanism of the decoder and perform up-sampling through deconvolution to obtain second output features of the decoder;

[0040] an image generation unit configured to repeatedly input the decoding processing unit multiple times along the depth direction of the medical image cross-modality generation network to obtain a target modality image;

[0041] a network optimization unit configured to perform result calculation on the target modality image based on a loss function and optimize the medical image cross-modality generation network.

[0042] Compared with the prior art, the present application has the following beneficial effects:

[0043] (1) The present application can effectively learn common features and unique features of multi-modality medical images by using multi-scale local-global features and high-frequency texture detail information from additional high-frequency components, thereby obtaining more accurate and fine target modality image generation results. The generation of missing modality images is realized in a relatively simple and easy-to-implement manner, which not only effectively deals with the limitations of medical conditions but also reduces the cost of obtaining multi-modality images, and has great application value.

[0044] (2) The convolution-Mamba (CMMB) block of the global branch encoder can effectively aggregate global context information and multi-scale local features, and can fully learn accurate information from different size anatomical structures and muscle textures, helping to generate fine edge textures and tissue details in the target modality image.

[0045] (3) The residual Swin-Transformer (RSTB) block of the high-frequency branch encoder can effectively extract high-frequency features from the wavelet high-frequency components of the input modality and aggregate them as auxiliary information with the global branch, explicitly helping the global branch to learn high-frequency information and promoting the texture fidelity of the generated image, making it closer to the real image.

[0046] (4) The cross-attention gating (CAG) mechanism of the decoder can effectively promote the feature interaction between the decoder and the encoder, remove the redundant information between them, and help achieve efficient image generation. BRIEF DESCRIPTION OF DRAWINGS

[0047] The exemplary embodiments of this application can be more fully understood with reference to the following drawings in which:

[0048] Figure 1 A flowchart of a medical image cross-modality generation method based on wavelet high-frequency enhancement provided by the embodiments of the present application;

[0049] Figure 2 A whole network structure diagram of a medical image cross-modality generation method based on wavelet high-frequency enhancement provided by the embodiments of the present application;

[0050] Figure 3 A network structure diagram of a convolution-Mamba (CMMB) hybrid block provided by the embodiments of the present application;

[0051] Figure 4 A network structure diagram of a cross-attention gating (CAG) mechanism provided by the embodiments of the present application;

[0052] Figure 5 An output effect diagram of a medical image cross-modality generation method based on wavelet high-frequency enhancement provided by the embodiments of the present application;

[0053] Figure 6 A structure diagram of a medical image cross-modality generation device based on wavelet high-frequency enhancement provided by the embodiments of the present application. DETAILED DESCRIPTION

[0054] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms without being limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0055] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0056] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0057] Embodiment 1

[0058] With reference to Figure 1 The present embodiment discloses a medical image cross-modal generation method based on wavelet high-frequency enhancement, comprising the following steps:

[0059] S1: constructing a medical image cross-modal generation network based on a Transformer-Mamba hybrid network, the medical image cross-modal generation network comprising a generator and a discriminator composed of a global branch encoder, a high-frequency branch encoder and a decoder;

[0060] Specifically, the discriminator is composed of a 3-layer convolutional neural network (CNN), which takes the generated results of the generator and the real medical images as input, and outputs true or false to discriminate the generated results of the generator.

[0061] S2: performing feature extraction and down-sampling on the original input medical image based on the global branch encoder to obtain global information and multi-scale local information;

[0062] Specifically, step S2 comprises:

[0063] S21: performing feature extraction on the original input image based on the convolution-Mamba hybrid block of the global branch encoder to obtain initial multi-scale local information and initial global information;

[0064] S22: adaptively aggregating the initial multi-scale local information and the initial global information and performing average pooling down-sampling to obtain multi-scale global information and local information.

[0065] Specifically, the convolution-Mamba mixed block (CMMB) block is composed of a parallel double branch, i.e., a Mamba branch for capturing global long-range dependencies and a multi-scale dilated convolution block branch for obtaining multi-scale local features. Then, the global and local features obtained by the two branches are adaptively fused through two learnable parameters and a 1x1 convolution, thereby providing rich contextual information and detailed information of different size anatomical structures, and finally helping to generate more accurate target modality images.

[0066] (1) For the Mamba branch, assuming is the input feature, is the number of layers of the encoder or decoder, which is first processed by a linear layer and then processed by a SiLU activation function to obtain the feature . Second, the is processed by a linear layer, then by a 1x1 depth separable convolution and a SiLU activation function to obtain the intermediate feature , and then by a 2D selection scan (SS2D) module and a normalization layer to obtain the global feature . Subsequently, the is element-wise multiplied with to realize the feature mixing of the two paths and highlight more important feature information. Finally, the result is passed to a linear layer to obtain the output of the Mamba branch, i.e., the global information. This process can be mathematically represented as:

[0067] (1);

[0068] (2);

[0069] (3);

[0070] (4);

[0071] wherein represents a linear layer, represents a SiLU activation function, represents a depth separable convolution with a convolution kernel size of 1x1, represents a normalization layer, represents element-wise multiplication.

[0072] Specifically, SS2D contains cross-scan, S6 block and cross-merge operation. The original medical input image is first traversed along four paths in different directions and unfolded into sequences. The four directions are: from top-left to bottom-right, from bottom-right to top-left, from top-right to bottom-left, from bottom-left to top-right. Then S6 block processes the four sequences in parallel to achieve the function of feature extraction, and finally the four groups of results are merged by summation.

[0073] (2) For the empty convolution block branch, suppose is the input feature, which is first processed by 1x1 convolution to enhance its representation ability, and then four parallel 3x3 empty convolutions are used to obtain four groups of features with different scales , , , The empty rates of the four empty convolutions are 1, 2, 3 and 4 respectively. Then the four groups of features are spliced, and finally a 1x1 convolution is used to adaptively aggregate the multi-scale features to restore the channel number and obtain the output of the empty convolution branch , i.e. multi-scale local information. This process can be mathematically represented as:

[0074] (5);

[0075] (6);

[0076] (7);

[0077] (8);

[0078] (9);

[0079] wherein represents a convolution with a 1x1 convolution kernel, represents an empty convolution with a 3x3 convolution kernel and an empty rate of 1, represents an empty convolution with a 3x3 convolution kernel and an empty rate of 2, represents an empty convolution with a 3x3 convolution kernel and an empty rate of 3, represents an empty convolution with a 3x3 convolution kernel and an empty rate of 4, represents a channel-wise splicing operation.

[0080] Finally, the global information and multi-scale local information obtained by the two branches are used to generate weights corresponding to the global information and multi-scale local information using a fully connected layer, and then fused by adaptive weighting, and then a 1x1 convolution is used to achieve further aggregation. This process can be mathematically represented as:

[0081] (10);

[0082] wherein, represents a convolution with a kernel size of 1x1, and represent two learnable parameters, initialized , and updated by a linear layer, and After aggregation, the CMMB block is passed to obtain .

[0083] In this example, the CMMB block adaptively learns the global context information and multi-scale local information, and further models the features aggregated with the high-frequency branch, to obtain fused features containing more anatomical structure and edge information, so that the network can generate more accurate target modality images.

[0084] S3: performing wavelet transform on the original input medical image based on the high-frequency branch encoder, and modeling and down-sampling the wavelet high-frequency component to obtain high-frequency detail information;

[0085] Specifically, step S3 includes:

[0086] S31: performing wavelet transform on the original input medical image based on the high-frequency branch encoder, and using the residual Swin-Transformer block to learn the high-frequency sub-band of the wavelet component to obtain high-frequency modeling features;

[0087] S32: performing average pooling down-sampling on the high-frequency modeling features to obtain high-frequency detail information.

[0088] In this embodiment, first, the original input medical image is subjected to discrete wavelet transform, and the high-frequency branch encoder models the wavelet high-frequency component of the original medical input image X through the residual Swin-Transformer (RSTB) block. This process can be mathematically represented as:

[0089] (11);

[0090] (12);

[0091] wherein, represents wavelet transform, is the wavelet low-frequency component of the original input medical image, are three wavelet high-frequency components of the original input medical image, is the initial input of the high-frequency branch.

[0092] Specifically, the RSTB block is composed of two 3x3 convolutions and six Swin Transformer (STB) blocks. This combination effectively balances local feature extraction and global dependency modeling, ensuring the preservation of high-frequency details and effective integration into the generation process. Assume the feature of the first module of the high-frequency branch is the input feature, which first goes through a 3x3 convolution for local feature extraction, then goes through a series of Swin Transformer blocks to learn the long-range dependencies of the features, and finally goes through a 3x3 convolution to further enhance the features. Finally, a residual connection is added to obtain the output of the RSTB block. This process can be mathematically represented as:

[0093] (13);

[0094] (14);

[0095] wherein, represents a convolution kernel with a size of 3x3.

[0096] Specifically, the STB block is composed of a series of multi-head self-attention (MSA) and multi-layer perceptron (MLP). Its processing process can be mathematically represented as:

[0097] (15);

[0098] (16);

[0099] wherein, and represent the input and output features of the STB block, is the output of MSA, and LN represents the normalization layer.

[0100] S4: aggregating the high-frequency detail information and the global information in multiple spatial dimensions to obtain aggregated features with high-frequency details;

[0101] Specifically, the features of the corresponding layers of the global branch and the high-frequency branch are aggregated. By adding the high-frequency detail information to the global information at each encoder layer and aggregating the features, aggregated features with high-frequency details are obtained, further improving the accuracy of the generation results. This process can be mathematically represented as:

[0102] (17);

[0103] wherein, represents the feature of the global branch, represents the feature of the high-frequency detail information, Representing set characteristics.

[0104] S5: repeating steps S2-S4 along the depth direction of the medical image cross-modality generation network multiple times to obtain the first output features of the encoder layer;

[0105] Specifically, in this embodiment, the global branch encoder and the high-frequency branch encoder each contain 5 layers, so this step is to repeat steps S2, S3 and S4 along the depth direction of the cross-modality generation network five times to obtain the output features of the encoder part.

[0106] S6: performing redundant information screening and splicing on the first output features based on the gating mechanism of the decoder, and performing upsampling through deconvolution to obtain the second output features of the decoder;

[0107] S61: performing redundant information screening on the current layer output features of the current layer encoder based on the cross-attention gating mechanism of the decoder, and splicing the previous layer output features of the previous layer decoder to obtain the first output features;

[0108] Specifically, the encoder contains a feature extraction process from the shallow layer to the deep layer. Generally, the features learned at the front end of the encoder can be considered as shallow features, and the features at the back end can be considered as deep features. Then the decoder is a mixture of shallow features and deep features. Specifically, shallow features and deep features are a relatively general concept. For example, we can consider that the first two layers of the two branches belong to shallow features, and the last three layers belong to deep features. The processing object of the cross-attention gating mechanism includes two, one is the output of the previous layer, and the other is the output of the encoder part. This operation mainly filters the features connected by the encoder and then splices them with the output of the previous layer.

[0109] S62: performing upsampling on the first output features after screening and splicing through deconvolution to obtain the second output features of the decoder.

[0110] Specifically, the CAG block processes the output features from the encoder part and the output features from the decoder part , captures the dependency between them, and effectively filters out redundant information. Specifically, first pass through 1x1 convolution and ReLU activation function, then element-wise multiplication with . Similarly, first pass through 1x1 convolution and ReLU activation function, then element-wise multiplication with , then add the two parts of features. Then, through 1x1 convolution and Sigmoid activation function, the cross-attention map is obtained, and is applied to realize feature interaction and screening, and then to obtain the output of the CAG block. This process can be mathematically represented as:

[0111] (18);

[0112] (19);

[0113] (20);

[0114] wherein, represents a convolution with a kernel size of 1x1, represents a ReLU activation function, represents element-wise multiplication, represents a Sigmoid activation function, represents a channel-wise concatenation operation, represents a deconvolution operation, is the output result of step S6.

[0115] S7: repeating step S6 along the depth direction of the medical image cross-modality generation network to obtain a target modality image;

[0116] Specifically, step S6 is repeated along the depth direction of the cross-modality generation network multiple times until the output result of the fifth decoder layer is obtained, and a generated target modality image is obtained.

[0117] S8: performing result calculation on the target modality image based on a loss function, and optimizing the medical image cross-modality generation network.

[0118] Specifically, step S8 includes:

[0119] S81: calculating a first loss between the target modality image and the real target modality image based on an L1 loss function;

[0120] S82: calculating a second loss between the target modality image and the real target modality image based on an adversarial loss function;

[0121] S83: fusing the first loss and the second loss to obtain an overall loss;

[0122] S84: optimizing the medical image cross-modality generation network based on the overall loss.

[0123] Specifically, the distance between the generated image and the real image is measured by the L1 loss, and the generated result is optimized by minimizing the distance, so that the generated image is more close to the real image. The adversarial loss is used to prompt the generator to generate better images to confuse the discriminator, while the discriminator constantly improves the ability to distinguish true and false to prompt the generator to continue learning and improvement, and finally get highly realistic target modality generated images.

[0124] The overall loss function of the network is composed of L1 loss and adversarial loss, and the formula is as follows:

[0125] (21) ;

[0126] Wherein, represents the L1 loss function, represents the adversarial loss, represents the hyperparameter, which is set to 100.

[0127] Specifically, the formula of L1 loss function is as follows:

[0128] (22) ;

[0129] Wherein, represents the generated target modality image, represents the real target modality image.

[0130] The formula of adversarial loss is as follows:

[0131] (23) ;

[0132] Wherein, G and D represent the generator and the discriminator respectively.

[0133] Specifically, the final weight can be obtained after training 200 times in S84, and then the medical image cross modality generation network is updated; the purpose of step S8 is to optimize the model by calculating the loss and back propagation, so that the template modality image output by the model in the next round is more accurate.

[0134] Referring to Figure 5 , the output effect of the medical image cross modality generation method based on wavelet high frequency enhancement provided by the embodiment of the application for different types of images is shown. Specifically, Figure 5 (a) is the input MR image, Figure 5 (b) is the CT image generated based on the method of the application, Figure 5 (c) is the real CT image, which is used to calculate the loss function to supervise the learning of the neural network; Figure 5 (d) is the input MR (flair) image, Figure 5 (e) is the MR (tlce) image generated based on the method of the application,Figure 5 (f) a real MR (tlce) image used to compute a loss function to supervise the learning of the neural network.

[0135] Embodiment 2

[0136] With reference to Figure 6 The embodiment of the present application discloses a medical image cross-modality generation device 20 based on wavelet high-frequency enhancement, comprising:

[0137] A network construction unit 201 is configured to construct a medical image cross-modality generation network based on a Transformer-Mamba hybrid network, wherein the medical image cross-modality generation network comprises a generator and a discriminator composed of a global branch encoder, a high-frequency branch encoder and a decoder.

[0138] A feature extraction unit 202 is configured to perform feature extraction and down-sampling on an original input medical image based on the global branch encoder to obtain global information and multi-scale local information.

[0139] A high-frequency modeling unit 203 is configured to perform wavelet transform on the original input medical image based on the high-frequency branch encoder, and model and down-sample wavelet high-frequency components to obtain high-frequency detail information.

[0140] A feature aggregation unit 204 is configured to perform feature aggregation on the high-frequency detail information and the global information in multiple spatial dimensions to obtain aggregated features with high-frequency details.

[0141] A deep search unit 205 is configured to repeatedly input the feature extraction unit, the high-frequency modeling unit and the feature aggregation unit multiple times along the depth direction of the medical image cross-modality generation network to obtain first output features of the encoder layer.

[0142] A decoding processing unit 206 is configured to perform redundant information screening and splicing on the first output features based on the gating mechanism of the decoder, and perform up-sampling through deconvolution to obtain second output features of the decoder.

[0143] An image generation unit 207 is configured to repeatedly input the decoding processing unit multiple times along the depth direction of the medical image cross-modality generation network to obtain a target modality image.

[0144] A network optimization unit 208 is configured to perform result calculation on the target modality image based on a loss function, and optimize the medical image cross-modality generation network.

[0145] The device provided by the embodiments of the present application can implement the method described above. The system can be implemented by software, hardware or a combination of software and hardware. For example, the system can include integrated or separated functional modules or units to perform the corresponding steps in the above methods. In some embodiments of the present application, the system provided by the embodiments of the present application has the same beneficial effects as the method provided by the foregoing embodiments of the present application. Details are not described here.

[0146] It should be noted that the flowcharts and block diagrams in the drawings show the architectural, functional and operational aspects of possible implementations of systems, methods and computer program products according to embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than those noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0147] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the system, device and unit described above can refer to the corresponding processes in the foregoing method embodiments, which are not described here.

[0148] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0149] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0150] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0151] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0152] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should be covered in the scope of the claims and the specification of the present application.

Claims

1. A medical image cross-modality generation method based on wavelet high-frequency enhancement, characterized in that, The method comprises the following steps: S1: constructing a medical image cross-modal generation network based on a Transformer-Mamba hybrid network, wherein the medical image cross-modal generation network comprises a generator and a discriminator composed of a global branch encoder, a high-frequency branch encoder and a decoder; S2: performing feature extraction and down-sampling on an original input medical image based on the global branch encoder to obtain global information and multi-scale local information; S3: performing wavelet transform on the original input medical image based on the high-frequency branch encoder, modeling and down-sampling the wavelet high-frequency component to obtain high-frequency detail information; S4: aggregating the high-frequency detail information and the global information in multiple spatial dimensions to obtain aggregated features with high-frequency details; S5: repeating steps S2-S4 multiple times along the depth direction of the medical image cross-modal generation network to obtain first output features of the encoder layer; S6: performing redundant information screening and splicing on the first output features based on the gating mechanism of the decoder, and performing up-sampling through deconvolution to obtain second output features of the decoder; S7: repeating step S6 along the depth direction of the medical image cross-modal generation network to obtain a target modality image; S8: calculating the result of the target modality image based on a loss function, and optimizing the medical image cross-modal generation network; S21: performing feature extraction on the original input medical image based on a convolution-Mamba hybrid block of the global branch encoder to obtain initial multi-scale local information and initial global information; S22: adaptively aggregating and average pooling down-sampling the initial multi-scale local information and the initial global information to obtain multi-scale global information and local information; The convolution-Mamba hybrid block comprises a Mamba branch for obtaining the global information and a multi-kernel hollow convolution block branch for obtaining the multi-scale local information; Step S3 comprises: S31: performing wavelet transform on the original input medical image based on the high-frequency branch encoder, and learning the high-frequency sub-band of the wavelet component using a residual Swin-Transformer block to obtain high-frequency modeling features; S32: average pooling down-sampling the high-frequency modeling features to obtain high-frequency detail information.

2. The method of claim 1, wherein, Step S4 comprises: Adding the high-frequency detail information to the global information at each encoder layer and performing feature aggregation to obtain aggregated features with high-frequency details.

3. The method of claim 2, wherein, Step S6 comprises: S61: performing redundant information screening on the current layer output features of the current layer encoder based on the cross-attention gating mechanism of the decoder, and splicing the previous layer output features of the previous layer decoder to obtain first output features; S62: up-sampling the first output features through deconvolution to obtain second output features of the decoder.

4. The method of claim 3, wherein, Step S8 comprises: S81: calculating a first loss between the target modality image and a real target modality image based on an L1 loss function; S82: calculating a second loss between the target modality image and the real target modality image based on an adversarial loss function; S83: fuse the first loss and the second loss to obtain an overall loss; S84: optimize the medical image cross-modality generation network based on the overall loss.

5. An apparatus for implementing the method of any one of claims 1 to 4, characterized in that, Comprise: a network construction unit, configured to construct a medical image cross-modality generation network based on a Transformer-Mamba hybrid network, the medical image cross-modality generation network comprising a generator and a discriminator composed of a global branch encoder, a high-frequency branch encoder, and a decoder; a feature extraction unit, configured to perform feature extraction and down-sampling on an original input medical image based on the global branch encoder to obtain global information and multi-scale local information; a high-frequency modeling unit, configured to perform wavelet transform on the original input medical image based on the high-frequency branch encoder, and model and down-sample wavelet high-frequency components to obtain high-frequency detail information; a feature aggregation unit, configured to perform feature aggregation on the high-frequency detail information and the global information in multiple spatial dimensions to obtain aggregated features with high-frequency details; a deep search unit, configured to repeatedly input the feature extraction unit, the high-frequency modeling unit, and the feature aggregation unit multiple times along the depth direction of the medical image cross-modality generation network to obtain first output features of an encoder layer; a decoding processing unit, configured to perform redundant information screening and splicing on the first output features based on the gating mechanism of the decoder, and perform up-sampling through deconvolution to obtain second output features of a decoder; an image generation unit, configured to repeatedly input the decoding processing unit multiple times along the depth direction of the medical image cross-modality generation network to obtain a target modality image; a network optimization unit, configured to perform result calculation on the target modality image based on a loss function, and optimize the medical image cross-modality generation network.

Images