Cross-modal ultrasound contrast image generation method and device based on diffusion model and readable storage medium thereof

Abstract

The invention provides a cross-modal ultrasound contrast image generation method and device based on a diffusion model and a readable storage medium, and belongs to the technical field of medical image processing and computer-aided diagnosis. The method is characterized in that a multi-scale structure sensing mechanism is introduced, and the generation of an anatomical structure is accurately guided through a dual structure guiding attention and self-adaptive structure fusion module by utilizing structure prior information (such as wavelet high-frequency components) of a source domain BUS image; a hybrid feature extraction network, a cascade state space model (SSM) unit and a Kolmogorov-Arnold network (KAN) unit are configured at a bottleneck so as to efficiently capture long-distance dependency and nonlinear features. According to the method, through the synergistic effect of structure guidance and efficient feature extraction, the structure fidelity of the generated CEUS image and the reality sense of capillary details are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical image processing and computer vision, and specifically relates to a cross-modality ultrasound contrast image generation method and device based on a diffusion model and a readable storage medium thereof. More specifically, the present application relates to converting one medical image modality (B-mode ultrasound image, BUS) to another medical image modality (ultrasound contrast image, CEUS) using a generative model, particularly a denoising diffusion probability model (DDPM), and applying it to a computer-aided diagnosis (CAD) system. BACKGROUND

[0002] B-mode ultrasound (BUS) is one of the most widely used basic imaging examination methods in clinical medicine. It can clearly show the anatomical structure, shape and boundary of tissues and organs, and provide key evidence for the preliminary screening of various diseases, due to its advantages of non-invasiveness, non-radiation, low cost and real-time imaging.

[0003] However, BUS imaging mainly relies on the difference in acoustic impedance between different tissues, and has inherent limitations in reflecting tissue blood perfusion and microvascular distribution. For many tumor lesions (such as liver, breast and other occupying lesions), the accurate differentiation of their benignity and malignancy often highly depends on their internal microvascular characteristics. Conventional BUS images are difficult to effectively present these key blood flow information, thereby limiting their application value in the diagnosis of complex cases.

[0004] In order to make up for the deficiency of BUS in blood flow information display, ultrasound contrast (CEUS) technology has been developed in clinical practice. CEUS can visualize the microvascular perfusion of tissues and organs in real time and dynamically by intravenous injection of ultrasound contrast agent (microbubbles), significantly improving the accuracy of diagnosis. However, the application of CEUS in clinical practice is still limited, for example, the contrast agent may cause allergic reactions in a small number of patients, and the examination price is relatively high, resulting in that it has not been as popular as BUS.

[0005] Therefore, in clinical practice, it is an urgent and important clinical value to develop a technology that can directly generate high-quality "pseudo" CEUS images from easily accessible and conventional BUS images through computational methods.

[0006] Currently, deep learning-based image-to-image translation algorithms, such as Generative Adversarial Networks (GANs), have been attempted for this task. However, GANs generally suffer from inherent defects such as training instability, pattern collapse, and poor reproduction of minute details. In recent years, the Denoising Diffusion Probabilistic Model (DDPM) has attracted attention due to its ability to generate more realistic and stable images. However, when dealing with high-noise and structurally complex scenarios such as ultrasound images, standard diffusion models still struggle to accurately guarantee the fidelity of key anatomical structures (such as lesion boundaries and tissue textures), and their performance in accurately generating microvascular details is unsatisfactory, resulting in "pseudo" CEUS images with limited clinical reference value. Summary of the Invention

[0007] This invention provides a method, apparatus, and readable storage medium for generating cross-modal ultrasound contrast images based on a diffusion model. It addresses the problem in existing technologies that, when converting high-noise B-mode ultrasound images into ultrasound contrast images, it is difficult to simultaneously ensure the fidelity of key anatomical structures and the realism of microvascular details in a high-noise environment.

[0008] The core technology of this invention mainly proposes a diffusion model that integrates a multi-scale structure perception mechanism and a hybrid KAN-Mamba bottleneck. Through the synergistic effect of structure prior guidance and efficient feature extraction, it achieves high-fidelity conversion from B-mode ultrasound to ultrasound contrast images.

[0009] In a first aspect, the present invention provides a method for generating cross-modal ultrasound contrast images based on a diffusion model, the method comprising the following steps: Obtain the B-mode ultrasound image to be converted as the source domain image; A generative network based on a denoising diffusion probability model is constructed, which includes an encoder, a bottleneck module, and a decoder. Generative networks are used to convert source domain images into target domain ultrasound contrast images; The steps involved in constructing the generative network include: A multi-scale structure-aware mechanism is introduced into the feature extraction paths of the encoder and decoder. The multi-scale structure-aware mechanism extracts the structural prior information of the source domain image and fuses the structural prior information into the intermediate feature map of the generator network to guide the generation of the image anatomical structure. The bottleneck module is configured as a hybrid feature extraction network, which includes at least cascaded state space model units and KAN units. The state space model units are used to capture long-distance dependencies in the feature sequence, and the KAN units are used to perform nonlinear feature transformation.

[0010] Furthermore, the multi-scale structure perception mechanism includes a dual-structure guided attention module set in the encoder or decoder; the execution process of the dual-structure guided attention module includes: Spatial dimension pooling is performed on the input feature map to generate an internal spatial attention map that captures the feature response region; The source domain image is transformed in the frequency domain to extract high-frequency components containing edge and texture information, and an external structure guide mask is generated based on the high-frequency components. The internal spatial attention map is fused with the external structural guidance mask to obtain the final attention map, which is then applied to the input feature map to achieve weighted control of the features.

[0011] Furthermore, the specific steps for performing frequency domain transformation on the source domain image are as follows: The source domain image is decomposed using two-dimensional discrete wavelet transform to extract high-frequency components in the horizontal, vertical, and diagonal directions. The high-frequency components are then concatenated and passed through a convolutional layer to generate an external structure guide mask.

[0012] Furthermore, the multi-scale structure-aware mechanism also includes an adaptive structure fusion module located at the connection between the decoder and encoder; this module replaces skip connections in the generative network and receives three input features: Deep features from the layer preceding the decoder, skip connection features from the corresponding layer of the encoder, and source domain structural features extracted from the source domain features that are independent and do not share weights with the encoder. The adaptive structural fusion module calculates fusion weights through a gating network, adaptively adjusts the pass rate of source domain structural features using fusion weights, and then splices and fuses them with deep features and skip connection features.

[0013] Furthermore, the bottleneck module adopts a cascaded structure, which includes, in sequence, a first state space model unit, a first KAN unit, a central processing unit, a second KAN unit, and a second state space model unit; The central processing unit includes a time-structure decoupling attention module, which is configured as follows: The source domain image is mapped to a structured position code using a source domain position encoder, and the structured position code is injected into the query vector and key vector of the attention mechanism. The temporal embedding vector of the diffusion model is mapped to gating coefficients, and the attention weight matrix is ​​weighted using the gating coefficients to decouple the influence of noise level and image structure on attention.

[0014] Furthermore, the calculation process for the KAN element is as follows: The input features are linearly projected, and after passing through an activation function, the features are nonlinearly mapped and weighted summed using learnable B-spline basis functions.

[0015] Furthermore, the training process of the generative network employs a composite loss function, which includes: Basic loss: Calculate the L1 distance between the predicted noise component and the residual component and the true value; Perceptual loss: High-level semantic features of the generated image and the real target image are extracted using a pre-trained convolutional neural network and their differences are calculated; Frequency domain loss: The high-frequency components of the generated image and the real target image are extracted using the Fast Fourier Transform and their differences are calculated; Among them, the perceptual loss and frequency domain loss are adjusted by dynamic time weights, which increase as the time step of the diffusion process progresses, in order to strengthen the constraints on detail generation in the later stage of denoising.

[0016] Secondly, the present invention provides a cross-modal ultrasound contrast imaging image generation system based on a diffusion model, comprising: The image acquisition module is used to acquire the B-mode ultrasound image to be converted; The model processing module stores the generative network model trained by the cross-modal ultrasound contrast imaging image generation method described above, which is used to receive B-mode ultrasound images and output ultrasound contrast imaging images. The display module is used to display the generated ultrasound contrast images.

[0017] Thirdly, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to execute the above-described method for generating cross-modal ultrasound contrast images based on a diffusion model.

[0018] Fourthly, the present invention provides a readable storage medium storing a computer program, the computer program including program code for controlling a process to execute the process, the process including the above-described method for generating cross-modal ultrasound contrast images based on a diffusion model.

[0019] The main contributions and innovations of this invention are as follows: 1. Significantly improves structural fidelity and reduces artifacts: This invention utilizes an innovative "multi-scale structure-aware fusion mechanism" (including DGAM and ASFB modules) to use the original BUS image as an accurate structural prior. In particular, the DGAM module uses wavelet transform to extract high-frequency edge information, and the ASFB module adaptively fuses structural features through a gating mechanism. This effectively solves the problems of blurred anatomical structures and unclear lesion boundaries in traditional models during the generation process, significantly improving the structural similarity (SSIM) and pixel accuracy (PSNR) of the generated image.

[0020] 2. Significantly Enhanced Realism of Microvascular Details: Targeting the core microvascular information in CEUS images, this invention employs a "hybrid KAN-Mamba bottleneck" and a "Time-Structure Decoupling Attention (TSDA)" module at the bottleneck, which can efficiently capture long-range dependencies and complex nonlinear features. Combined with the high-frequency FFT loss during training, the model can generate clearer and more continuous microvascular networks, solving the pain point of poor fidelity in the reproduction of minute details in existing models.

[0021] 3. Improved perceptual quality of generated images: The SKM-Net of this invention significantly outperforms existing technologies in the LPIPS (Learned Perceptual Image Domain Similarity) metric, indicating that the generated CEUS images are more realistic and have fewer artifacts in human visual perception. This is due to the synergistic effect of structure guidance and efficient bottlenecks, avoiding "overly smooth" or "stiff" textures.

[0022] 4. Achieved high efficiency and robustness in the model: Compared to traditional Transformers and MLPs, the bottleneck-level Mamba (state-space model) and KAN (Kolmogorov-Arnold network) provide powerful feature extraction capabilities while maintaining lower computational complexity. Furthermore, the TSDA module decouples time (noise) from structure (content), enabling targeted optimization at different denoising stages, thus enhancing model stability and convergence performance.

[0023] 5. Strong clinical applicability and generalization ability: The invention has been validated on large-scale real clinical datasets from multiple centers and multiple devices. The results show that the invention has reached the state-of-the-art (SOTA) level in all key indicators, which proves that the method has good generalization ability and robustness to data from different sources, and provides a more reliable tool for clinical auxiliary diagnosis.

[0024] Details of one or more embodiments of the present invention are set forth in the following drawings and description, so that other features, objects and advantages of the invention will be more readily understood. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a structural diagram of a cross-modal ultrasound image generation diffusion model according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the dual-structure guided attention module (DGAM) according to an embodiment of the present invention; Figure 3This is a schematic diagram of the Adaptive Structure Fusion Module (ASFB) according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the bottleneck part of SKM-Net according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the Mamba module structure according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the Time Decoupling Attention Module (TSDA) structure according to an embodiment of the present invention; Figure 7 This is a visual comparison chart of multiple methods for converting the same set of BUS inputs to CEUS according to an embodiment of the present invention; Figure 8 This is a residual visualization comparison chart between the results generated by each method according to the embodiments of the present invention and the actual CEUS; Figure 9 This is a visualization comparison chart of the baseline model according to an embodiment of the present invention and the combination of modules such as DGAM, ASFB, TSDA and KAN-Mamba bottleneck respectively. Figure 10 This is a comparison chart of the visualization effects of differential residuals under different module configurations according to embodiments of the present invention; Figure 11 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0027] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0028] Example 1 This invention provides a cross-modal ultrasound image generation method that integrates structural perception and hybrid bottlenecks, aiming to solve the problems of insufficient preservation of anatomical structures, blurred details of small blood vessels, and low model efficiency in the existing technology when generating contrast-enhanced ultrasound images (CEUS) from B-mode ultrasound images (BUS).

[0029] Reference Figure 1 This invention proposes a cross-modal ultrasound image generation and diffusion model called SKM-Net, which integrates structure-aware sensing and a hybrid KAN-Mamba bottleneck. This model is based on the Residual Denoising Diffusion Model (RDDM). Its core is a U-Net (U-shaped network) architecture generative network, which includes an encoding layer, a bottleneck module, and a decoding layer.

[0030] The workflow of this method is as follows: During the training phase, the model learns to gradually denoise from a noisy image and a BUS conditional image (source domain image) to reconstruct the CEUS target image. During the inference phase, the model uses a BUS image as a structure-guided condition, starting from pure Gaussian noise, and gradually denoises through iterative sampling (such as DDIM sampling) to finally generate the corresponding CEUS image.

[0031] The core improvement of this invention lies in two key parts of the U-Net generative network: 1. Multi-scale structure perception mechanism: Dual structure-guided attention module (DGAM) is introduced in the encoding and decoding layers, and adaptive structure fusion module (ASFB) is introduced in the decoding layer to replace the traditional skip connections.

[0032] 2. Hybrid KAN-Mamba bottleneck: At the deepest part of U-Net (bottleneck module), a hybrid design of Mamba, KAN and time-structure decoupled attention (TSDA) is adopted.

[0033] The following will explain in detail the composition and function of each core module.

[0034] I. Multi-scale structural sensing mechanism To address the problem of insufficient preservation of structural information in existing image-to-image conversion models, this invention designs a multi-scale structure-aware mechanism. This mechanism utilizes high-resolution BUS images as structural priors to accurately guide the decoding process.

[0035] 1. Dual-path Guided Attention Module (DGAM) Reference Figure 2 The DGAM module is integrated into various stages of the encoder and decoder in the U-Net architecture to force the network to focus on anatomically critical regions at the feature level.

[0036] For the input feature map The calculation process of the DGAM module includes: ① Internal Space Attention Map (M) spat Generation of ) First, the input features Figure X Average pooling (AvgPool) and max pooling (MaxPool) are performed along the channel dimension respectively.

[0037] The two pooled results are concatenated and passed through a shared convolutional layer with a kernel size of 7x7. 7x7 Finally, the internal spatial attention map M is generated by using the sigmoid activation function σ (which normalizes the output values ​​to the (0,1) range and generates attention weights). spat It captures the regions in the feature map where the response is most significant. This process can be summarized as follows:

[0038] in, Represented as input feature map Perform average pooling along the channel dimension. Represented as input feature map Perform max pooling along the channel dimension.

[0039] ② External structure boot mask (M) struct Generation of ) This module receives raw, high-resolution B-mode ultrasound images. BUS As an additional input.

[0040] First, a two-dimensional discrete wavelet transform (DWT) layer is used to process I. BUS The components are decomposed and their high-frequency components are extracted, namely edge and texture information in the horizontal (LH), vertical (HL), and diagonal (HH) directions.

[0041] Concatenate these three high-frequency components to obtain F. high :

[0042] Among them, F high First, a 1x1 convolutional layer is used to adjust the number of channels, followed by a 3x3 convolutional layer for smoothing. Finally, a sigmoid activation function σ is used to generate the external structure guide mask Mstruct. This process can be summarized as follows:

[0043] in, , , These are respectively represented as the original image I BUS Three high-frequency components extracted after performing two-dimensional discrete wavelet transform (DWT); This is represented as a convolutional layer with a kernel size of 3x3, used for smoothing features. This is represented as a convolutional layer with a kernel size of 1x1, with the number of channels adjusted.

[0044] ③Final attention fusion: Internal spatial attention map M spat With external structure boot mask M struct Perform element-wise multiplication ( This yields a fused final attention map M, precisely calibrated by the BUS structure. final =M spat M struct .

[0045] The M final Apply to original features Figure X The output X of the module is obtained above. out =X M final .

[0046] Through this design, the DGAM module enables the model to focus computational resources on regions that are important at the current feature level and contain rich structures (such as blood vessels and lesion boundaries) in the original BUS image, thereby alleviating the problems of structural blurring and loss of detail.

[0047] 2. Adaptive Structural Fusion Block (ASFB) Reference Figure 3 The ASFB module is used to replace the traditional skip connections in U-Net, realizing the intelligent fusion of decoder features and encoder features, and is the core of achieving accurate generation of structural information.

[0048] The ASFB module receives three inputs: d: Deep features from the layer preceding the decoder.

[0049] s: Skip connection features from the corresponding layer of the encoder.

[0050] b: Structural features at the corresponding scale from a separate BUS feature encoder (similar in architecture to the U-Net encoder but without sharing weights). In addition to using the original BUS features b, the pure structural information separated by their wavelet transform is utilized, and the fusion weights are adaptively determined through a gating mechanism.

[0051] The BUS feature encoder shares a similar architecture with the encoder of the main U-Net model, but does not share weights. This encoding layer first contains a 7x7 initial convolutional layer to receive the raw BUS image and project it onto the initial feature dimension. Subsequently, this initial feature map enters multiple cascaded encoder stages. The feature map output before the downsampling operation is the... That is, structural feature b.

[0052] The calculation process of the ASFB module includes: ① Preparation of hybrid guiding features: First, wavelet transform (DWT) is performed on structural feature b to extract high-frequency components b. struct .

[0053] b struct Upsample to the same size as b and concatenate with b to form a hybrid guiding feature b containing the original features and pure structural information. comb .

[0054] The specific steps are as follows: First, the BUS feature encoder will start from the high-resolution BUS condition image. Extract a multi-scale feature pyramid:

[0055] in, It is the initial feature. This is the structural feature map at the i-th scale, where L is the total number of layers. This set of features It contains complete information on the anatomy of the bus, from coarse to fine.

[0056] Then, for structural feature b, which is the corresponding Wavelet decomposition is performed, and its high-frequency components are concatenated with the original features to form a hybrid guiding feature. :

[0057]

[0058] in, This represents a splicing operation. This represents an upsampling operation, used to restore the size of the high-frequency feature map after wavelet transform to be the same as b, making it easier to stitch together.

[0059] ② Generation of gating signals: The three input features d, s, b comb Perform concat operations.

[0060] The concatenated features are fed into a lightweight gate controller, which is preferably composed of a 3x3 convolution, batch normalization (BN), ReLU activation function, a 1x1 convolution and a sigmoid activation function σ.

[0061] The gating network outputs an attention gate G of the same size as bcomb:

[0062] in, Represented as Convolutional layers, Indicated Convolutional layers together form a gating network. Represented as Activation function.

[0063] ③ Feature fusion: The hybrid guidance feature b is controlled by the gating signal G. comb After filtering, the weighted structural feature b is obtained. filtered =b comb G.

[0064] Finally, the filtered structural feature b filtered The features are concatenated with deep features d and skip connection features s, and then fused through a 1x1 convolutional layer to generate the output Y of this decoder stage. out :

[0065] in, Let d be a 1x1 convolutional layer, and s be the deep features of the layer preceding the decoder and the skip connection features of the corresponding layer of the encoder, respectively.

[0066] Through the ASFB module, the model can dynamically and adaptively extract and fuse relevant structural information from BUS images at each scale, ensuring the accurate transmission of structural information during the decoding process.

[0067] II. Hybrid KAN-Mamba Bottleneck To address the efficiency and performance contradiction between traditional CNNs and Transformers at the U-Net bottleneck, this invention designs a novel hybrid network structure.

[0068] Reference Figure 4The bottleneck module employs a novel hybrid network structure, combining KAN and Mamba networks, and integrating a Temporal-Structural Decoupled Attention (TSDA) module at the center of the bottleneck. This combined design replaces the traditional convolutional or Transformer bottleneck, achieving powerful feature extraction and fusion capabilities with low computational cost. The overall design is a cascaded Mamba-KAN-(ResNet+TSDA)-KAN-Mamba structure.

[0069] 1. Mamba module (SSM unit, State Space Model) Reference Figure 5 The Mamba module (a state-space model, SSM) is used to efficiently capture long-range dependencies. In this invention, SSM is the core component of the Mamba module, used to efficiently capture long-range dependencies in image feature sequences.

[0070] When feature map (e.g.) When the U-Net reaches its deepest point, it first enters the Mamba layer for processing. It then undergoes a linear layer projection and passes through the SiLU activation function to obtain the intermediate representation X. proj :

[0071] The SiLU activation function is calculated using the following formula: , It is represented as a linear projection layer.

[0072] Then X proj It is flattened into a feature sequence x and enters the SSM core.

[0073] The SSM core performs transformations through a discretized state-space system, whose recursive formula can be expressed as:

[0074]

[0075] Among them, parameters It is based on the input Dynamic generation allows the model to selectively focus on or forget information based on the content.

[0076] Finally, the output sequence y is linearly projected and connected to the original input through a residual concatenation. Add them together to get the output X of the Mamba layer. mamba :

[0077] Then The feature map will be fed into the KAN layer.

[0078] 2. KAN module (KAN unit) The KAN (Kolmogorov-Arnold Network) module is used to replace the traditional MLP, providing stronger nonlinear modeling capabilities.

[0079] The core computations of the KAN module (such as the KANLinear layer) utilize learnable B-spline basis functions (B_spline) on the input features x. i Perform nonlinear transformation:

[0080] in, It is represented as a B-spline basis function.

[0081] Each of them The core calculation formula for the layer is:

[0082] in, This can be represented as summing over all i-th input neurons. Represented as learnable weights from input i to output j. It is represented as the Spline learnable weights from input i to output j.

[0083] That is, the activation function of each connection is no longer fixed (such as ReLU), but is composed of learnable spline curves.

[0084] like Figure 4 As shown, KANBlock and MambaBlock are used in cascade. After Mamba efficiently captures global dependencies, KAN performs deep non-linear feature extraction.

[0085] This transformed feature The final output is obtained by performing a residual connection with the original input, which has passed through a linear projection layer.

[0086] in, Represented as a linear projection layer, used for matching Dimensions.

[0087] After this, the features enter the bottleneck core, passing through the standard residual network block before entering the Temporally Decoupled Attention (TSDA) module and a second residual network block. In the standard self-attention mechanism of the diffusion model, the attention weights are affected by both the noise level (time t) and the image content (structure), which may lead to excessive focus on structure at a certain noise stage while neglecting the denoising task, or vice versa.

[0088] 3. Time-Structure Decoupling Attention Module (TSDA) Reference Figure 6 The TSDA module, located at the heart of the bottleneck, addresses the mutual interference between noise level (time t) and image content (structure) in standard self-attention mechanisms. For input features... TSDA first calculates the standard Query(Q), Key(K), Value(V):

[0089] in, They are respectively represented as those used for generation The learnable weight matrix.

[0090] The core of the TSDA module is to decouple these two effects: ①Structural decoupling: First, a lightweight structured position encoder (SPE encoder, preferably several cascaded 3x3 convolutions (each followed by a ReLU activation function)) receives the raw BUS image I. BUS (Needs to be resized to match the input feature X) kan Same spatial dimensions to ensure dimensional alignment.

[0091] SPE encoder from I BUS Extracting structural position code P struct (The 3-channel input image is first projected onto the 16-channel intermediate features, and then further extracted into a 32-channel structural location code, i.e.) ).

[0092] When calculating attention, P struct Adding these values ​​to the standard Query(Q) and Key(K) respectively, we get Q' = Q + P. struct and K'=K+P struct This injects an explicit spatial structure prior into the attention mechanism.

[0093] ②Time decoupling: Meanwhile, a small multilayer perceptron (MLP) embeds the temporal embedding vector of the diffusion model. Mapped to a set of time-gated coefficients Gt (after Sigmoid activation):

[0094] in, Represented as Activation function This represents a multilayer perceptron.

[0095] When calculating the final attention weights, the time-gating coefficient Gt is element-wise ( ) operates on the standard attention score matrix ( Then perform Softmax and matrix multiplication with Value(V) on top of that.

[0096] This process can be summarized as follows: .

[0097] In this way, the TSDA module can dynamically adjust the attention weights based on the current time step t (noise level), deciding whether to focus more on the content structure or the noise pattern at the current stage.

[0098] Then it will enter another KAN layer and Mamba layer for cascading processing. This cascading processing can efficiently resolve the contradiction between performance and efficiency.

[0099] III. Model Training and Implementation Details To implement the model of this invention, this embodiment provides a preferred design of training parameters and loss function, as follows: ① Basic framework: Residual denoising diffusion model (RDDM) is adopted. The total diffusion time step T is set to 1000. During inference, the DDIM sampling strategy is adopted, and the number of sampling steps is set to 30.

[0100] ②Loss function: A composite loss function Ltotal is used.

[0101] Basic loss L basic Using L1 loss (mean absolute error), the residuals (P0) of the network predictions are calculated. res ) and noise (P noise ) and the true value (G) res G noise The L1 norm between () and (). Its calculation formula is as follows:

[0102]

[0103] in, It is the actual residual. It is sampled Gaussian noise. ; represent Norm (mean absolute error).

[0104] Enhanced loss L extra This loss is calculated in the final denoised CEUS image predicted by the network. With real CEUS images The enhancement loss consists of two parts: Perceived loss L perc High-level semantic features are extracted using a pre-trained VGG-19 network, and computation is performed. and The L1 distance on the features of the i-th layer of the VGG-19 network. Its formula can be expressed as:

[0105] in, This represents the feature extractor of the i-th layer of the VGG-19 network. It is the weight coefficient corresponding to this layer.

[0106] High frequency loss L hf The method utilizes Fast Fourier Transform (FFT) and a high-pass filter mask (M) to calculate the L1 distance between the two components in the high-frequency components, thereby enhancing edge and texture details. The formula can be expressed as:

[0107] Here, FFT and IFFT represent Fast Fourier Transform and Inverse Fourier Transform, respectively, and M is a high-pass filter mask used to mask low-frequency components. This represents element-wise multiplication.

[0108] Dynamic time weight W t The augmentation loss Lextra is also subject to a dynamic time weight W. t =(1 The adjustment of t / T). This allows the model to perform better in the later stages of the denoising process (when t approaches 0, W... t Approaching point 1) allows for greater focus on optimizing image perceptual quality and high-frequency details. Here, t represents the current diffusion time step, and T represents the total number of steps. Summary for:

[0109] Among them, w perc and w hf It is a balancing weight.

[0110] Total loss .

[0111] ③ Training parameters: All input and target images were processed to a resolution of 256×256.

[0112] The Adam optimizer was used, with an initial learning rate of 8×10⁻⁶. -5 A batch size of 1 is used, combined with 8 steps of gradient accumulation, to achieve an effective batch size of 8.

[0113] The model was trained for a total of 20,000 steps.

[0114] The model weights were smoothed using an exponential moving average (EMA) strategy, with the decay rate ema_decay set to 0.995.

[0115] IV. Feasibility Verification and Beneficial Effects To verify the effectiveness of the embodiments of the present invention, the invention was trained and tested on a large-scale multi-center dataset containing 971 images from 8 medical centers (2 internal and 6 external). Of these, 602 internal data points (168,095 images) were used for training, and 369 internal and external data points (40,459 images) were used for testing. This dataset is highly heterogeneous, covering images acquired from different brands and models of ultrasound equipment, ensuring the robustness and broad applicability of the experimental results.

[0116] 1. Quantitative comparison using the SOTA method As shown in Table 1, the SKM-Net proposed in this invention (referred to as "this invention" in the table) was quantitatively compared with several state-of-the-art (SOTA) methods (CycleGAN, MENTOR, DDPM, Medfusion, RDDM). The results show that the model of this invention achieved the best performance in three key metrics: PSNR (28.3236), SSIM (0.8561), and LPIPS (0.1127). In particular, LPIPS (perceptual similarity) is significantly lower than other methods, demonstrating that the generated images are more realistic and have fewer artifacts in human visual perception.

[0117] Table 1. Quantitative comparison of image transformation with the SOTA method on a multicenter dataset.

[0118] In this table and Table 2 below, PSNR stands for Peak Signal-to-Noise Ratio. It is a commonly used metric to measure the level of image distortion or noise. The higher the value, the closer the pixel values ​​of the generated image are to the real image, and the less distortion there is.

[0119] SSIM: Structural Similarity Index. Measures the similarity between a generated image and a real image in terms of macroscopic structure. A higher value indicates greater structural similarity.

[0120] LPIPS: Learned Perceptual Image Patch Similarity. A metric for measuring the perceptual similarity of images; the lower the score, the more similar and realistic the two images are in human visual perception.

[0121] CHC: Contrast Histogram Correlation. This is a metric used to evaluate the contrast and texture similarity of an image; a higher value indicates a better correlation.

[0122] Bhattacharyya Distance: A measure of the similarity between two probability distributions (usually referring to the histogram distribution of an image). The smaller the distance, the closer the distributions are.

[0123] HSV: Usually refers to a similarity or distance measure calculated in the HSV (Hue, Saturation, Lightness) color space.

[0124] 2. Ablation test proves As shown in Table 2, detailed ablation experiments were conducted to verify the effectiveness of each innovative module of the present invention.

[0125] Table 2. Quantitative comparison of validation of innovations on multicenter datasets

[0126] As can be seen, the PSNR of the baseline model (RDDM) is 26.7546.

[0127] The LPIPS metric deteriorated after adding DGAM or ASFB alone, indicating that strong structural constraints alone may lead to "over-smoothing".

[0128] Adding only the hybrid bottleneck (Baseline+TSDA+Mamba+KAN) improved the PSNR to 27.4228 and reduced the LPIPS to 0.1539, demonstrating the efficiency of the KAN-Mamba-TSDA bottleneck.

[0129] When all structure-aware modules and bottleneck modules work together (i.e., the method of the present invention), all metrics are significantly improved (PSNR reaches 28.3236, LPIPS decreases to 0.1127).

[0130] This strongly demonstrates that there is a powerful positive synergistic effect between the "multi-scale structure perception mechanism" and the "hybrid KAN-Mamba bottleneck" proposed in this invention, and that the combination of the two is necessary to achieve the best generation effect.

[0131] 3. Visualization Results Reference Figure 7 (SOTA comparison and detailed magnified image) This image shows a visualization comparison of multiple methods for converting the same set of BUS inputs into CEUS, with magnified windows set at lesion boundaries and microvessel-rich areas. It is evident that the method of this invention is closer to real CEUS in terms of lesion contour continuity, boundary sharpness, and fine-grained texture reconstruction. It effectively suppresses speckle artifacts and "halo" blurring, and maintains the orientation and connectivity of micro-bifurcated vessels more completely. This subjective observation is consistent with the quantitative results: the method of this invention outperforms the comparative model in PSNR, SSIM, and LPIPS indices, demonstrating its comprehensive advantages in structural similarity and perceived quality.

[0132] Reference Figure 8 (SOTA Comparison Residual Plot) This plot visualizes the residuals between the results generated by each method and the actual CEUS. Compared with other methods, the residual distribution of the method in this invention is more uniform, the overall amplitude is lower, and the "structured residual stripes" along the lesion boundary and blood vessel trajectory are significantly reduced, indicating that the model reconstructs key anatomical boundaries and high-frequency textures more accurately. This is consistent with its advantages in perceptual consistency (lower LPIPS) and structural similarity (higher SSIM).

[0133] Reference Figure 9 (Comparison and detailed magnified images of ablation experimental structures) This image compares the visualization effects of the baseline model with combinations of modules such as DGAM, ASFB, TSDA, and KAN-Mamba bottlenecks, magnified at the same ROI. It can be observed that when strong structural constraints are introduced individually (such as DGAM or ASFB alone), overly smoothed details or harsh textures may appear; adding only the hybrid bottleneck (TSDA + Mamba + KAN) significantly alleviates the lack of global dependence but still results in some local texture loss. After the complete scheme is combined, the lesion boundary transition is more natural, the fine vascular texture is clearer, and background artifacts are further reduced, with visual effects consistent with the quantitative ablation conclusions.

[0134] Reference Figure 10 (Visualization of ablation experiments and residual plots) This figure shows the differential residuals under different module configurations. The baseline model shows obvious boundary-related residuals at the lesion edge and in areas with dense texture; adding only the hybrid bottleneck can reduce the residuals over a large area, but details are still missed; using only structure guidance is prone to over-constrained residuals in uniform areas. After combining the complete scheme, the residuals converge simultaneously globally and locally, and the residual "halo" at the boundary and along the blood vessel are further weakened, indicating that the synergy between the structure perception and hybrid bottleneck modules can simultaneously improve the global modeling and local high-frequency recovery capabilities.

[0135] Example 2 Based on the same concept, this invention also provides a cross-modal ultrasound image generation system that integrates structure perception and hybrid bottlenecks. This system may include an image acquisition module, a model processing module, and a display module. The model processing module stores a generative network model trained using the method described above.

[0136] Example 3 This embodiment also provides an electronic device, see reference. Figure 11 It includes a memory 404 and a processor 402, wherein the memory 404 stores a computer program and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0137] Specifically, the processor 402 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement embodiments of the present invention.

[0138] Memory 404 may include a mass storage device for data or instructions. For example, and not limitingly, memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 404 may include removable or non-removable (or fixed) media. Where appropriate, memory 404 may be internal or external to a data processing device. In a particular embodiment, memory 404 is non-volatile memory. In a particular embodiment, memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0139] The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 402.

[0140] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the diffusion model-based cross-modal ultrasound contrast image generation methods in the above embodiments.

[0141] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402, and the input / output device 408 is connected to the processor 402.

[0142] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 406 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0143] Input / output device 408 is used to input or output information.

[0144] Example 4 This embodiment also provides a readable storage medium storing a computer program, the computer program including program code for controlling a process to execute the process, the process including the diffusion model-based cross-modal ultrasound contrast image generation method according to Embodiment 1.

[0145] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0146] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0147] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products), including software routines, applets, and / or macros, can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer-executable components configured to perform embodiments when the program is run. One or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted that any block of a logical flow may represent a program step, or interconnected logical circuits, blocks and functions, or a combination of program steps and logical circuits, blocks and functions. The software may be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0148] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The above embodiments are merely illustrative of several implementations of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. A method for generating cross-modal ultrasound contrast images based on a diffusion model, characterized in that, Includes the following steps: Obtain the B-mode ultrasound image to be converted as the source domain image; A generative network based on a denoising diffusion probability model is constructed, wherein the generative network includes an encoder, a bottleneck module, and a decoder; The source domain image is converted into a target domain ultrasound contrast image using the generative network. The steps involved in constructing the generative network include: A multi-scale structure-aware mechanism is introduced into the feature extraction path of the encoder and decoder. The multi-scale structure-aware mechanism extracts the structural prior information of the source domain image and fuses the structural prior information into the intermediate feature map of the generator network to guide the generation of image anatomical structure. The bottleneck module is configured as a hybrid feature extraction network, which includes at least cascaded state space model units and KAN units. The state space model units are used to capture long-distance dependencies of feature sequences, and the KAN units are used to perform nonlinear feature transformation.

2. The method for generating cross-modal ultrasound contrast images as described in claim 1, characterized in that, The multi-scale structure perception mechanism includes a dual structure-guided attention module disposed in the encoder or decoder. The execution process of the dual-structure guided attention module includes: Spatial dimension pooling is performed on the input feature map to generate an internal spatial attention map that captures the feature response region; The source domain image is subjected to frequency domain transformation to extract high-frequency components containing edge and texture information, and an external structure guide mask is generated based on the high-frequency components. The internal spatial attention map is fused with the external structural guidance mask to obtain the final attention map, and the final attention map is applied to the input feature map to achieve weighted control of the features.

3. The method for generating cross-modal ultrasound contrast images as described in claim 2, characterized in that, The specific steps for performing frequency domain transformation on the source domain image are as follows: The source domain image is decomposed using two-dimensional discrete wavelet transform to extract high-frequency components in the horizontal, vertical, and diagonal directions. The high-frequency components are then concatenated and passed through a convolutional layer to generate the external structure guide mask.

4. The method for generating cross-modal ultrasound contrast images as described in claim 1, characterized in that, The multi-scale structure perception mechanism further includes an adaptive structure fusion module located at the connection between the decoder and the encoder; the adaptive structure fusion module replaces skip connections in the generator network and receives three input features: Deep features from the layer preceding the decoder, skip connection features from the corresponding layer of the encoder, and source domain structural features extracted from the source domain features that are independent and do not share weights with the encoder. The adaptive structural fusion module calculates fusion weights through a gating network, adaptively adjusts the pass rate of the source domain structural features using the fusion weights, and then splices and fuses them with the deep features and skip connection features.

5. The method for generating cross-modal ultrasound contrast images as described in claim 1, characterized in that, The bottleneck module adopts a cascaded structure, which includes a first state space model unit, a first KAN unit, a central processing unit, a second KAN unit, and a second state space model unit in sequence. The central processing unit includes a time-structure decoupling attention module, which is configured as follows: The source domain image is mapped to a structural position code using a source domain position encoder, and the structural position code is injected into the query vector and key vector of the attention mechanism. The temporal embedding vector of the diffusion model is mapped to gating coefficients, and the attention weight matrix is ​​weighted using the gating coefficients to decouple the influence of noise level and image structure on attention.

6. The method for generating cross-modal ultrasound contrast images as described in claim , characterized in that, The calculation process of the KAN unit is as follows: The input features are linearly projected, and after passing through an activation function, the features are nonlinearly mapped and weighted summed using learnable B-spline basis functions.

7. The method for generating cross-modal ultrasound contrast images as described in any one of claims 1 to 6, characterized in that, The training process of the generative network employs a composite loss function, which includes: Basic loss: Calculate the L1 distance between the predicted noise component and the residual component and the true value; Perceptual loss: High-level semantic features of the generated image and the real target image are extracted using a pre-trained convolutional neural network and their differences are calculated; Frequency domain loss: The high-frequency components of the generated image and the real target image are extracted using the Fast Fourier Transform and their differences are calculated; The perceptual loss and frequency domain loss are adjusted by dynamic time weights, which increase as the diffusion process progresses through time steps, in order to strengthen the constraints on detail generation in the later stages of denoising.

8. A cross-modal ultrasound contrast imaging image generation system based on a diffusion model, characterized in that, include: The image acquisition module is used to acquire the B-mode ultrasound image to be converted; The model processing module stores a generative network model trained by the cross-modal ultrasound contrast imaging image generation method as described in any one of claims 1 to 7, and is used to receive the B-mode ultrasound image and output the ultrasound contrast imaging image. The display module is used to display the generated ultrasound contrast images.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the cross-modal ultrasound contrast image generation method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program, the computer program including program code for controlling a process to execute the process, the process including the cross-modal ultrasound contrast image generation method according to any one of claims 1 to 7.

Images