Diffusion model-based cross-modality ultrasound contrast image generation method and device and readable storage medium thereof

Abstract

The application provides a cross-modal ultrasound contrast image generation method and device based on a diffusion model and a readable storage medium thereof, and belongs to the technical field of medical image processing and computer-aided diagnosis. The core lies in that a multi-scale structure perception mechanism is introduced, structural prior information (such as a wavelet high-frequency component) of a source domain BUS image is utilized, a double structure guiding attention and an adaptive structure fusion module are used to accurately guide the generation of an anatomical structure, a hybrid feature extraction network is configured at a bottleneck, a state space model (SSM) unit and a Kolmogorov-Arnold network (KAN) unit are cascaded to efficiently capture long-distance dependence and nonlinear features. Through the synergistic effect of structure guiding and efficient feature extraction, the structural fidelity of the generated CEUS image and the realism of microvessel details are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical image processing and computer vision, and specifically relates to a cross-modality ultrasound contrast image generation method and device based on a diffusion model and a readable storage medium thereof. More specifically, the present application relates to converting one medical image modality (B-mode ultrasound image, BUS) to another medical image modality (ultrasound contrast image, CEUS) using a generative model, particularly a denoising diffusion probability model (DDPM), and applying it to a computer-aided diagnosis (CAD) system. BACKGROUND

[0002] B-mode ultrasound (BUS) is one of the most widely used basic imaging examination methods in clinical medicine. It can clearly show the anatomical structure, shape and boundary of tissues and organs, and provide key evidence for the preliminary screening of various diseases, due to its advantages of non-invasiveness, non-radiation, low cost and real-time imaging.

[0003] However, BUS imaging mainly relies on the difference in acoustic impedance between different tissues, and has inherent limitations in reflecting tissue blood perfusion and microvascular distribution. For many tumor lesions (such as liver, breast and other occupying lesions), the accurate differentiation of their benignity and malignancy often highly depends on their internal microvascular characteristics. Conventional BUS images are difficult to effectively present these key blood flow information, thereby limiting their application value in the diagnosis of complex cases.

[0004] In order to make up for the deficiency of BUS in blood flow information display, ultrasound contrast (CEUS) technology has been developed in clinical practice. CEUS can visualize the microvascular perfusion of tissues and organs in real time and dynamically by intravenous injection of ultrasound contrast agent (microbubbles), significantly improving the accuracy of diagnosis. However, the application of CEUS in clinical practice is still limited, for example, the contrast agent may cause allergic reactions in a small number of patients, and the examination price is relatively high, resulting in that it has not been as popular as BUS.

[0005] Therefore, in clinical practice, it is an urgent and important clinical value to develop a technology that can directly generate high-quality "pseudo" CEUS images from easily accessible and conventional BUS images through computational methods.

[0006] Currently, image-to-image conversion algorithms based on deep learning, such as generative adversarial networks (GANs), have been tried for this task. However, GANs universally suffer from inherent defects such as training instability, mode collapse, and poor restoration of minor details. In recent years, denoising diffusion probabilistic models (DDPMs) have attracted attention due to their ability to generate more realistic and stable images. However, when dealing with ultrasound images, which are highly noisy and complex in structure, standard diffusion models still struggle to accurately ensure the fidelity of key anatomical structures (such as lesion boundaries and tissue textures) and perform poorly in accurately generating microvascular details, resulting in limited clinical reference value of the generated "pseudo" CEUS images. SUMMARY

[0007] The embodiments of the present application provide a diffusion model-based cross-modal ultrasound contrast image generation method, device and readable storage medium thereof, which aims to solve the problem that the prior art is difficult to simultaneously ensure the fidelity of key anatomical structures and the generation authenticity of microvascular details in a high-noise environment when converting high-noise B-mode ultrasound images into ultrasound contrast images.

[0008] The core technology of the present application mainly proposes a diffusion model that combines a multi-scale structure perception mechanism and a hybrid KAN-Mamba bottleneck, which realizes high-fidelity conversion from B-mode ultrasound to ultrasound contrast images through the synergistic effect of structure prior guidance and efficient feature extraction.

[0009] In a first aspect, the present application provides a diffusion model-based cross-modal ultrasound contrast image generation method, which comprises the following steps:

[0010] Obtaining a B-mode ultrasound image to be converted as a source domain image;

[0011] Constructing a generation network based on a denoising diffusion probabilistic model, the generation network comprising an encoder, a bottleneck module and a decoder;

[0012] Converting the source domain image into an ultrasound contrast image of a target domain by using the generation network;

[0013] The step of constructing the generation network comprises:

[0014] Introducing a multi-scale structure perception mechanism in the feature extraction paths of the encoder and the decoder, the multi-scale structure perception mechanism extracting structure prior information of the source domain image and fusing the structure prior information into intermediate feature maps of the generation network to guide the generation of image anatomical structures;

[0015] Configuring the bottleneck module as a hybrid feature extraction network, the hybrid feature extraction network comprising at least a state space model unit and a KAN unit arranged in cascade, the state space model unit being used to capture long-distance dependency relationships of feature sequences, and the KAN unit being used to perform nonlinear feature transformation.

[0016] Further, the multi-scale structure perception mechanism includes a double structure guided attention module arranged in the encoder or the decoder; the execution process of the double structure guided attention module includes:

[0017] performing spatial dimension pooling processing on the input feature map to generate an internal spatial attention map capturing the response region of the feature;

[0018] performing frequency domain transformation on the source domain image to extract high-frequency components containing edge and texture information, and generating an external structure guided mask based on the high-frequency components;

[0019] fuse the internal spatial attention map and the external structure guided mask to obtain a final attention map, and apply the final attention map to the input feature map to achieve weighted regulation of the feature.

[0020] Further, the specific steps of performing frequency domain transformation on the source domain image are:

[0021] performing decomposition on the source domain image using two-dimensional discrete wavelet transform, extracting high-frequency components in horizontal, vertical and diagonal directions, and generating an external structure guided mask by convolving the high-frequency components after splicing.

[0022] Further, the multi-scale structure perception mechanism further includes an adaptive structure fusion module arranged at the connection between the decoder and the encoder; the adaptive structure fusion module is used to replace the skip connection in the generation network, which receives three input features:

[0023] deep features from the previous layer of the decoder, skip connection features from the corresponding layer of the encoder, and source domain structure features extracted from an independent source domain feature encoder which does not share weights with the encoder;

[0024] The adaptive structure fusion module calculates the fusion weight through a gating network, adaptively adjusts the pass rate of the source domain structure features using the fusion weight, and splices and fuses them with the deep features and the skip connection features.

[0025] Further, the bottleneck module adopts a cascade structure, which includes a first state space model unit, a first KAN unit, a center processing unit, a second KAN unit and a second state space model unit in sequence;

[0026] The center processing unit includes a time-structure decoupling attention module, which is configured to:

[0027] map the source domain image to a structure position code using a source domain position encoder, and inject the structure position code into the query vector and key vector of the attention mechanism;

[0028] The time embedding vector of the diffusion model is mapped to a gating coefficient, and the gating coefficient is used to weight the attention weight matrix to decouple the influence of noise level and image structure on attention.

[0029] Further, the calculation process of the KAN unit is:

[0030] The input features are linearly projected, and after the activation function, the features are non-linearly mapped and weighted summed using the learnable B-spline basis function.

[0031] Further, the training process of the generation network uses a composite loss function, which includes:

[0032] The basic loss: calculate the L1 distance between the predicted noise component and the residual component and the true value;

[0033] The perception loss: use a pre-trained convolutional neural network to extract the high-level semantic features of the generated image and the real target image and calculate their difference;

[0034] The frequency domain loss: use the fast Fourier transform to extract the high-frequency components of the generated image and the real target image and calculate their difference;

[0035] The perception loss and the frequency domain loss are adjusted by the dynamic time weight, which increases with the advancement of the time step of the diffusion process, to strengthen the constraint of detail generation in the later denoising stage.

[0036] In a second aspect, the present application provides a diffusion model-based cross-modal ultrasound contrast image generation system, comprising:

[0037] An image acquisition module for acquiring a B-mode ultrasound image to be converted;

[0038] A model processing module storing a generation network model trained by the cross-modal ultrasound contrast image generation method described above, for receiving a B-mode ultrasound image and outputting an ultrasound contrast image;

[0039] A display module for displaying the generated ultrasound contrast image.

[0040] In a third aspect, the present application provides an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to execute the diffusion model-based cross-modal ultrasound contrast image generation method described above.

[0041] In a fourth aspect, the present application provides a readable storage medium, the readable storage medium storing a computer program, the computer program comprising program code for controlling a process to execute the process, the process comprising the diffusion model-based cross-modal ultrasound contrast image generation method described above.

[0042] The main contributions and innovations of the present application are as follows:

[0043] 1. Significantly improve structural fidelity and reduce artifacts: The present application uses the original BUS image as an accurate structural prior through the innovative "multi-scale structure perception fusion mechanism" (including DGAM and ASFB modules). In particular, the DGAM module uses wavelet transform to extract high-frequency edge information, and the ASFB module adaptively fuses structural features through a gating mechanism, effectively solving the problem of blurred anatomical structure and unclear lesion boundary in the generation process of traditional models, significantly improving the structural similarity (SSIM) and pixel accuracy (PSNR) of the generated image.

[0044] 2. Significantly enhance the realism of microvascular details: For the core microvascular information of CEUS images, the "mixed KAN-Mamba bottleneck" and "time-structure decoupling attention (TSDA)" modules designed at the bottleneck can efficiently capture long-distance dependencies and complex nonlinear features. Combined with the FFT high-frequency loss in training, the model can generate clearer and more continuous microvascular networks, solving the pain point of poor restoration of existing models in micro details.

[0045] 3. Improve the perceptual quality of the generated image: The SKM-Net of the present application significantly outperforms existing technologies in LPIPS (learned perceptual image similarity) indicators, indicating that the generated CEUS image is more realistic and has fewer artifacts in human visual perception. This is due to the synergistic effect of structure guidance and efficient bottleneck, avoiding "over-smoothing" or "rigid" textures.

[0046] 4. Achieve high efficiency and robustness of the model: Compared with traditional Transformer and MLP, Mamba (state space model) and KAN (Kolmogorov-Arnold network) in the bottleneck provide strong feature extraction capability while maintaining low computational complexity. In addition, the TSDA module decouples time (noise) and structure (content), allowing the model to be optimized in different denoising stages, enhancing the stability and convergence effect of the model.

[0047] 5. Strong clinical applicability and generalization ability: The present application is verified on a large-scale real clinical dataset from multiple centers and devices, and the results show that the present application achieves SOTA (industry-leading) level in various key indicators, proving that the method has good generalization ability and robustness for data from different sources, providing a more reliable tool for clinical auxiliary diagnosis.

[0048] The details of one or more embodiments of the present application are presented in the following drawings and description, so that other features, objects and advantages of the present application are more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0050] Figure 1 is a structure diagram of a cross-modality ultrasound image generation diffusion model according to an embodiment of the present application;

[0051] Figure 2 is a structure diagram of a dual structure guided attention module (DGAM) according to an embodiment of the present application;

[0052] Figure 3 is a structure diagram of an adaptive structure fusion block (ASFB) according to an embodiment of the present application;

[0053] Figure 4 is a structure diagram of a SKM-Net bottleneck part according to an embodiment of the present application;

[0054] Figure 5 is a structure diagram of a Mamba module according to an embodiment of the present application;

[0055] Figure 6 is a structure diagram of a time-decoupled attention module (TSDA) according to an embodiment of the present application;

[0056] Figure 7 is a comparison diagram of multi-method visualization of the same group of BUS input converted to CEUS according to an embodiment of the present application;

[0057] Figure 8 is a comparison diagram of residual error between the results generated by each method and the real CEUS according to an embodiment of the present application;

[0058] Figure 9 is a comparison diagram of the visualization effects of the baseline model and the combination of DGAM, ASFB, TSDA and KAN-Mamba bottleneck modules respectively according to an embodiment of the present application;

[0059] Figure 10 is a comparison diagram of the visualization effects of the differential residual error under different module configurations according to an embodiment of the present application;

[0060] Figure 11 is a structure diagram of a hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0061] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements, unless the context clearly dictates otherwise. The following description of exemplary embodiments is not representative of all possible embodiments consistent with one or more aspects of the present specification. Rather, it is merely an example of apparatus and methods consistent with some aspects of one or more embodiments of the present specification as detailed in the appended claims.

[0062] It should be noted that the steps of the respective methods are not necessarily performed in the order shown and described in the present specification in other embodiments. In some other embodiments, the steps included in the methods can be more or less than described in the present specification. Furthermore, a single step described in the present specification can be broken down into multiple steps in other embodiments; and multiple steps described in the present specification can be combined into a single step in other embodiments.

[0063] Embodiment One

[0064] The embodiment of the present application provides a cross-modal ultrasound image generation method fusing structure perception and a hybrid bottleneck, aiming at solving the problems of insufficient anatomical structure preservation, blurred microvessel details and low model efficiency when generating an ultrasound contrast image (CEUS) from a B-mode ultrasound image (BUS) in the prior art.

[0065] Reference Figure 1 , the embodiment of the present application proposes a cross-modal ultrasound image generation diffusion model named SKM-Net fusing structure perception and a hybrid KAN-Mamba bottleneck. The model takes a residual denoising diffusion model (RDDM) as a basic framework. Its core is a U-Net (U-shaped network) architecture generation network, which includes an encoding layer, a bottleneck module and a decoding layer.

[0066] The workflow of the method is as follows: in the training stage, the model learns to start from a noisy image and a BUS conditional image (source domain image), and gradually denoises to reconstruct a CEUS target image. In the inference stage, the model takes a BUS image as a structure guiding condition, starts from a pure Gaussian noise, gradually denoises through iterative sampling (such as DDIM sampling), and finally generates a corresponding CEUS image.

[0067] The core improvement of the present application lies in two key parts of the U-Net generation network:

[0068] 1. Multi-scale structure perception mechanism: double structure guiding attention modules (DGAM) are introduced in the encoding layer and the decoding layer, and an adaptive structure fusion module (ASFB) is introduced in the decoding layer to replace the traditional skip connection.

[0069] 2. Hybrid KAN-Mamba bottleneck: A hybrid design of Mamba, KAN, and time-structure decoupled attention (TSDA) is adopted at the deepest part of the U-Net (bottleneck module).

[0070] The composition and functions of each core module will be described in detail below.

[0071] I. Multi-scale structure perception mechanism

[0072] To solve the problem of insufficient structural information preservation in existing image-to-image conversion models, the present invention designs a multi-scale structure perception mechanism that uses high-resolution BUS images as structural priors to accurately guide the decoding process.

[0073] 1. Dual-path guided attention module (DGAM)

[0074] Reference Figure 2 The DGAM module is integrated into each stage of the encoder and decoder of the U-Net architecture to force the network to focus on key anatomical regions at the feature level.

[0075] For input feature map The calculation process of the DGAM module includes:

[0076] ① Generation of internal spatial attention map (M spat ):

[0077] First, the input feature Figure X is respectively averaged pooled (AvgPool) and max-pooled (MaxPool) along the channel dimension.

[0078] Concatenate the two pooled results and pass them through a shared convolution layer with a kernel size of 7x7 (f 7x7 ), and finally pass them through a Sigmoid activation function σ (normalize the output value to the range (0, 1) to generate attention weights), to generate the internal spatial attention map M spat , which captures the most significant regions in the feature map. This process can be summarized as:

[0079]

[0080] where, represents the average pooling operation along the channel dimension for input feature map , represents the max-pooling operation along the channel dimension for input feature map .

[0081] ② External structure boot mask (M) struct Generation of )

[0082] This module receives raw, high-resolution B-mode ultrasound images. BUS As an additional input.

[0083] First, a two-dimensional discrete wavelet transform (DWT) layer is used to process I. BUS The components are decomposed and their high-frequency components are extracted, namely edge and texture information in the horizontal (LH), vertical (HL), and diagonal (HH) directions.

[0084] Concatenate these three high-frequency components to obtain F. high :

[0085]

[0086] Among them, F high First, a 1x1 convolutional layer is used to adjust the number of channels, followed by a 3x3 convolutional layer for smoothing, and finally, a sigmoid activation function σ is used to generate the external structure guide mask Mstruct. This process can be summarized as follows:

[0087]

[0088] in, , , These are respectively represented as the original image I BUS Three high-frequency components extracted after performing two-dimensional discrete wavelet transform (DWT); This is represented as a convolutional layer with a kernel size of 3x3, used for smoothing features. This is represented as a convolutional layer with a kernel size of 1x1, with the number of channels adjusted.

[0089] ③Final attention fusion:

[0090] Internal spatial attention map M spat With external structure boot mask M struct Perform element-wise multiplication ( This yields a fused final attention map M, precisely calibrated by the BUS structure. final =M spat M struct .

[0091] The M final Apply to original features Figure X The output X of the module is obtained above. out =X M final .

[0092] With this design, the DGAM module enables the model to concentrate computational resources on regions that are both important at the current feature level and contain rich structures (e.g. blood vessels, lesion boundaries) in the original BUS image, thereby alleviating the problems of structure blurring and detail loss.

[0093] 2. Adaptive Structural Fusion Block (ASFB)

[0094] Reference Figure 3 The ASFB module is used to replace the traditional skip connection in the U-Net, enabling intelligent fusion of decoder features and encoder features, and is the core of accurate structure information generation.

[0095] The ASFB module receives three inputs:

[0096] d: deep features from the previous layer of the decoder.

[0097] s: skip connection features from the corresponding layer of the encoder.

[0098] b: structure features of the corresponding scale from an independent BUS feature encoder (similar to the U-Net encoder architecture but not sharing weights). Not only the original BUS feature b is used, but also the pure structure information separated by wavelet transform, and the fusion weight is adaptively determined by the gating mechanism.

[0099] Wherein, the BUS feature encoder shares a similar architecture with the encoder of the main U-Net model, but does not share weights, wherein the encoding layer first contains a 7x7 initial convolutional layer to receive the original BUS image and project it to the initial feature dimension, then the initial feature map enters multiple cascaded encoder stages, and the feature map output before downsampling operation is , that is, the structure feature b.

[0100] The calculation process of the ASFB module includes:

[0101] ① Preparation of mixed guidance features:

[0102] First, perform wavelet transform (DWT) on the structure feature b to extract the high-frequency component b struct .

[0103] Upsample b struct to the same size as b and concatenate it with b to form a mixed guidance feature b comb containing original features and pure structure information.

[0104] The specific steps are:

[0105] First, the BUS feature encoder extracts a multi-scale feature pyramid from the high-resolution BUS condition image

[0106]

[0107] where, is the initial feature, is the i-th scale structural feature map, and L is the total number of layers. This set of features contains complete BUS anatomical structure information from coarse to fine.

[0108] Then, the structural feature b, that is, the corresponding is wavelet decomposed, and its high-frequency component and the original feature are spliced to form a hybrid guide feature :

[0109]

[0110]

[0111] where, denotes the splicing operation, denotes the up-sampling operation, which is used to restore the size of the wavelet-transformed high-frequency feature map to the same size as b, facilitating splicing.

[0112] ② Generation of the gating signal:

[0113] The three input features d, s, and b comb are spliced (Concat).

[0114] The spliced features are sent to a lightweight gate network (Gate Controller), which is preferably composed of a 3x3 convolution, batch normalization (BN), a ReLU activation function, a 1x1 convolution, and a Sigmoid activation function σ.

[0115] The gate network outputs an attention gate G with the same size as bcomb:

[0116]

[0117] where, denotes the convolution layer of , and denotes the convolution layer of together constitute the gate network, denotes the activation function.

[0118] ③ Feature fusion: ​

[0119] The hybrid guidance feature b is controlled by the gating signal G. comb After filtering, the weighted structural feature b is obtained. filtered =b comb G.

[0120] Finally, the filtered structural feature b filtered The features are concatenated with deep features d and skip connection features s, and then fused through a 1x1 convolutional layer to generate the output Y of this decoder stage. out :

[0121]

[0122] in, Let d be a 1x1 convolutional layer, and s be the deep features of the layer preceding the decoder and the skip connection features of the corresponding layer of the encoder, respectively.

[0123] Through the ASFB module, the model can dynamically and adaptively extract and fuse relevant structural information from BUS images at each scale, ensuring the accurate transmission of structural information during the decoding process.

[0124] II. Hybrid KAN-Mamba Bottleneck

[0125] To address the efficiency and performance trade-off between traditional CNNs and Transformers at the U-Net bottleneck, this invention designs a novel hybrid network structure.

[0126] Reference Figure 4 The bottleneck module employs a novel hybrid network structure, combining KAN and Mamba networks, and integrating a Temporal-Structural Decoupled Attention (TSDA) module at the center of the bottleneck. This combined design replaces the traditional convolutional or Transformer bottleneck, achieving powerful feature extraction and fusion capabilities with low computational cost. The overall design is a cascaded Mamba-KAN-(ResNet+TSDA)-KAN-Mamba structure.

[0127] 1. Mamba module (SSM unit, State Space Model)

[0128] Reference Figure 5 The Mamba module (a state-space model, SSM) is used to efficiently capture long-range dependencies. In this invention, SSM is the core component of the Mamba module, used to efficiently capture long-range dependencies in image feature sequences.

[0129] When the feature map (e.g. ) reaches the deepest layer of the U-Net, it will first enter the Mamba layer for processing. First, it will pass through a linear layer projection and pass through the SiLU activation function to obtain the intermediate representation X proj :

[0130]

[0131] where the SiLU activation function is calculated as , is represented as a linear projection layer.

[0132] Then X proj is flattened into a feature sequence x and enters the SSM core.

[0133] The SSM core is transformed by a discretized state space system, and its recursive formula can be expressed as:

[0134]

[0135]

[0136] where the parameter is dynamically generated according to the input , so that the model can selectively focus on or forget information according to the content.

[0137] Finally, the output sequence y is projected linearly and added to the original input through a residual connection to obtain the output X mamba of the Mamba layer:

[0138]

[0139] Then the feature map will enter the KAN layer.

[0140] 2. KAN module (KAN unit)

[0141] The KAN (Kolmogorov-Arnold Network) module is used to replace the traditional MLP, providing stronger non-linear modeling capabilities.

[0142] The core calculation of the KAN module (such as the KANLinear layer) uses a learnable B-spline basis function (B_spline) to perform non-linear transformation on the input feature x i :

[0143]

[0144] where is represented as a B-spline basis function.

[0145] where each The core formula of the layer is:

[0146]

[0147] where, is represented as a summation over all input neurons i, is represented as a learnable weight from input i to output j, is represented as a Spline learnable weight from input i to output j.

[0148] That is, the activation function of each connection is no longer fixed (such as ReLU), but is composed of a learnable spline curve.

[0149] As Figure 4 shown, KANBlock is used in cascade with MambaBlock, and after Mamba efficiently captures global dependencies, KAN performs deep nonlinear feature refinement.

[0150] This transformed feature is connected in residual with the original input through a linear projection layer to obtain the final output:

[0151]

[0152] where, is represented as a linear projection layer to match the dimension of .

[0153] After this, the feature enters the bottleneck core, and after the standard residual network block, it enters the time-decoupled attention module (TSDA) and the second residual network block. In the diffusion model, the standard self-attention mechanism is affected by both the noise level (time t) and the image content (structure), which may cause excessive attention to the structure and ignore the denoising task at a certain noise stage, or vice versa.

[0154] 3. Time-structure decoupled attention module (TSDA)

[0155] Referring to Figure 6 , the TSDA module is located at the center of the bottleneck, which is used to solve the problem of mutual interference between the noise level (time t) and the image content (structure) in the standard self-attention mechanism. For input feature , TSDA first calculates the standard Query (Q), Key (K), Value (V):

[0156]

[0157] in, They are respectively represented as those used for generation The learnable weight matrix.

[0158] The core of the TSDA module is to decouple these two effects:

[0159] ①Structural decoupling:

[0160] First, a lightweight structured position encoder (SPE encoder, preferably several cascaded 3x3 convolutions (each followed by a ReLU activation function)) receives the raw BUS image I. BUS (Needs to be resized to match the input feature X) kan Same spatial dimensions to ensure dimensional alignment.

[0161] SPE encoder from I BUS Extracting structural position code P struct (The 3-channel input image is first projected onto the 16-channel intermediate features, and then further extracted into a 32-channel structural location code, i.e.) ).

[0162] When calculating attention, P struct Adding these values ​​to the standard Query(Q) and Key(K) respectively, we get Q' = Q + P. struct and K'=K+P struct This injects an explicit spatial structure prior into the attention mechanism.

[0163] ②Time decoupling:

[0164] Meanwhile, a small multilayer perceptron (MLP) embeds the temporal embedding vector of the diffusion model. Mapped to a set of time-gated coefficients Gt (after Sigmoid activation):

[0165]

[0166] in, Represented as Activation function This represents a multilayer perceptron.

[0167] When calculating the final attention weights, the time-gating coefficient Gt is element-wise ( ) operates on the standard attention score matrix ( Then perform Softmax and matrix multiplication with Value(V) on top of that.

[0168] This process can be summarized as follows: .

[0169] In this way, the TSDA module can dynamically adjust the attention weight according to the current time step t (noise level) to determine whether to pay more attention to the content structure or the noise pattern at the current stage.

[0170] Then, the KAN layer and the Mamba layer are entered again for cascade processing, and the cascade processing can efficiently solve the contradiction between performance and efficiency.

[0171] III. Model training and implementation details

[0172] To implement the model of the application, the preferred training parameters and loss function design are provided in this embodiment, as follows:

[0173] 1. Basic framework: residual denoising diffusion model (RDDM) is adopted. The total diffusion time step T is set to 1000. During inference, the DDIM sampling strategy is adopted, and the sampling step number is set to 30.

[0174] 2. Loss function: a set of composite loss functions Ltotal is adopted.

[0175] Basic loss L basic : L1 loss (mean absolute error) is adopted to calculate the L1 norm between the network-predicted residual (P res ) and noise (P noise ) and the true value (G res , G noise ). The calculation formula is as follows:

[0176]

[0177]

[0178] wherein, is the true residual, is the sampled Gaussian noise ; represents norm (mean absolute error).

[0179] Enhanced loss L extra : the loss is calculated between the network-predicted final denoised CEUS image and the true CEUS image . The enhanced loss consists of two parts:

[0180] Perception loss L perc : a pre-trained VGG-19 network is used to extract high-level semantic features, and the L1 distance between and on the i-th layer features of the VGG-19 network is calculated. The formula can be expressed as:

[0181]

[0182] where, represents the feature extractor of the i-th layer of VGG-19 network, is the corresponding weight coefficient of the layer.

[0183] High frequency loss L hf : The L1 distance of the two in high frequency components is calculated by using fast Fourier transform (FFT) and high pass filter mask (M) to strengthen the edge and texture details. Its formula can be expressed as:

[0184]

[0185] where FFT and IFFT represent fast Fourier transform and inverse transform respectively, M is a high pass filter mask used to shield low frequency components, represents element-wise multiplication.

[0186] Dynamic time weight W t : The enhanced loss Lextra is also subject to a dynamic time weight W t =(1 t / T). This makes the model pay more attention to the optimization of the perceptual quality and high frequency details of the image in the later stage of the denoising process (t approaches 0, W t approaches 1). Where t is the current diffusion time step, T is the total number of steps. Summary is:

[0187]

[0188] where w perc and w hf are balance weights.

[0189] Total loss .

[0190] ③ Training parameters:

[0191] All input and target images are processed to 256x256 resolution.

[0192] The Adam optimizer is used, and the initial learning rate is set to 8x10 -5 . A batchsize of 1 is used, combined with 8-step gradient accumulation to achieve an effective batch size of 8.

[0193] The model is trained for a total of 20000 steps.

[0194] The exponential moving average (EMA) strategy is used to smooth the model weights, and the decay rate ema_decay is set to 0.995.

[0195] IV. Feasibility verification and beneficial effects

[0196] To verify the effectiveness of the embodiments of the present application, the present application is trained and tested on a large-scale multi-center dataset containing 971 image data from 8 medical centers (2 internal and 6 external). Among them, 602 internal data (a total of 168095 image pictures) are used for training, and 369 internal and external data (a total of 40459 image pictures) are used for testing. The dataset has high heterogeneity, covering images collected from different brands and different models of ultrasound equipment, ensuring the robustness and wide applicability of the experimental results.

[0197] 1. Quantitative comparison of SOTA method

[0198] As shown in Table 1, the SKM-Net proposed by the present application (“the present application” in the table) is quantitatively compared with various SOTA methods (CycleGAN, MENTOR, DDPM, Medfusion, RDDM). The results show that the model of the present application achieves the best performance in the three key indicators of PSNR (28.3236), SSIM (0.8561) and LPIPS (0.1127). In particular, the LPIPS (perceptual similarity) is much lower than other methods, proving that the generated images are more realistic and have fewer artifacts in human visual perception.

[0199] Table 1 Quantitative comparison of image conversion with SOTA methods on multi-center dataset

[0200]

[0201] In this table and Table 2 below, PSNR: Peak Signal-to-Noise Ratio. A commonly used indicator to measure image distortion or noise level. The higher the value, the closer the pixel value of the generated image to the true image, and the smaller the distortion.

[0202] SSIM: Structural Similarity Index. Measures the similarity of the generated image in macro structure with the true image. The higher the value, the more similar the structure.

[0203] LPIPS: Learned Perceptual Image Patch Similarity. An indicator to measure the perceptual similarity of images. The lower the score, the more similar and realistic the two images are in human visual perception.

[0204] CHC: Contrast Histogram Correlation. It is an index for evaluating the similarity of image contrast and texture, the higher the value, the better the correlation.

[0205] Bhattacharyya Distance: An index for measuring the similarity of two probability distributions (usually referring to the histogram distribution of images here), the smaller the distance, the closer the distribution.

[0206] HSV: Usually refers to some similarity or distance metric calculated in the HSV (Hue, Saturation, Value) color space.

[0207] 2. Ablation test

[0208] As shown in Table 2, in order to verify the effectiveness of each innovative module of the present application, detailed ablation experiments were carried out.

[0209] Table 2 Quantitative comparison of innovative points on multi-center dataset

[0210]

[0211] It can be seen that the PSNR of the baseline model (Baseline, i.e. RDDM) is 26.7546.

[0212] After adding DGAM or ASFB alone, the LPIPS index deteriorates, indicating that strong structural constraints alone may cause "over-smoothing".

[0213] Only add mixed bottleneck (Baseline+TSDA+Mamba+KAN), PSNR is improved to 27.4228, LPIPS is reduced to 0.1539, which proves the high efficiency of KAN-Mamba-TSDA bottleneck.

[0214] When all structure perception modules and bottleneck modules work together (i.e. the method of the present application), all indicators are greatly improved (PSNR reaches 28.3236, LPIPS is reduced to 0.1127).

[0215] This strongly proves that there is a strong positive synergistic effect between the "multi-scale structure perception mechanism" and the "mixed KAN-Mamba bottleneck" proposed in the present application, and only the combination of the two can achieve the best generation effect.

[0216] 3. Visualization results

[0217] Reference Figure 7(SOTA comparison and detail magnification figure), which shows the multi-method visualization comparison of converting the same set of BUS input to CEUS, and sets a magnification window at the lesion boundary and microvascular rich area. It can be seen that the method of the application is closer to the real CEUS in terms of lesion contour continuity, boundary sharpness and fine grain texture reconstruction, and can effectively suppress speckle artifacts and "halo" blur, and the trend and connectivity of the micro-divided blood vessels are more complete. The subjective observation is consistent with the quantitative results: in the PSNR, SSIM and LPIPS indicators, the method of the application is better than the comparison model, showing its comprehensive advantages in structural similarity and perceptual quality.

[0218] Reference Figure 8 (SOTA residual error figure), which gives the residual error visualization between the results generated by each method and the real CEUS. Compared with other methods, the residual error distribution of the method of the application is more uniform, the overall amplitude is lower, and the "structured residual stripe" along the lesion boundary and blood vessel track is significantly reduced, indicating that the model is more accurate in reconstructing key anatomical boundaries and high-frequency textures. This is consistent with its advantages in perceptual consistency (lower LPIPS) and structural similarity (higher SSIM).

[0219] Reference Figure 9 (Ablation experiment structure comparison and detail magnification figure), which compares the visualization effects of the baseline model and the combination of DGAM, ASFB, TSDA and KAN-Mamba bottleneck modules respectively, and zooms in on the same ROI. It can be observed that: when introducing strong structure constraints alone (such as only DGAM or only ASFB), there may be excessive smoothing or harsh texture at the details; only adding a hybrid bottleneck (TSDA + Mamba + KAN) can significantly alleviate the lack of global dependence but still have local texture missing. After the complete scheme is coordinated, the lesion boundary transition is more natural, the blood vessel fine branch texture is clearer, and the background artifact is further reduced, and the visual effect is consistent with the quantitative ablation conclusion.

[0220] Reference Figure 10 (Ablation experiment visualization and residual error figure), which shows the difference in residual error under different module configurations. The baseline model has obvious boundary-related residual error at the lesion edge and texture-dense area; only adding a hybrid bottleneck can reduce large-scale residual error but details are still missing; only using structure guidance is easy to appear over-constrained residual error in uniform area. After the complete scheme is combined, the residual error converges globally and locally at the same time, the boundary residual error "halo" and the blood vessel along the line residual error are further weakened, indicating that the synergy of structure perception and hybrid bottleneck module can improve the global modeling and local high-frequency restoration ability at the same time.

[0221] Example Two

[0222] Based on the same concept, the application further provides a cross-modal ultrasound image generation system fusing structure perception and mixed bottleneck, which can include an image acquisition module, a model processing module and a display module. The model processing module stores a generation network model trained according to the method.

[0223] Embodiment three

[0224] The embodiment also provides an electronic device, which refers to Figure 11 , including a memory 404 and a processor 402, the memory 404 stores a computer program, and the processor 402 is configured to execute the computer program to perform the steps in any of the above method embodiments.

[0225] Specifically, the processor 402 can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the application.

[0226] The memory 404 can include a mass storage that stores data or instructions. For example, and without limitation, the memory 404 can include a Hard Disk Drive (HDD), a floppy disk drive, a Solid State Drive (SSD), a flash drive, a Compact Disc Read Only Memory (CD-ROM), a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. The memory 404 can be removable and / or non-removable (or fixed) as appropriate. The memory 404 can be internal or external as appropriate. In particular embodiments, the memory 404 is a Non-Volatile memory. In particular embodiments, the memory 404 includes a Read-Only Memory (ROM) and a Random Access Memory (RAM). The ROM can be a mask-programmed ROM, a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), an Electrically Alterable ROM (EAROM), or a FLASH memory, or a combination of two or more of these, as appropriate. The RAM can be a Static Random-Access Memory (SRAM) or a Dynamic Random Access Memory (DRAM), which can be a Fast Page Mode Dynamic Random Access Memory (FPMDRAM), an Extended Data Output Dynamic Random Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), or the like, as appropriate.

[0227] The memory 404 can be used to store or cache various data files needed for processing and / or communication, and possible computer program instructions executed by the processor 402.

[0228] The processor 402 implements any one of the above-described diffusion model based cross-modality ultrasound contrast image generation methods in the embodiments by reading and executing the computer program instructions stored in the memory 404.

[0229] Optionally, the above-described electronic device can further include a transmission device 406 connected with the processor 402 and an input / output device 408 connected with the processor 402.

[0230] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network can include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (NIC) which can be connected with other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (RF) module which is used to communicate with the Internet in a wireless manner.

[0231] The input / output device 408 is used to input or output information.

[0232] Embodiment Four

[0233] The embodiment also provides a readable storage medium, and the readable storage medium stores a computer program. The computer program includes program codes for controlling a process to execute the process. The process includes the diffusion model based cross-modality ultrasound contrast image generation method according to the first embodiment.

[0234] It should be noted that specific examples in the embodiment can refer to examples described in the above-described embodiments and optional implementation manners, and the embodiment will not be described here again.

[0235] Generally, various embodiments can be implemented in hardware or special-purpose circuitry, software, logic or any combination thereof. Some aspects of the application can be implemented in hardware, while other aspects can be implemented in firmware or software to be executed by a controller, microprocessor or other computing device, but the application is not limited thereto. While various aspects of the application can be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein can be implemented in hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controler or other computing devices, or some combination thereof.

[0236] Embodiments of the application can be implemented by computer software executable by a data processor of the mobile device such as in the processor entity, or by hardware, or by a combination of software and hardware. Computer software or program, also called program product, including software routines, applets and / or macros, can be stored in any apparatus-readable data storage medium and they include program instructions to implement certain tasks. The program product can include one or more computer-executable components tangibly embodied in a computer- readable medium, when executed, for implementing one or more embodiments of the present application. The one or more computer-executable components can be one or more of: a process; a function; a routine; a sag; a subroutine; a plug-in; an app; or a program. The one or more computer-executable components can include a computer program that is written in any form of programming language, including code.

[0237] Those skilled in the art should clearly understand that each technical feature in the above embodiments can be combined with any other technical feature, and for the sake of brevity, not all possible combinations of the technical features in the above embodiments are described herein, however, as long as the combinations of the technical features do not exist contradictions, they should be considered within the scope of the present disclosure.

[0238] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be pointed out that for those skilled in the art, some modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for generating cross-modal ultrasound contrast images based on a diffusion model, characterized in that, Includes the following steps: Obtain the B-mode ultrasound image to be converted as the source domain image; A generative network based on a denoising diffusion probability model is constructed, wherein the generative network includes an encoder, a bottleneck module, and a decoder; The source domain image is converted into a target domain ultrasound contrast image using the generative network. The steps involved in constructing the generative network include: A multi-scale structure-aware mechanism is introduced into the feature extraction path of the encoder and decoder. The multi-scale structure-aware mechanism extracts the structural prior information of the source domain image and fuses the structural prior information into the intermediate feature map of the generator network to guide the generation of image anatomical structure. The bottleneck module is configured as a hybrid feature extraction network, which includes at least cascaded state space model units and KAN units. The state space model units are used to capture long-distance dependencies of feature sequences, and the KAN units are used to perform nonlinear feature transformation.

2. The method for generating cross-modal ultrasound contrast images as described in claim 1, characterized in that, The multi-scale structure perception mechanism includes a dual structure-guided attention module disposed in the encoder or decoder. The execution process of the dual-structure guided attention module includes: Spatial dimension pooling is performed on the input feature map to generate an internal spatial attention map that captures the feature response region; The source domain image is subjected to frequency domain transformation to extract high-frequency components containing edge and texture information, and an external structure guide mask is generated based on the high-frequency components. The internal spatial attention map is fused with the external structural guidance mask to obtain the final attention map, and the final attention map is applied to the input feature map to achieve weighted control of the features.

3. The method for generating cross-modal ultrasound contrast images as described in claim 2, characterized in that, The specific steps for performing frequency domain transformation on the source domain image are as follows: The source domain image is decomposed using two-dimensional discrete wavelet transform to extract high-frequency components in the horizontal, vertical, and diagonal directions. The high-frequency components are then concatenated and passed through a convolutional layer to generate the external structure guide mask.

4. The method for generating cross-modal ultrasound contrast images as described in claim 1, characterized in that, The multi-scale structure perception mechanism further includes an adaptive structure fusion module located at the connection between the decoder and the encoder; the adaptive structure fusion module replaces skip connections in the generator network and receives three input features: Deep features from the layer preceding the decoder, skip connection features from the corresponding layer of the encoder, and source domain structural features extracted from the source domain features that are independent and do not share weights with the encoder. The adaptive structural fusion module calculates fusion weights through a gating network, adaptively adjusts the pass rate of the source domain structural features using the fusion weights, and then splices and fuses them with the deep features and skip connection features.

5. The method for generating cross-modal ultrasound contrast images as described in claim 1, characterized in that, The bottleneck module adopts a cascaded structure, which includes a first state space model unit, a first KAN unit, a central processing unit, a second KAN unit, and a second state space model unit in sequence. The central processing unit includes a time-structure decoupling attention module, which is configured as follows: The source domain image is mapped to a structural position code using a source domain position encoder, and the structural position code is injected into the query vector and key vector of the attention mechanism. The temporal embedding vector of the diffusion model is mapped to gating coefficients, and the attention weight matrix is ​​weighted using the gating coefficients to decouple the influence of noise level and image structure on attention.

6. The method for generating cross-modal ultrasound contrast images as described in claim 1, characterized in that, The calculation process of the KAN unit is as follows: The input features are linearly projected, and after passing through an activation function, the features are nonlinearly mapped and weighted summed using learnable B-spline basis functions.

7. The method for generating cross-modal ultrasound contrast images as described in any one of claims 1 to 6, characterized in that, The training process of the generative network employs a composite loss function, which includes: Basic loss: Calculate the L1 distance between the predicted noise component and the residual component and the true value; Perceptual loss: High-level semantic features of the generated image and the real target image are extracted using a pre-trained convolutional neural network and their differences are calculated; Frequency domain loss: High-frequency components of the generated image and the real target image are extracted using fast Fourier transform and their differences are calculated; The perceptual loss and frequency domain loss are adjusted by dynamic time weights, which increase as the diffusion process progresses through time steps, in order to strengthen the constraints on detail generation in the later stages of denoising.

8. A cross-modal ultrasound contrast imaging image generation system based on a diffusion model, characterized in that, include: The image acquisition module is used to acquire the B-mode ultrasound image to be converted; The model processing module stores a generative network model trained by the cross-modal ultrasound contrast imaging image generation method as described in any one of claims 1 to 7, and is used to receive the B-mode ultrasound image and output the ultrasound contrast imaging image. The display module is used to display the generated ultrasound contrast images.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the cross-modal ultrasound contrast image generation method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program, the computer program including program code for controlling a process to execute the process, the process including the cross-modal ultrasound contrast image generation method according to any one of claims 1 to 7.

Images