Medical image data enhancement method based on progressive mask fusion and related apparatus

Abstract

This invention provides a method and apparatus for medical image data enhancement based on progressive mask fusion. The method includes: acquiring a target medical image; generating a real lesion mask based on the target medical image; performing mask enhancement on the real lesion mask to generate a virtual mask; inputting the virtual mask into a trained Med-ControlNet generation model to obtain a synthetic medical image. The generation model includes a backbone network and a control branch. The control branch receives the virtual mask to extract multi-scale control features. A progressive mask fusion module generates a dynamic soft mask based on the received time step information, adjusts the fusion ratio of multi-scale control features and backbone network features based on the dynamic soft mask, and outputs the fused features, injecting them into the corresponding layers of the backbone network. Finally, the backbone network outputs the synthetic medical image. Based on this, this invention can generate synthetic medical images with natural edges and coherent textures.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a method and apparatus for enhancing medical image data based on progressive mask fusion. Background Technology

[0002] With the widespread application of deep learning technology in smart healthcare, computer-aided diagnostic (CAD) systems have made breakthroughs in tasks such as polyp segmentation and tumor detection. However, the performance of deep learning models is highly dependent on the scale and quality of the training data. In the field of medical imaging, obtaining large-scale, high-quality, and finely labeled datasets faces multiple challenges, including "data silos," strict privacy protection, and high costs associated with expert annotation. Even more serious is the prevalent "long-tail distribution" phenomenon in medical data: data on common lesions is relatively abundant, while samples of rare lesions, microlesions, or lesions in specific growth stages are extremely scarce. This uneven data distribution directly leads to insufficient generalization ability of models when faced with complex clinical cases, easily resulting in missed detections or misdiagnoses. Currently, the main technical solutions in the field of medical image generation include: methods based on generative adversarial networks (GANs), methods based on standard diffusion models (DDPM / LDM), and diffusion models based on structured control (ControlNet, T2I-Adapter).

[0003] To alleviate the aforementioned problems, data augmentation has become a standard procedure in model training. Traditional data augmentation mainly includes rotation, flipping, scaling, cropping, and color transformation. While these methods are computationally inexpensive, they only perform simple geometric or photometric transformations at the pixel level, failing to generate new semantic information or simulate the complex nonlinear changes in lesion morphology, and the model is prone to overfitting. In recent years, methods represented by Generative Adversarial Networks (GANs) have been used to synthesize medical images, but GANs suffer from inherent defects such as training instability and pattern collapse, making it difficult to generate diverse high-resolution samples.

[0004] In contrast, diffusion models, with their superior distribution coverage and training stability, are gradually becoming the new mainstream in medical image synthesis. To achieve controlled generation (i.e., generating lesions of a specified shape at a specified location), researchers have proposed conditional diffusion models. The most similar existing solution is ControlNet, which is currently one of the most advanced controlled generation architectures.

[0005] Although ControlNet performs well in the field of natural image generation, it has significant technical shortcomings when directly applied to medical image enhancement, which has extremely high requirements for accuracy and realism. These shortcomings are mainly reflected in the problems of "temporal misalignment of feature fusion" and "edge artifacts". In other words, the images generated by the existing technology have serious edge fragmentation and texture inconsistency. Summary of the Invention

[0006] This invention provides a method and apparatus for enhancing medical image data based on progressive mask fusion, which can generate synthetic medical images with natural edges and coherent textures.

[0007] In a first aspect, embodiments of the present invention provide a method for enhancing medical image data based on progressive mask fusion, comprising:

[0008] Acquire the target medical image to be augmented; Generate a mask of the actual lesion based on the target medical image; The actual lesion mask is enhanced to generate a virtual mask; The virtual mask is input into a trained Med-ControlNet generative model to obtain a synthetic medical image. The Med-ControlNet generative model is constructed based on a latent diffusion model and includes a backbone network and a control branch. A progressive mask fusion module is embedded between each corresponding layer of the backbone network and the control branch. The control branch receives the virtual mask to extract multi-scale control features containing lesion spatial structure information. The control branch inputs the multi-scale control features and the time step information of the current diffusion process into the progressive mask fusion module. The progressive mask fusion module generates a dynamic soft mask based on the received time step information, dynamically adjusts the fusion ratio of the multi-scale control features and the backbone network features based on the dynamic soft mask, and outputs the fused features into the corresponding layer of the backbone network. Finally, the backbone network outputs the synthetic medical image.

[0009] In some embodiments, it also includes: The synthetic medical image is subjected to quality screening according to a preset threshold to obtain a high-quality synthetic image. The quality screening is achieved by calculating the intersection-union ratio of the synthetic medical image and the corresponding mask. The high-quality synthetic image and the real medical image corresponding to the real mask are mixed in a specific ratio to construct a mixed training set; The downstream medical image segmentation model is trained based on the hybrid training set.

[0010] In some embodiments, the training method for the Med-ControlNet generative model includes: Obtain medical image training samples and corresponding labeled masks, wherein the labeled masks are extracted from a virtual mask pool; The Med-ControlNet generative model is trained based on the medical image training samples and the labeled mask. During the training process, only the parameters of the control branch and the progressive mask fusion module are updated, while the parameters of the backbone network remain frozen. The parameters of the Med-ControlNet generative model are optimized based on the standard noise prediction mean square error loss function to obtain the trained Med-ControlNet generative model.

[0011] In some embodiments, the method for constructing the virtual mask pool includes: Apply elastic deformation or mesh distortion to a single lesion mask to generate deformable masks with different shapes; Multiple lesion masks are cropped, scaled, and rotated and then stitched together into the same mask to generate a multi-lesion composite mask. The virtual mask pool is constructed based on the deformation mask and the multi-lesion synthesis mask.

[0012] In some embodiments, the backbone network adopts a U-Net structure, and the backbone network includes an encoder, intermediate blocks and a decoder. The backbone network is also used to receive and process noise latent variables.

[0013] In some embodiments, the control branch reuses the encoder architecture of the backbone network in the network structure. The control branch extracts the multi-scale control features containing lesion spatial structure information at each resolution level through hierarchical downsampling, and sends the multi-scale control features to the progressive mask fusion module of the corresponding level.

[0014] In some embodiments, the progressive mask fusion module generates a dynamic soft mask based on the received time step information, modulates the fusion weights of the multi-scale control features and the backbone network features based on the dynamic soft mask, and outputs fused features, including: Encode the time step information into a sinusoidal position vector with continuous temporal semantics; The sinusoidal position vector is mapped to a bias vector of the feature channel dimension through a projection layer; The bias vector is expanded in the spatial dimension to align with the feature size of the backbone network through a broadcast operation, resulting in an expanded temporal bias vector. The extended temporal bias vector is added to the backbone network features, and then passed through a zero convolutional layer and a Sigmoid activation function in sequence to generate a dynamic soft mask alpha. The multi-scale control features are weighted using the dynamic soft mask alpha to obtain the first weighted feature; The backbone network features are weighted using complementary weights 1-alpha to obtain the second-path weighted features; The first weighted feature and the second weighted feature are added together to output the fused feature.

[0015] Secondly, embodiments of the present invention also provide a medical image data enhancement device based on progressive mask fusion, the device comprising: The acquisition module is used to acquire the target medical image to be augmented. The generation module is used to generate a mask of the real lesion based on the target medical image; The enhancement module is used to enhance the mask of the real lesion and generate a virtual mask; A synthesis module is used to input the virtual mask into a trained Med-ControlNet generative model to obtain a synthesized medical image. The Med-ControlNet generative model is constructed based on a latent diffusion model and includes a backbone network and a control branch. A progressive mask fusion module is embedded between each corresponding layer of the backbone network and the control branch. The control branch receives the virtual mask to extract multi-scale control features containing lesion spatial structure information. The control branch inputs the multi-scale control features and the time step information of the current diffusion process into the progressive mask fusion module. The progressive mask fusion module generates a dynamic soft mask based on the received time step information, dynamically adjusts the fusion ratio of the multi-scale control features and the backbone network features based on the dynamic soft mask, and outputs the fused features into the corresponding layer of the backbone network. Finally, the backbone network outputs the synthesized medical image.

[0016] Thirdly, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the medical image data enhancement method based on progressive mask fusion as described in the first aspect.

[0017] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for performing the medical image data enhancement method based on progressive mask fusion as described in the first aspect.

[0018] The present invention provides a method and related apparatus for medical image data enhancement based on progressive mask fusion. The method includes: acquiring a target medical image to be enhanced; generating a real lesion mask based on the target medical image; performing mask enhancement on the real lesion mask to generate a virtual mask; and inputting the virtual mask into a trained Med-ControlNet generation model to obtain a synthesized medical image. The Med-ControlNet generation model is constructed based on a latent diffusion model. The invention comprises a backbone network and a control branch. A progressive mask fusion module is embedded between each corresponding level of the backbone network and the control branch. The control branch receives a virtual mask to extract multi-scale control features containing lesion spatial structure information. The control branch inputs the multi-scale control features and the time step information of the current diffusion process into the progressive mask fusion module. The progressive mask fusion module generates a dynamic soft mask based on the received time step information, dynamically adjusts the fusion ratio of the multi-scale control features and the backbone network features based on the dynamic soft mask, and outputs the fused features, injecting them into the corresponding level of the backbone network. Finally, the backbone network outputs a synthesized medical image. This invention introduces a time-aware dynamic control mechanism through the progressive mask fusion module. This mechanism can sense the current time step of the diffusion process and adaptively generate a dynamic soft mask accordingly to dynamically adjust the fusion ratio of foreground control features and background generated features, promoting the natural penetration and fusion of lesion texture and surrounding mucosal tissue. Based on this, embodiments of the invention can generate synthetic medical images with natural edges and coherent textures. Attached Figure Description

[0019] Figure 1A This is the main flowchart of a medical image data enhancement method based on progressive mask fusion provided in one embodiment of the present invention; Figure 1B This is an overall architecture diagram of the Med-ControlNet generative model provided in one embodiment of the present invention; Figure 2 This is an architecture diagram of a progressive mask fusion module provided in one embodiment of the present invention; Figure 3 This is a flowchart of a virtual mask construction process based on geometric transformation provided in one embodiment of the present invention; Figure 4 This is a flowchart of a hybrid training closed-loop process based on quality screening provided in one embodiment of the present invention; Figure 5 This is a comparison chart of the effects of the baseline method and the method proposed in this invention; Figure 6This is a schematic diagram of the structure of a medical image data enhancement device based on progressive mask fusion provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the following drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0022] In this embodiment of the invention, the terms "furthermore," "exemplarily," or "optionally" are used as examples, illustrations, or descriptions and should not be construed as being more preferred or advantageous than other embodiments or designs. The use of the terms "furthermore," "exemplarily," or "optionally" is intended to present the relevant concepts in a specific manner.

[0023] First, let's analyze some of the terms used in this invention: Diffusion Models: A class of generative models based on thermodynamic non-equilibrium processes, which generate clear images by progressively removing noise from the image. This invention is primarily based on Latent Diffusion Models (LDM).

[0024] ControlNet: A neural network architecture for controlling pre-trained diffusion models. This invention improves upon this architecture. The original ControlNet extracts conditional features (such as edge maps and pose maps) by copying the weights of the backbone encoder. This invention retains its parallel branching structure but improves its feature injection method (i.e., by replacing direct addition with a PMF module).

[0025] The PMF (Progressive Mask Fusion) module refers to a feature fusion device / algorithm module designed in this invention. This module differs from the direct additive connections of traditional ControlNet; its core mechanism lies in the introduction of time-aware embedding. PMF acquires the time-step information of the current diffusion process, maps it to a bias vector, modulates the input features, and generates a spatially adaptive soft mask through zero-convolutional layers and activation functions. This soft mask is used to dynamically adjust the fusion ratio of control branch features and backbone network features, thereby achieving smooth and complementary feature injection at different stages of diffusion generation (early stage emphasizing structure, late stage emphasizing texture).

[0026] Zero Convolution: Refers to a 1×1 convolutional layer whose weights and biases are initialized to zero. In the initial training phase, its output is zero, ensuring that the control branch does not interfere with the pre-trained weights of the backbone network. As training progresses, it gradually learns effective control signals.

[0027] To facilitate a more convenient description of the working principle of the embodiments of the present invention, the following introduction of relevant technical scenarios is given first.

[0028] With the widespread application of deep learning technology in smart healthcare, computer-aided diagnostic (CAD) systems have made breakthroughs in tasks such as polyp segmentation and tumor detection. However, the performance of deep learning models is highly dependent on the scale and quality of the training data. In the field of medical imaging, obtaining large-scale, high-quality, and finely labeled datasets faces multiple challenges, including "data silos," strict privacy protection, and high costs associated with expert annotation. Even more serious is the prevalent "long-tail distribution" phenomenon in medical data: data on common lesions is relatively abundant, while samples of rare lesions, microlesions, or lesions in specific growth stages are extremely scarce. This uneven data distribution directly leads to insufficient generalization ability of models when faced with complex clinical cases, easily resulting in missed detections or misdiagnoses. Currently, the main technical solutions in the field of medical image generation include: methods based on generative adversarial networks (GANs), methods based on standard diffusion models (DDPM / LDM), and diffusion models based on structured control (ControlNet, T2I-Adapter).

[0029] To alleviate the aforementioned problems, data augmentation has become a standard procedure in model training. Traditional data augmentation mainly includes rotation, flipping, scaling, cropping, and color transformation. While these methods are computationally inexpensive, they only perform simple geometric or photometric transformations at the pixel level, failing to generate new semantic information or simulate the complex nonlinear changes in lesion morphology, and the model is prone to overfitting. In recent years, methods represented by Generative Adversarial Networks (GANs) have been used to synthesize medical images, but GANs suffer from inherent defects such as training instability and pattern collapse, making it difficult to generate diverse high-resolution samples.

[0030] In contrast, diffusion models, with their superior distribution coverage and training stability, are gradually becoming the new mainstream in medical image synthesis. To achieve controlled generation (i.e., generating lesions of a specified shape at a specified location), researchers have proposed conditional diffusion models. The most similar existing solution is ControlNet, which is currently one of the most advanced controlled generation architectures.

[0031] Although ControlNet performs well in the field of natural image generation, it has significant technical shortcomings when directly applied to medical image enhancement, which has extremely high requirements for accuracy and realism. These shortcomings are mainly reflected in the problems of "temporal misalignment of feature fusion" and "edge artifacts". In other words, the images generated by the existing technology have serious edge fragmentation and texture inconsistency.

[0032] Based on this, the present invention provides a method and related apparatus for enhancing medical image data based on progressive mask fusion. The medical image data augmentation method based on progressive mask fusion includes: acquiring the target medical image to be augmented; generating a real lesion mask based on the target medical image; performing mask augmentation on the real lesion mask to generate a virtual mask; and inputting the virtual mask into a trained Med-ControlNet generative model to obtain a synthetic medical image. The Med-ControlNet generative model is constructed based on a latent diffusion model and includes a backbone network and a control branch. A progressive mask fusion module is embedded between each corresponding layer of the backbone network and the control branch. The control branch receives the virtual mask to extract multi-scale control features containing lesion spatial structure information. The control branch inputs the multi-scale control features and the time step information of the current diffusion process into the progressive mask fusion module. The progressive mask fusion module generates a dynamic soft mask based on the received time step information, dynamically adjusts the fusion ratio of the multi-scale control features and the backbone network features based on the dynamic soft mask, and outputs the fused features into the corresponding layer of the backbone network. Finally, the backbone network outputs the synthetic medical image. This invention introduces a time-aware dynamic control mechanism through a progressive mask fusion module. This mechanism can sense the current time step of the diffusion process and adaptively generate a dynamic soft mask accordingly. This dynamically adjusts the fusion ratio of foreground control features and background generation features, promoting the natural penetration and fusion of lesion texture with surrounding mucosal tissue. Based on this, embodiments of this invention can generate synthetic medical images with natural edges and coherent textures.

[0033] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0034] like Figure 1A As shown, Figure 1A This is a flowchart of a medical image data enhancement method based on progressive mask fusion provided in an embodiment of the present invention. The medical image data enhancement method based on progressive mask fusion may include, but is not limited to, steps S101 to S104.

[0035] Step S101: Obtain the target medical image to be augmented; Step S102: Generate a mask of the actual lesion based on the target medical image; Step S103: Perform mask enhancement on the real lesion mask to generate a virtual mask; Step S104: Input the virtual mask into the trained Med-ControlNet generative model to obtain a synthetic medical image. The Med-ControlNet generative model is constructed based on a latent diffusion model. The Med-ControlNet generative model includes a backbone network and a control branch. A progressive mask fusion module is embedded between each corresponding layer of the backbone network and the control branch. The control branch is used to receive the virtual mask to extract multi-scale control features containing lesion spatial structure information. The control branch inputs the multi-scale control features and the time step information of the current diffusion process into the progressive mask fusion module. The progressive mask fusion module generates a dynamic soft mask based on the received time step information, dynamically adjusts the fusion ratio of the multi-scale control features and the backbone network features based on the dynamic soft mask, and outputs the fused features to be injected into the corresponding layer of the backbone network. Finally, the backbone network outputs the synthetic medical image.

[0036] Understandably, this invention introduces a time-aware dynamic control mechanism by employing a progressive mask fusion PMF module. This mechanism can sense the current time step of the diffusion process and adaptively generate a soft mask accordingly, dynamically adjusting the fusion ratio of foreground control features and background generated features. In the early stage of generation (structure building stage), the PMF module enhances the control signal to ensure the accurate shape of the generated lesion; in the late stage of generation (texture filling stage), the PMF module automatically weakens the rigid structural constraints, promoting the natural penetration and fusion of the lesion texture with the surrounding mucosal tissue. Based on this, by generating synthetic medical images with natural edges and realistic textures, the invention effectively solves the problems of rigid feature injection methods, harsh edges, and unnatural fusion in existing technologies.

[0037] It is understood that the medical image data enhancement method based on progressive mask fusion of the present invention introduces dynamic information of the time dimension into the spatial feature fusion process, and achieves fine control of the generation process by constructing an adaptive dynamic soft mask that can evolve with time step.

[0038] The implementation of this invention can be divided into two core stages: the first stage is the construction and training of the Med-ControlNet generative model, and the second stage is the generation and enhancement of high-quality samples based on virtual masks.

[0039] Phase 1: Construction and Training of the Med-ControlNet Generative Model The goal of this stage is to train a medical image generation model that can accurately follow anatomical shape control while possessing extremely high texture fusion capabilities.

[0040] Step S1: Construct a generative architecture based on the latent space This invention is based on the Latent Diffusion Model (LDM) and constructs, as follows: Figure 1BThe overall architecture shown is mainly composed of a backbone network and a control branch. The backbone network, which can adopt the U-Net structure of Stable Diffusion V1.5 (including an encoder, intermediate blocks, and a decoder), is used to receive and process noisy latent variables. The control branch structurally corresponds to the encoder part of the backbone network and is used to receive masked input to extract multi-scale control features. As shown in the figure, the multi-scale control features output by the control branch, along with time step t, are input to the PMF module. After processing by the PMF module, they are injected into the corresponding layers of the backbone network, and finally, the backbone network outputs the generated image. To reduce computational resource consumption, the model uses a pre-trained variational autoencoder (VAE) to compress the input image from the pixel space to the latent space to obtain a latent representation. All diffusion denoising processes are performed in the latent space. Simultaneously, to preserve the general image generation prior learned by the model on large-scale datasets, all parameters of the backbone network are frozen during training, and only the control branch and related modules are optimized.

[0041] Step S2: Construct parallel control branches To incorporate explicit control information on lesion shape during the generation process, a structure such as... Figure 1B The control branch is shown. This branch completely reuses the encoder architecture of the backbone network in terms of network structure to ensure dimensional alignment of the feature space. In specific implementation, the binarized lesion segmentation mask is used as the control condition input to this branch. The control branch extracts multi-scale control features containing lesion spatial structure information at each resolution level through hierarchical downsampling. Figure 1B As shown in the connection relationship, these multi-scale control features are not incorporated into the main body through the traditional simple addition method, but are sent to the PMF modules of the corresponding levels as the input basis for subsequent feature modulation and fusion.

[0042] Step S3: Construct and embed the progressive mask fusion module This invention proposes a progressive mask fusion PMF module, aiming to address the questions of "when to fuse" and "how to fuse" multi-scale control features and backbone network features. For example... Figure 2 As shown, the specific processing flow of the PMF module is as follows: First, the time step t of the current diffusion model is obtained. To inject time scalar information into spatial features, it is first encoded as a sinusoidal position vector with continuous temporal semantics, and then... Figure 2The projection layer maps the temporal bias vector to a bias vector along the feature channel dimension. Then, a broadcast operation expands this bias vector spatially, aligning its size perfectly with the backbone network features. Next, the expanded temporal bias vector is added to and fused with the backbone network features, combining information from both the "current generation stage" and the "current image content." The fused features are then passed through a zero-convolutional layer and a sigmoid activation function, outputting a dynamic soft mask alpha with values ​​ranging from (0, 1).

[0043] like Figure 2 As shown in the right-hand branch, alpha is used as a weight to control the multi-scale feature f. ctrl Weighting is applied; simultaneously, complementary weights (1-alpha) are generated through the 1-x module, and these complementary weights are used to weight the backbone network features h. enc Weighting is performed. Finally, the weighted features from both paths are added together to obtain the fused feature h. out The calculation formula is expressed as: h out = alpha f ctrl +(1 - alpha) h enc .

[0044] Through the above mechanism, alpha can change dynamically with time step t: in the early stage of diffusion (when alpha is large), the control branch dominates to establish the lesion outline; in the later stage of diffusion (when alpha decreases), the backbone network branch dominates to optimize texture details, thereby eliminating the harsh cutting feeling of the edges and achieving a smooth transition of features.

[0045] Step S4: Model Training and Optimization During training, pairs of medical images and labeled masks are input. The model aims to predict the noise epsilon added to the latent variables. A standard mean squared error loss for noise prediction (MSE Loss) is used: L = || epsilon - epsilon theta ( z_t, t, c txt , c mask Only the parameters of the control branch and PMF module are updated, while the parameters of the backbone network remain frozen. This strategy ensures training efficiency while avoiding overfitting caused by small sample medical data.

[0046] Phase Two: Data Augmentation Closed Loop Based on Virtual Mask The trained Med-ControlNet generative model has the ability to transform masks of arbitrary shapes into realistic lesion images. This stage generates high-difficulty samples in batches by constructing diverse "virtual masks".

[0047] Step S5: Constructing a High-Difficulty Virtual Mask To address the long-tail distribution problem of medical image data and generate lesion morphologies that are rare or difficult to acquire in clinical practice, this invention constructs a data augmentation workflow as shown in the figure. It uses real single-lesion masks (such as mask A and mask B) as input sources, augments the data through two parallel paths, and finally merges them into a virtual mask pool. Figure 3 As shown in the upper part, a transformation library containing various deformation algorithms is established. Specifically, elastic deformation is used to simulate the natural compression and twisting of biological soft tissue, and mesh distortion is used to locally stretch or compress the mask mesh. Through these deformation operations, the geometric morphological features of the lesion are greatly enriched, outputting deformation masks with diverse shapes. Figure 3 As shown in the lower part of the diagram, for the task involving multiple lesions, a combination operation is performed. Specifically, through a random cropping and splicing algorithm, lesion masks from different patients are randomly selected. After scaling and rotation transformations, these masks are randomly "planted" at different positions on the same blank mask, constructing a complex multi-lesion composite mask with overlapping and adhering features. Finally, the thousands of deformed masks and multi-lesion composite masks generated above are uniformly aggregated into a virtual mask pool, serving as strong conditional inputs for subsequent model training. This enables the model to cope with the highly challenging lesion morphologies in real-world clinical scenarios.

[0048] Step S6: Generation and Blending Enhancement The trained Med-ControlNet generative model is used to build a data augmentation and application closed loop, such as... Figure 4 As shown, the specific process is as follows: (1) Texture generation and fusion. The virtual mask obtained in step S4 is input into the Med-ControlNet generative model. Under the dynamic control of the PMF module, the model can accurately generate the corresponding lesion texture according to the geometry of the mask and naturally fuse it into the background mucosa to output a synthetic medical image.

[0049] (2) Automated quality screening. To eliminate samples of poor quality, a process such as automated quality screening is introduced. Figure 4 The quality screening mechanism is shown in the middle diamond box. Specifically, it calculates the Intersection over Union (IoU) between the synthesized image and the corresponding mask, sets a threshold of 0.85, and only retains high-quality synthesized images with IoU > 0.85 to proceed to the next stage.

[0050] (3) Hybrid Training and Application. The selected high-quality synthetic images are mixed with the real images corresponding to the real masks at a specific ratio (e.g., 1:1) to construct a hybrid training set. Finally, this hybrid training set is used to train downstream segmentation models (such as U-Net, ResUNet++, etc.) to improve the generalization ability of the segmentation model in complex lesion scenarios.

[0051] like Figure 5 As shown, it presents a visual comparison of the generation effect between the method proposed in this invention and the benchmark model. The key point is to highlight the advantages of this invention in handling complex edge details and texture fusion by magnifying the details locally (dashed circles). Compared with the benchmark model, the synthetic medical image generated by the method proposed in this invention makes the texture transition between the edge of the lesion and the background mucosa smooth and fluid, completely eliminating the "texture feel".

[0052] Based on this, the present invention introduces a time-aware dynamic control mechanism through a progressive mask fusion module. This dynamic control mechanism can sense the current time step of the diffusion process and adaptively generate a dynamic soft mask accordingly to dynamically adjust the fusion ratio of foreground control features and background generation features, promoting the natural penetration and fusion of lesion texture and surrounding mucosal tissue. Based on this, embodiments of the present invention can generate synthetic medical images with natural edges and coherent textures.

[0053] Compared to existing technologies such as ControlNet, this invention has at least the following significant advantages: The edges blend naturally, reducing artificial artifacts. Existing technologies use "static hard injection," which often results in harsh edge cut lines in the generated images. This invention introduces time awareness through the PMF module, simulating the natural process of biological tissue growth—first shaping, then granulation. In the later stages of generation, the blending weights are adaptively adjusted, resulting in a smooth and fluid texture transition between the lesion edges and the background mucosa, completely eliminating the "texture-like" appearance.

[0054] The generated samples exhibit greater diversity and complexity. Relying on a robust fusion mechanism, this model demonstrates exceptional adaptability to the morphology of the input mask. Whether it's an extremely distorted elastic deformation mask or a mask for tiny early-stage cancer lesions, the model can generate reasonable texture filling. This enables the invention to utilize aggressive mask enhancement strategies to mass-produce "long-tailed samples" and "difficult samples" that are difficult to capture clinically.

[0055] This invention significantly improves the generalization ability of downstream models. Experiments demonstrate that training segmentation models using the augmented data generated by this invention significantly enhances the model's ability to identify blurred boundaries and irregular lesions. In tasks such as polyp segmentation, both mDice and mIoU metrics show substantial improvements compared to traditional augmentation methods.

[0056] In addition, such as Figure 6 As shown, one embodiment of the present invention also discloses a medical image data enhancement device based on progressive mask fusion, the device comprising: The acquisition module 110 is used to acquire the target medical image to be augmented. The generation module 120 is used to generate a mask of the real lesion based on the target medical image; Enhancement module 130 is used to enhance the mask of the real lesion and generate a virtual mask; The synthesis module 140 is used to input a virtual mask into a trained Med-ControlNet generative model to obtain a synthetic medical image. The Med-ControlNet generative model is constructed based on a latent diffusion model. The Med-ControlNet generative model includes a backbone network and a control branch. A progressive mask fusion module is embedded between each corresponding layer of the backbone network and the control branch. The control branch is used to receive the virtual mask to extract multi-scale control features containing lesion spatial structure information. The control branch inputs the multi-scale control features and the time step information of the current diffusion process into the progressive mask fusion module. The progressive mask fusion module generates a dynamic soft mask based on the received time step information, dynamically adjusts the fusion ratio of the multi-scale control features and the backbone network features based on the dynamic soft mask, and outputs the fused features to be injected into the corresponding layer of the backbone network. Finally, the backbone network outputs the synthetic medical image.

[0057] The medical image data enhancement device based on progressive mask fusion in this embodiment of the invention is used to execute the medical image data enhancement method based on progressive mask fusion in the above embodiments. Its specific processing procedure is the same as that of the medical image data enhancement method based on progressive mask fusion in the above embodiments, and will not be described in detail here.

[0058] In addition, such as Figure 7 As shown, one embodiment of the present invention also discloses an electronic device, including: at least one processor 210; at least one memory 220 for storing at least one program; when the at least one program is executed by the at least one processor 210, it implements the medical image data enhancement method based on progressive mask fusion as in any of the preceding embodiments.

[0059] In addition, one embodiment of the present invention discloses a computer-readable storage medium storing computer-executable instructions for performing a medical image data enhancement method based on progressive mask fusion as described in any of the preceding embodiments.

[0060] The system architecture and application scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0061] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0062] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0063] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process or execution thread, and components may be located on a single computer or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, or a network, such as the Internet interacting with other systems via signals).

Claims

1. A method for enhancing medical image data based on progressive mask fusion, comprising: Acquire the target medical image to be augmented; Generate a mask of the actual lesion based on the target medical image; The actual lesion mask is enhanced to generate a virtual mask; The virtual mask is input into a trained Med-ControlNet generative model to obtain a synthetic medical image. The Med-ControlNet generative model is constructed based on a latent diffusion model and includes a backbone network and a control branch. A progressive mask fusion module is embedded between each corresponding layer of the backbone network and the control branch. The control branch receives the virtual mask to extract multi-scale control features containing lesion spatial structure information. The control branch inputs the multi-scale control features and the time step information of the current diffusion process into the progressive mask fusion module. The progressive mask fusion module generates a dynamic soft mask based on the received time step information, dynamically adjusts the fusion ratio of the multi-scale control features and the backbone network features based on the dynamic soft mask, and outputs the fused features into the corresponding layer of the backbone network. Finally, the backbone network outputs the synthetic medical image.

2. The method according to claim 1, characterized in that, Also includes: The synthetic medical image is subjected to quality screening according to a preset threshold to obtain a high-quality synthetic image. The quality screening is achieved by calculating the intersection-union ratio of the synthetic medical image and the corresponding mask. The high-quality synthetic image and the real medical image corresponding to the real mask are mixed in a specific ratio to construct a mixed training set; The downstream medical image segmentation model is trained based on the hybrid training set.

3. The method according to claim 1, characterized in that, The training method for the Med-ControlNet generative model includes: Obtain medical image training samples and corresponding labeled masks, wherein the labeled masks are extracted from a virtual mask pool; The Med-ControlNet generative model is trained based on the medical image training samples and the labeled mask. During the training process, only the parameters of the control branch and the progressive mask fusion module are updated, while the parameters of the backbone network remain frozen. The parameters of the Med-ControlNet generative model are optimized based on the standard noise prediction mean square error loss function to obtain the trained Med-ControlNet generative model.

4. The method according to claim 1, characterized in that, The method for constructing the virtual mask pool includes: Apply elastic deformation or mesh distortion to a single lesion mask to generate deformable masks with different shapes; Multiple lesion masks are cropped, scaled, and rotated and then stitched together into the same mask to generate a multi-lesion composite mask. The virtual mask pool is constructed based on the deformation mask and the multi-lesion synthesis mask.

5. The method according to claim 1, characterized in that, The backbone network adopts a U-Net structure and includes an encoder, intermediate blocks, and a decoder. The backbone network is also used to receive and process noise latent variables.

6. The method according to claim 5, characterized in that, The control branch reuses the encoder architecture of the backbone network in the network structure. The control branch extracts the multi-scale control features containing the spatial structure information of the lesions at each resolution level through hierarchical downsampling, and sends the multi-scale control features into the progressive mask fusion module of the corresponding level.

7. The method according to claim 1, characterized in that, The progressive mask fusion module generates a dynamic soft mask based on the received time step information, modulates the fusion weights of the multi-scale control features and backbone network features based on the dynamic soft mask, and outputs fused features, including: Encode the time step information into a sinusoidal position vector with continuous temporal semantics; The sinusoidal position vector is mapped to a bias vector of the feature channel dimension through a projection layer; The bias vector is expanded in the spatial dimension to align with the feature size of the backbone network through a broadcast operation, resulting in an expanded temporal bias vector. The extended temporal bias vector is added to the backbone network features and then passed through a zero convolutional layer and a Sigmoid activation function in sequence to generate a dynamic soft mask alpha. The multi-scale control features are weighted using the dynamic soft mask alpha to obtain the first weighted feature; The backbone network features are weighted using complementary weights 1-alpha to obtain the second-path weighted features; The first weighted feature and the second weighted feature are added together to output the fused feature.

8. A medical image data enhancement device based on progressive mask fusion, characterized in that, The device includes: The acquisition module is used to acquire the target medical image to be augmented. The generation module is used to generate a mask of the real lesion based on the target medical image; The enhancement module is used to enhance the mask of the real lesion and generate a virtual mask; A synthesis module is used to input the virtual mask into a trained Med-ControlNet generative model to obtain a synthesized medical image. The Med-ControlNet generative model is constructed based on a latent diffusion model and includes a backbone network and a control branch. A progressive mask fusion module is embedded between each corresponding layer of the backbone network and the control branch. The control branch receives the virtual mask to extract multi-scale control features containing lesion spatial structure information. The control branch inputs the multi-scale control features and the time step information of the current diffusion process into the progressive mask fusion module. The progressive mask fusion module generates a dynamic soft mask based on the received time step information, dynamically adjusts the fusion ratio of the multi-scale control features and the backbone network features based on the dynamic soft mask, and outputs the fused features into the corresponding layer of the backbone network. Finally, the backbone network outputs the synthesized medical image.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the medical image data enhancement method based on progressive mask fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions for performing the medical image data enhancement method based on progressive mask fusion as described in any one of claims 1 to 7.

Images