A multi-slice prenatal ultrasound image generation model method controlled by a layout frame

Abstract

The present invention discloses a multi-section prenatal ultrasound image generation model method controlled by a layout frame, which belongs to the field of generative algorithms of deep learning technology. The method constructs a diffusion generative model structure based on the U-Net network structure, and uses the acquired prenatal ultrasound image set, the layout frame of the extracted prenatal ultrasound image anatomical structure, and the control conditions composed of the region of interest mask to form a training set. The constructed model is trained to obtain a prenatal ultrasound image generation model; for the prenatal ultrasound image to be synthesized, the prenatal ultrasound image generation model is applied, combined with the new control conditions composed of the modified layout frame and mask region, to obtain the expected synthesized prenatal ultrasound image. The region of interest mask method proposed in the present invention can make the model applicable to small-scale data sets, and the layout frame control method can accurately allow the model to synthesize ultrasound images with controllable appearance. It is simple to operate and user-friendly, and can quickly generate corresponding ultrasound images according to user needs.

Description

Technical Field

[0001] The present invention belongs to the field of generative algorithms of deep learning technology, and in particular relates to a multi-plane prenatal ultrasound image synthesis method controlled by a layout frame. Background Art

[0002] Prenatal ultrasound examinations play a crucial role in assessing fetal development and screening for anomalies. Particularly during the first and second trimesters, these examinations often involve screening across dozens of ultrasound planes, all crucial for a comprehensive assessment of fetal well-being. Given their complexity and the high demands for accuracy, these examinations are typically performed by experienced radiologists to ensure that potential abnormalities can be identified and a precise diagnosis can be made.

[0003] In clinical practice, prenatal screening requires a comprehensive screening of the fetus. However, the fetus moves frequently during the examination, making the acquisition of prenatal ultrasound images difficult and the amount of data relatively small. This has an adverse effect on the training of new radiologists. In addition, during the training of ultrasound physicians, it is very difficult to obtain images of multiple fetal sections in a single posture. For certain specific ultrasound sections, different fetuses may present completely different postures, which makes the qualitative learning of doctors complicated and difficult. In clinical practice, cases of deformity or maldevelopment are very rare. For example, the probability of cleft lip and palate is only one in a thousand. This means that the training of radiologists often requires a large number of abnormal cases to learn from.

[0004] In recent years, advances in artificial intelligence (AI) technology have revolutionized the field of prenatal ultrasound. Existing AI-assisted systems have demonstrated significant potential in improving the efficiency and accuracy of prenatal ultrasound screening. Leveraging advanced algorithms such as deep learning, AI tools not only help clinicians quickly locate key anatomical structures but also assist in identifying abnormalities, thereby optimizing the screening process. Training AI models requires extensive data, but prenatal ultrasound data, particularly for rare conditions such as malformations or developmental abnormalities, is difficult to obtain. Most existing work attempts to extract more effective image features through simple data augmentation methods (such as translation and flipping), but these methods offer limited improvement. Furthermore, it is important to consider whether these data augmentation methods provide positive improvements, rather than introducing irrelevant information that degrades model performance. For training anomaly detection algorithms, the paucity of abnormal data in clinical practice makes it difficult to obtain high-performing anomaly detection models.

[0005] Currently, existing medical image synthesis methods have the following main problems:

[0006] Existing technologies ignore the control of text over generated results, and can only synthesize based on a single ultrasound image, which has difficulties in scalability in the face of new categories; (2) Existing technology training is based on a large number of medical images, such as using 60,000 images to train a model to synthesize the pharynx, or using more than 30,000 datasets to train a brain MRI generation model. Such large-scale datasets are difficult to obtain in multi-section prenatal ultrasound; (3) Some technologies introduce auxiliary conditional information, such as edge maps, to make the generated image structure more realistic, but this condition limits the user-controllable customized image editing function, so that the model can only synthesize distributed data and it is difficult to synthesize ultrasound images outside the distribution range; (4) In most studies, rare abnormal cases are excluded; however, in clinical settings, the actual observed abnormal cases are more in demand, and the synthesis of abnormal cases is of great significance for practical applications; some technologies push the synthetic data to the label boundary at the feature level, but this method requires a large number of real positive samples to learn the data distribution, and also lacks interpretability at the image level. Summary of the Invention

[0007] In view of the problems existing in the prior art, the present invention proposes a multi-section prenatal

[0008] An ultrasound image generation model method is proposed. This method is based on a diffusion model and introduces layout boxes, local masks, and text as control conditions to synthesize multi-section prenatal ultrasound images with high medical image fidelity.

[0009] In order to solve the above problems, the present invention adopts the following technical solutions:

[0010] A method for generating a multi-slice prenatal ultrasound image model controlled by a layout frame, the method comprising the following steps:

[0011] Step 1: obtaining a set of prenatal ultrasound images of different sections and preprocessing each prenatal ultrasound image;

[0012] Step 2: Use a pre-trained object detection algorithm (such as PAIR software) to extract control conditions from each prenatal ultrasound image pre-processed in step 1. This includes extracting the layout box corresponding to the anatomical structure and the name of the structure it contains, marking the region of interest within the layout box, and creating a corresponding region of interest mask. For each prenatal ultrasound image, the corresponding structural text information is saved as a text control condition.

[0013] Step 3: Construct a diffusion generative model structure based on the U-Net network structure, add a control network structure to the diffusion generative model structure, and introduce scale-space adaptive sampling at the decoding end of the U-Net network structure;

[0014] Step 4: The prenatal ultrasound images obtained in step 1 and the corresponding control conditions extracted in step 2 constitute a training data set; based on the training data set, each prenatal ultrasound image is gradually noised through a diffusion process to obtain a noisy prenatal ultrasound image, each noisy prenatal ultrasound image is used as the input of the U-Net network structure, the corresponding control conditions extracted are used as the input of the control network structure, and the prenatal ultrasound image generated by combining the control conditions is used as the output. Combined with the loss function, the constructed diffusion generative model structure based on the U-Net network structure is trained to obtain a trained model, i.e., a prenatal ultrasound image generation model;

[0015] Step 5: The desired synthesized prenatal ultrasound image is gradually denoised through a diffusion process to obtain a noisy prenatal ultrasound image. The noisy prenatal ultrasound image is used as the input of the U-Net model, and the extracted corresponding layout box and structural text information are used as the input of the control network model. The prenatal ultrasound image generation model is applied to generate a synthesized prenatal ultrasound image. The user can freely edit the block diagram of each anatomical structure in the layout box and reapply the prenatal ultrasound image generation model to controllably generate a synthesized prenatal ultrasound image.

[0016] Furthermore, the preprocessing of the prenatal ultrasound image in step 1 includes cropping and normalization, and the image is adjusted to 512×512 pixels.

[0017] Furthermore, the control network structure includes encoding blocks of four resolutions and an intermediate block, with resolutions of 64×64, 32×32, 16×16, and 8×8 respectively. The encoding block of each resolution is repeated three times, and the output results are added to the corresponding jump connections and intermediate blocks of the U-Net network structure through zero convolution, and fused with the prenatal ultrasound image features extracted by the U-Net network structure to guide the generation process of prenatal ultrasound images.

[0018] Furthermore, the prenatal ultrasound image of the missing structure and the corresponding regional mask, the extracted layout box, and the structural text information are encoded by their respective adapted encoders and then input into the control network structure, where the structural text information is the global prompt information.

[0019] Furthermore, a pre-alignment feature module is added to the input end of the control network structure. The features of the layout box after passing through its encoder and the features of the ultrasound image combined with the regional mask after passing through its encoder are pre-aligned by the pre-alignment feature module and then input into the control network structure. Through the pre-alignment operation, the distance between different data at the distribution level is shortened, so that the sparse layout box features can better guide the controllable generation of prenatal ultrasound images.

[0020] Furthermore, the scale-space adaptive sampling is based on the grayscale difference changes of medical ultrasound images, and adopts isotropic diffusion with random directions and anisotropic diffusion that adjusts the diffusion intensity based on gradient information to remove image noise; while using isotropic diffusion to remove noise, the overall smoothness and details of the image are maintained; using anisotropic diffusion, the edges and details are retained by reducing the diffusion coefficient at the edge position, while removing noise; as the time step increases, the acoustic impedance difference of the organ edge in ultrasound imaging is retained, and the homogeneous area is denoised, thereby better preserving the image details and structure.

[0021] Furthermore, to further refine the image or synthesize abnormal samples, multi-scale mask sampling is introduced after scale-space adaptive sampling. DDIM (Denoising Diffusion Implicit Models) sampling is used in the mask area of each scale to extract image features. The features extracted from different scales are fused together to generate the final synthesized prenatal ultrasound image.

[0022] Furthermore, when training the diffusion generative model structure based on the U-Net network structure, the loss function includes the mean square error loss and the structural similarity loss of the mask area; the mean square error loss in the mask area is used as the local pixel difference loss generated by the model; in order to better integrate the global generation ability of the model, the structural similarity index loss is used to evaluate the structural similarity between the synthetic image and the real image; the specific expressions of the mean square error loss and the structural similarity index loss of the mask area are as follows:

[0023]

[0024] Where N is the number of samples, is the true value, is a generated value;

[0025]

[0026] in, x 、 y are real images and synthetic images respectively. 、 Is a real image x and composite images y The mean of 、 Is a real image x and composite images y variance; Is a real image x and composite images y covariance of 、 is a stability constant used to prevent the denominator from being zero.

[0027] Furthermore, the specific process of step 4 for training the diffusion generative model structure based on the U-Net network structure is as follows: based on the training data set, each prenatal ultrasound image is gradually denoised through a diffusion process to obtain a noisy prenatal ultrasound image, and each noisy prenatal ultrasound image is used as the input of the U-Net network structure, and the image features are extracted through the encoder; the extracted corresponding control conditions are used as the input of the control network structure, and the control condition features are extracted by the control network structure, and the control condition features are added to the jump connections and intermediate blocks corresponding to the U-Net network structure through zero convolution, and are fused with the prenatal ultrasound image features extracted by the U-Net network structure to guide the decoder of the U-Net network structure to generate predicted prenatal ultrasound images; combined with the loss function, the model parameters are optimized, and when the loss function converges, the training ends and the trained model is obtained.

[0028] Furthermore, in step 5, the user can freely edit the frame diagrams of each anatomical structure in the layout frame, including changing the relative position of the frame diagrams, adjusting the size of the structures, constructing missing structures, and making masks of the corresponding missing structures. The changed layout frame and regional mask are used as new control conditions, and the prenatal ultrasound image generation model is applied to output the expected synthesized prenatal ultrasound image.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] Based on the synthesis of realistic multi-section prenatal ultrasound images, the present invention can generate ultrasound images with different anatomical structures based on user-editable functions. This technology is of great significance for synthesizing images of malformed or deformed fetuses. In addition, the present invention can be used as a refined means to improve low-quality synthesized ultrasound images, restore the texture of ultrasound images and enhance anatomical structure details. By applying the synthesis results to AI model training for downstream tasks, the performance of the classification model can be effectively improved by mixing the synthesized multi-section normal ultrasound images and real data, and the large number of synthesized abnormal samples can be used to train the anomaly detection model, solving the problem of scarcity of abnormal samples in clinical practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flow chart of the method provided by the present invention;

[0032] Figure 2 A specific architecture diagram of the diffusion generative model structure based on the U-Net network structure provided by the present invention;

[0033] Figure 3 An architectural diagram of a control network of a diffusion-generating model structure provided by the present invention;

[0034] Figure 4 This is a visualization effect diagram of a prenatal ultrasound image obtained using the model provided by the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0036] like Figure 1 、 Figure 2 、 Figure 3 As shown, the present invention provides a multi-section prenatal ultrasound image generation model method controlled by a layout frame, the method comprising the following steps:

[0037] Step 1: Obtain a set of prenatal ultrasound images of three sections, crop and normalize each prenatal ultrasound image, and adjust the image to 512×512 pixels;

[0038] Step 2: Use the pair software to extract control conditions from each prenatal ultrasound image preprocessed in step 1. This includes extracting the bounding box of each organ, marking the region of interest, and combining the independent bounding boxes into a layout box; creating a corresponding region of interest mask based on the coordinate position of the region of interest; and saving the corresponding structural text information for each prenatal ultrasound image as a text control condition.

[0039] Step 3: Construct a diffusion generative model structure based on the U-Net network structure, and add a control network structure to the diffusion generative model structure. The control network structure includes coding blocks of four resolutions and an intermediate block, with resolutions of 64×64, 32×32, 16×16, and 8×8, respectively. The coding block of each resolution is repeated three times, and the output result is added to the jump connection and intermediate block corresponding to the U-Net network structure through zero convolution, and fused with the image features extracted by the U-Net network structure; a pre-alignment module is added to the input end of the control network structure; and scale space adaptive sampling and multi-scale mask sampling are introduced at the decoding end of the U-Net network structure.

[0040] Step 4: The prenatal ultrasound images obtained in step 1 and the corresponding control conditions extracted in step 2 constitute a training data set; based on the training data set, each prenatal ultrasound image is gradually noised through a diffusion process to obtain a noisy prenatal ultrasound image, each noisy prenatal ultrasound image is used as the input of the U-Net network structure, the corresponding control conditions extracted are used as the input of the control network structure, and the synthetic prenatal ultrasound image generated by combining the control conditions is used as the output. Combined with the loss function, the constructed diffusion generative model structure based on the U-Net network structure is trained to obtain a trained model, i.e., a prenatal ultrasound image generation model;

[0041] Step 5: The desired synthesized prenatal ultrasound image is gradually denoised through a diffusion process to obtain a noisy prenatal ultrasound image. The noisy prenatal ultrasound image is used as the input of the U-Net model, and the extracted corresponding layout box and structural text information are used as the input of the control network model. The prenatal ultrasound image generation model is applied to generate a synthesized prenatal ultrasound image. The user can freely edit the block diagram of each anatomical structure in the layout box and reapply the prenatal ultrasound image generation model to controllably generate a synthesized prenatal ultrasound image.

[0042] The scale-space adaptive sampling is based on the grayscale difference changes of medical ultrasound images, and adopts isotropic diffusion with random directions and anisotropic diffusion with diffusion intensity adjusted based on gradient information to remove image noise; while using isotropic diffusion to remove noise, the overall smoothness and details of the image are maintained; using anisotropic diffusion, the edges and details are retained while removing noise by reducing the diffusion coefficient at the edge position; as the time step increases, the acoustic impedance difference of the organ edge in ultrasound imaging is retained, and the homogeneous area is denoised, thereby better preserving the image details and structure.

[0043] To further refine images or synthesize abnormal samples, multi-scale mask sampling is introduced after scale-space adaptive sampling. DDIM (Denoising Diffusion Implicit Models) sampling is applied to the mask region at each scale to extract image features. Image features extracted from different scales are fused together to generate the final synthesized prenatal ultrasound image.

[0044] When training the model structure, the loss function includes the mean square error loss and structural similarity loss of the mask area; the mean square error loss in the mask area is used as the local pixel difference loss generated by the model; in order to better integrate the global generation ability of the model, the structural similarity index loss is used to evaluate the structural similarity between the synthetic image and the real image; the specific expressions of the mean square error loss and structural similarity index loss of the mask area are as follows:

[0045]

[0046] Where N is the number of samples, is the true value, is a generated value;

[0047]

[0048] in, x 、 y are real images and synthetic images respectively. 、 Is a real image x and composite images yThe mean of 、 Is a real image x and composite images y variance; Is a real image x and composite images y covariance of 、 is a stability constant used to prevent the denominator from being zero.

[0049] Based on the training data set, each prenatal ultrasound image is gradually denoised through a diffusion process to obtain a noisy prenatal ultrasound image. Each noisy prenatal ultrasound image is used as the input of the U-Net network structure, and the image features are extracted through the encoder. The extracted corresponding control conditions are used as the input of the control network structure, and the control condition features are extracted by the control network structure. The control condition features are added to the corresponding jump connections and intermediate blocks of the U-Net network structure through zero convolution, and are fused with the prenatal ultrasound image features extracted by the U-Net network structure to guide the decoder of the U-Net network structure to generate predicted prenatal ultrasound images. Combined with the loss function, the model parameters are optimized. When the loss function converges, the training ends and the trained model is obtained.

[0050] For the desired synthesized prenatal ultrasound image, the user can freely edit the block diagrams of each anatomical structure in the layout frame, including changing the relative position of the block diagrams, adjusting the structure size, constructing missing structures, and making masks for the corresponding missing structures. The prenatal ultrasound image generation model is applied using the modified layout frame and regional mask as new control conditions to generate the expected synthesized prenatal ultrasound image. Figure 4 A visualization diagram of multi-section prenatal ultrasound images generated by applying the prenatal ultrasound image generation model is presented.

[0051] The present invention provides a multi-section prenatal ultrasound image generation model method controlled by a layout frame. The proposed region of interest masking method can make the model applicable to small-scale data sets, and the layout frame control method can accurately allow the model to synthesize ultrasound images with controllable appearance. The operation is simple and user-friendly. By using an encoder adapted to different control conditions, the effective features of the control conditions are extracted, so that the model can learn the relationship between the control conditions and the image, thereby quickly generating corresponding ultrasound images according to user needs.

[0052] The method provided by the present invention utilizes the powerful learning ability of the generative model, and has the characteristics of global consistency, local diversity and controllability of generated images. It can assist in the training of clinicians and provide a large amount of synthetic data support for training deep learning models.

Claims

1. A method for generating a multi-section prenatal ultrasound image model controlled by a layout frame, characterized in that: The method comprises the following steps: Step 1: obtaining a set of prenatal ultrasound images of different sections and preprocessing each prenatal ultrasound image; Step 2: Using a pre-trained object detection algorithm, control conditions are extracted from each prenatal ultrasound image pre-processed in step 1, including extracting the layout box corresponding to the anatomical structure and the name of the structure contained therein; marking the region of interest within the layout box and creating a corresponding region of interest mask; and saving the corresponding structural text information for each prenatal ultrasound image as a text control condition. Step 3: Construct a diffusion generative model structure based on a U-Net network structure, add a control network structure to the diffusion generative model structure, and introduce scale-space adaptive sampling at the decoding end of the U-Net network structure; scale-space adaptive sampling is based on the grayscale difference changes of medical ultrasound images, adopts isotropic diffusion with random directions and anisotropic diffusion with diffusion intensity adjusted based on gradient information, retains the acoustic impedance difference of organ edges in ultrasound imaging, and denoises homogeneous areas at the same time; after scale-space adaptive sampling, multi-scale mask sampling is introduced, and DDIM sampling is applied to the mask area of each scale to extract image features, and the features extracted from different scales are fused together to generate a synthetic prenatal ultrasound image; Step 4: The prenatal ultrasound images obtained in step 1 and the corresponding control conditions extracted in step 2 constitute a training data set; based on the training data set, each prenatal ultrasound image is gradually noised through a diffusion process to obtain a noisy prenatal ultrasound image, each noisy prenatal ultrasound image is used as the input of the U-Net network structure, the corresponding control conditions extracted are used as the input of the control network structure, and the synthetic prenatal ultrasound image generated by combining the control conditions is used as the output. Combined with the loss function, the constructed diffusion generative model structure based on the U-Net network structure is trained to obtain a trained model, i.e., a prenatal ultrasound image generation model; Step 5. The desired synthesized prenatal ultrasound image is gradually denoised through a diffusion process to obtain a noisy prenatal ultrasound image. The noisy prenatal ultrasound image is used as the input of the U-Net model, and the corresponding layout box and structural text information are used as the input of the control network model. The prenatal ultrasound image generation model is applied to generate a synthesized prenatal ultrasound image. The user can freely edit the block diagram of each anatomical structure in the layout box and reapply the prenatal ultrasound image generation model to controllably generate a synthesized prenatal ultrasound image.

2. The method for generating a multi-slice prenatal ultrasound image model controlled by a layout frame according to claim 1, characterized in that: In step 1, the preprocessing of each prenatal ultrasound image includes cropping and normalization, and the image is adjusted to 512×512 pixels.

3. The method for generating a multi-slice prenatal ultrasound image model controlled by a layout frame according to claim 2, characterized in that: The control network structure contains encoding blocks of four resolutions and an intermediate block, with resolutions of 64×64, 32×32, 16×16, and 8×8 respectively. The encoding block of each resolution is repeated three times, and the output results are added to the corresponding jump connections and intermediate blocks of the U-Net network structure through zero convolution, and fused with the image features extracted by the U-Net network structure to guide the image generation process.

4. The method for generating a multi-slice prenatal ultrasound image model controlled by a layout frame according to claim 3, wherein: The prenatal ultrasound image of the missing structure and the corresponding regional mask, the extracted layout box, and the structural text information are encoded by their respective adapted encoders and then input into the control network structure, where the structural text information is the global prompt information.

5. The method for generating a multi-slice prenatal ultrasound image model controlled by a layout frame according to claim 4, characterized in that: A pre-alignment feature module is added to the input end of the control network structure. The features of the layout frame after passing through its encoder and the features of the ultrasound image of the combined area mask after passing through its encoder are pre-aligned by the pre-alignment feature module and then input into the control network structure.

6. The method for generating a multi-slice prenatal ultrasound image model controlled by a layout frame according to claim 1, characterized in that: When training the diffusion generative model structure based on the U-Net network structure, the loss functions used include the mean square error loss of the mask area and the structural similarity index loss. The specific expressions of the mean square error loss of the mask area and the structural similarity index loss are as follows: Where N is the number of samples, y i is the true value, is a generated value; Among them, x and y are real image and synthetic image respectively, μ x 、μ y is the mean of the real image x and the synthesized image y, is the variance between the real image x and the synthesized image y; σ xy is the covariance between the real image x and the synthesized image y; C1 and C2 are stable constants used to avoid the denominator being zero.

7. The method for generating a multi-slice prenatal ultrasound image model controlled by a layout frame according to claim 1, characterized in that: In step 5, the user can freely edit the frame diagrams of each anatomical structure in the layout frame, including changing the relative position of the frame diagrams, adjusting the structure size, constructing missing structures, and making masks of corresponding missing structures. The changed layout frame and regional mask are used as new control conditions, and the prenatal ultrasound image generation model is applied to output the expected synthesized prenatal ultrasound image.

Images