Multi-section antenatal ultrasound image generation model method controlled by layout frame

Abstract

The invention discloses a multi-section antenatal ultrasound image generation model method controlled by a layout frame, and belongs to the field of deep learning technology generation algorithms. Forming a training set by using a control condition consisting of the obtained antenatal ultrasound image set, the extracted layout frame of the antenatal ultrasound image anatomy structure and the mask of the region of interest, and training the constructed model to obtain an antenatal ultrasound image generation model; and for the antenatal ultrasound image needing to be synthesized, an antenatal ultrasound image generation model is applied, and an expected synthesized antenatal ultrasound image is obtained in combination with a new control condition composed of the changed layout frame and the mask area. According to the region-of-interest masking method provided by the invention, the model can be suitable for a small-scale data set, the model can be accurately synthesized into the ultrasonic image with the controllable appearance through the layout frame control method, the operation is simple and convenient, the user-friendly effect is realized, and the corresponding ultrasonic image can be quickly generated according to the user demand.

Description

Technical Field

[0001] The present invention belongs to the field of generative algorithms of deep learning technology, and in particular to a multi-section prenatal ultrasound image synthesis method controlled by a layout frame. Background Art

[0002] Prenatal ultrasound examinations play a vital role in evaluating fetal development and screening for malformations. Especially in the first and second trimesters, prenatal ultrasound examinations usually involve screening of dozens of ultrasound sections, which are essential for a comprehensive assessment of the health of the fetus. Given its complexity and high requirements for accuracy, such examinations are usually performed by experienced radiologists to ensure that potential abnormalities can be identified and an accurate diagnosis can be made.

[0003] In clinical practice, prenatal screening requires a comprehensive screening of the fetus, but the fetus moves frequently during the examination, which makes it difficult to collect prenatal ultrasound images and the amount of data is relatively small, which has an adverse effect on the training of new radiologists. In addition, during the training of ultrasound physicians, it is very difficult to obtain images of multiple fetal sections in a single posture. For certain specific ultrasound sections, different fetuses may present completely different postures, which makes the qualitative learning of doctors complicated and difficult. In clinical practice, cases of deformity or maldevelopment are very rare. For example, the probability of cleft lip and palate is only one in a thousand, which means that the training of radiologists often requires a large number of abnormal cases for learning.

[0004] In recent years, the development of artificial intelligence (AI) technology has brought revolutionary changes to the field of prenatal ultrasound. Existing AI-assisted systems have shown great potential in improving the efficiency and accuracy of prenatal ultrasound screening. Through advanced algorithms such as deep learning, AI tools can not only assist doctors in quickly locating key anatomical structures, but also assist in identifying abnormal features, thereby optimizing the screening process. In terms of AI model training, the training of deep learning models requires a large amount of data, but prenatal ultrasound medical data, especially image data of rare diseases such as dysplasia or deformity, is difficult to obtain. Most existing works try to obtain more effective image features through simple data enhancement methods (such as translation, flipping, etc.), but the improvement brought by these methods is very limited. In addition, it is necessary to consider whether these data enhancement methods can bring positive improvement, rather than introducing irrelevant information, resulting in a decrease in model performance. For the training of anomaly detection algorithms, there is very little abnormal data in clinical practice, and it is difficult to obtain anomaly detection models with excellent performance.

[0005] At present, the existing technology for medical image synthesis methods mainly has the following problems: Existing technologies ignore the control of text over generated results, so they can only synthesize based on a single ultrasound image, and have difficulties in scalability when facing new categories; (2) Existing technology training is based on a large number of medical images, such as using 60,000 images to train a model to synthesize the pharynx, or using more than 30,000 data sets to train a brain MRI generation model. Such large-scale data sets are difficult to obtain in multi-section prenatal ultrasound; (3) Some technologies introduce auxiliary conditional information, such as edge maps, to make the generated image structure more realistic, but this condition limits the user-controllable customized image editing function, so that the model can only synthesize distributed data and it is difficult to synthesize ultrasound images outside the distribution range; (4) In most studies, rare abnormal cases are excluded; however, in clinical settings, the actual observed abnormal cases have greater actual needs, and the synthesis of abnormal cases is of great significance for practical applications; Some technologies push the synthetic data to the label boundary at the feature level, but this method requires a large number of real positive samples to learn the data distribution, and also lacks interpretability at the image level. Summary of the invention

[0006] In view of the problems existing in the prior art, the present invention proposes a multi-section prenatal An ultrasound image generation model method is proposed. This method is based on a diffusion model and introduces layout boxes, local masks, and text as control conditions to synthesize multi-section prenatal ultrasound images with high medical image fidelity.

[0007] To solve the above problems, the present invention adopts the following technical solutions: A method for generating a model of a multi-section prenatal ultrasound image controlled by a layout frame, the method comprising the following steps: Step 1, obtaining a set of prenatal ultrasound images of different sections, and preprocessing each prenatal ultrasound image; Step 2: Use a pre-trained target detection algorithm (such as pair software) to extract control conditions from each prenatal ultrasound image pre-processed in step 1, including extracting the layout box corresponding to the anatomical structure and the structure name contained therein, marking the region of interest in the layout box and making a corresponding region of interest mask, and saving the corresponding structural text information for each prenatal ultrasound image as a text control condition; Step 3: construct a diffusion generative model structure based on the U-Net network structure, add a control network structure to the diffusion generative model structure, and introduce scale space adaptive sampling at the decoding end of the U-Net network structure; Step 4, the prenatal ultrasound images obtained in step 1 and the corresponding control conditions extracted in step 2 constitute a training data set; based on the training data set, each prenatal ultrasound image is gradually denoised through a diffusion process to obtain a noisy prenatal ultrasound image, each noisy prenatal ultrasound image is used as an input of the U-Net network structure, the corresponding control conditions extracted are used as an input of the control network structure, and the prenatal ultrasound image generated in combination with the control conditions is used as an output, and the diffusion generative model structure constructed based on the U-Net network structure is trained in combination with the loss function to obtain a trained model, i.e., a prenatal ultrasound image generation model; Step 5. The desired synthesized prenatal ultrasound image is gradually denoised through a diffusion process to obtain a noisy prenatal ultrasound image. The noisy prenatal ultrasound image is used as the input of the U-Net model, and the corresponding layout frame and structural text information extracted are used as the input of the control network model. The prenatal ultrasound image generation model is applied to generate a synthesized prenatal ultrasound image. The user can freely edit the block diagram of each anatomical structure in the layout frame, reapply the prenatal ultrasound image generation model, and controllably generate a synthesized prenatal ultrasound image.

[0008] Furthermore, the preprocessing of the prenatal ultrasound image in step 1 includes cropping and normalization, and the image is adjusted to 512×512 pixels.

[0009] Furthermore, the control network structure includes encoding blocks of four resolutions and an intermediate block, with resolutions of 64×64, 32×32, 16×16, and 8×8 respectively. The encoding block of each resolution is repeated three times, and the output results are added to the jump connections and intermediate blocks corresponding to the U-Net network structure through zero convolution, and are fused with the prenatal ultrasound image features extracted by the U-Net network structure to guide the generation process of prenatal ultrasound images.

[0010] Furthermore, the prenatal ultrasound image of the missing structure and the corresponding regional mask, the extracted layout frame, and the structural text information are encoded by their respective adapted encoders and then input into the control network structure, and the structural text information is the global prompt information.

[0011] Furthermore, a pre-alignment feature module is added to the input end of the control network structure, and the features of the layout box after passing through its encoder and the features of the ultrasound image combined with the regional mask after passing through its encoder are pre-aligned by the pre-alignment feature module and then input into the control network structure; through the pre-alignment operation, different data are brought closer at the distribution level, so that the sparse layout box features can better guide the controllable generation of prenatal ultrasound images.

[0012] Furthermore, the scale space adaptive sampling is based on the grayscale difference changes of medical ultrasound images, and adopts isotropic diffusion with random directions and anisotropic diffusion that adjusts the diffusion intensity based on gradient information to remove image noise; while isotropic diffusion is used to remove noise, the overall smoothness and details of the image are maintained; anisotropic diffusion is used to retain edges and details while removing noise by reducing the diffusion coefficient at the edge position; as the time step increases, the acoustic impedance difference of the organ edge in ultrasound imaging is retained, and the homogeneous area is denoised, thereby better retaining the image details and structure.

[0013] Furthermore, in order to further refine the image or synthesize abnormal samples, multi-scale mask sampling is introduced after scale space adaptive sampling. DDIM (Denoising Diffusion Implicit Models) sampling is used in the mask area of ​​each scale to extract image features; the features extracted from different scales are fused together to generate the final synthesized prenatal ultrasound image.

[0014] Furthermore, when training the diffusion generative model structure based on the U-Net network structure, the loss function includes the mean square error loss and the structural similarity loss of the mask area; the mean square error loss in the mask area is used as the local pixel difference loss generated by the model; in order to better integrate the global generation ability of the model, the structural similarity index loss is used to evaluate the structural similarity between the synthetic image and the real image; the specific expressions of the mean square error loss and the structural similarity index loss of the mask area are as follows:

[0015] Where N is the number of samples, is the true value, is the generated value;

[0016] in, x , y are real images and synthetic images respectively. , Is a real image x and synthetic images y The mean of , Is a real image x and synthetic images y The variance of Is a real image x and synthetic images y The covariance of , is a stability constant used to avoid the denominator being zero.

[0017] Furthermore, the specific process of step 4 for training the diffusion generative model structure based on the U-Net network structure is as follows: based on the training data set, each prenatal ultrasound image is gradually denoised through a diffusion process to obtain a noisy prenatal ultrasound image, and each noisy prenatal ultrasound image is used as the input of the U-Net network structure, and the image features are extracted through the encoder; the corresponding control conditions extracted are used as the input of the control network structure, and the control condition features are extracted by the control network structure, and the control condition features are added to the jump connections and intermediate blocks corresponding to the U-Net network structure through zero convolution, and are fused with the prenatal ultrasound image features extracted by the U-Net network structure to guide the decoder of the U-Net network structure to generate predicted prenatal ultrasound images; combined with the loss function, the model parameters are optimized, and when the loss function converges, the training ends and the trained model is obtained.

[0018] Furthermore, in step 5, the user can freely edit the frame diagrams of each anatomical structure in the layout frame, including changing the relative position of the frame diagram, adjusting the size of the structure, constructing missing structures, and making masks of corresponding missing structures. The changed layout frame and regional mask are used as new control conditions, and the prenatal ultrasound image generation model is applied to output the expected synthetic prenatal ultrasound image.

[0019] Compared with the prior art, the present invention has the following beneficial effects: Based on the synthesis of realistic multi-section prenatal ultrasound images, the present invention can generate ultrasound images with different anatomical structures based on user-editable functions. This technology is of great significance for synthesizing images of malformed or deformed fetuses. In addition, the present invention can be used as a refined means to improve low-quality synthesized ultrasound images, restore the texture of ultrasound images and enhance anatomical structure details. By applying the synthesis results to AI model training for downstream tasks, the performance of the classification model can be effectively improved by mixing the synthesized multi-section normal ultrasound images and real data, and the large number of synthesized abnormal samples can be used to train the anomaly detection model to solve the problem of scarcity of abnormal samples in clinical practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flow chart of the method provided by the present invention; Figure 2 A specific architecture diagram of the diffusion generation model structure based on the U-Net network structure provided by the present invention; Figure 3 An architectural diagram of a control network of a diffusion generation model structure provided by the present invention; Figure 4 This is a visualization effect diagram of a prenatal ultrasound image obtained using the model provided by the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0022] like Figure 1 , Figure 2 , Figure 3 As shown, the present invention provides a multi-section prenatal ultrasound image generation model method controlled by a layout frame, the method comprising the following steps: Step 1, obtaining a set of prenatal ultrasound images of three sections, cropping and normalizing each prenatal ultrasound image, and adjusting the image to 512×512 pixels; Step 2, using pair software to extract control conditions from each prenatal ultrasound image preprocessed in step 1: including extracting the bounding box of each organ, marking the region of interest, and synthesizing the independent bounding boxes into a layout box; making a corresponding region of interest mask according to the coordinate position of the region of interest; saving the corresponding structural text information for each prenatal ultrasound image as a text control condition; Step 3, construct a diffusion generative model structure based on the U-Net network structure, and add a control network structure to the diffusion generative model structure, the control network structure includes four encoding blocks of resolution and an intermediate block, the resolutions are 64×64, 32×32, 16×16, 8×8, the encoding block of each resolution is repeated three times, the output result is added to the jump connection and the intermediate block corresponding to the U-Net network structure through zero convolution, and is fused with the image features extracted by the U-Net network structure; and a pre-alignment module is added to the input end of the control network structure; scale space adaptive sampling and multi-scale mask sampling are introduced at the decoding end of the U-Net network structure; Step 4, the prenatal ultrasound images obtained in step 1 and the corresponding control conditions extracted in step 2 constitute a training data set; based on the training data set, each prenatal ultrasound image is gradually denoised through a diffusion process to obtain a noisy prenatal ultrasound image, each noisy prenatal ultrasound image is used as an input of the U-Net network structure, the corresponding control conditions extracted are used as an input of the control network structure, and the synthetic prenatal ultrasound image generated by combining the control conditions is used as an output, and the diffusion generative model structure constructed based on the U-Net network structure is trained in combination with the loss function to obtain a trained model, i.e., a prenatal ultrasound image generation model; Step 5. The desired synthesized prenatal ultrasound image is gradually denoised through a diffusion process to obtain a noisy prenatal ultrasound image. The noisy prenatal ultrasound image is used as the input of the U-Net model, and the corresponding layout frame and structural text information extracted are used as the input of the control network model. The prenatal ultrasound image generation model is applied to generate a synthesized prenatal ultrasound image. The user can freely edit the block diagram of each anatomical structure in the layout frame, reapply the prenatal ultrasound image generation model, and controllably generate a synthesized prenatal ultrasound image.

[0023] The scale space adaptive sampling is based on the grayscale difference change of medical ultrasound images, and adopts isotropic diffusion with random direction and anisotropic diffusion that adjusts the diffusion intensity based on gradient information to remove image noise; while isotropic diffusion is used to remove noise, the overall smoothness and details of the image are maintained; anisotropic diffusion is used to retain edges and details while removing noise by reducing the diffusion coefficient of edge positions; as the time step increases, the acoustic impedance difference of the organ edge in ultrasound imaging is retained, and the homogeneous area is denoised, thereby better retaining image details and structure.

[0024] In order to further refine the requirements of images or synthesize abnormal samples, the multi-scale mask sampling is introduced after the scale space adaptive sampling, and DDIM (Denoising Diffusion Implicit Models) sampling is used in the mask area of ​​each scale to extract image features; the image features extracted from different scales are fused together to generate the final synthetic prenatal ultrasound image.

[0025] When training the model structure, the loss function includes the mean square error loss and the structural similarity loss of the mask area; the mean square error loss in the mask area is used as the local pixel difference loss generated by the model; in order to better integrate the global generation ability of the model, the structural similarity index loss is used to evaluate the structural similarity between the synthetic image and the real image; the specific expressions of the mean square error loss and the structural similarity index loss of the mask area are as follows:

[0026] Where N is the number of samples, is the true value, is the generated value;

[0027] in, x , y are real images and synthetic images respectively. , Is a real image x and synthetic images y The mean of , Is a real image x and synthetic images y The variance of Is a real image x and synthetic images y The covariance of , is a stability constant used to avoid the denominator being zero.

[0028] Based on the training data set, each prenatal ultrasound image is gradually denoised through a diffusion process to obtain a noisy prenatal ultrasound image, and each noisy prenatal ultrasound image is used as the input of the U-Net network structure, and the image features are extracted through the encoder; the extracted corresponding control conditions are used as the input of the control network structure, and the control condition features are extracted by the control network structure. The control condition features are added to the jump connections and intermediate blocks corresponding to the U-Net network structure through zero convolution, and are fused with the prenatal ultrasound image features extracted by the U-Net network structure to guide the decoder of the U-Net network structure to generate predicted prenatal ultrasound images; combined with the loss function, the model parameters are optimized. When the loss function converges, the training ends and the trained model is obtained.

[0029] For the desired synthesized prenatal ultrasound image, the user can freely edit the block diagram of each anatomical structure in the layout frame, including changing the relative position of the block diagram, adjusting the size of the structure, constructing the missing structure, and making the mask of the corresponding missing structure. The changed layout frame and regional mask are used as new control conditions, and the prenatal ultrasound image generation model is applied to generate the expected synthesized prenatal ultrasound image; Figure 4 A visualization diagram of a multi-section prenatal ultrasound image generated by using the prenatal ultrasound image generation model is shown.

[0030] The present invention provides a multi-section prenatal ultrasound image generation model method controlled by a layout frame. The region of interest mask method proposed by the present invention can make the model suitable for small-scale data sets, and the layout frame control method can accurately allow the model to synthesize ultrasound images with controllable appearance, and the operation is simple and user-friendly. By using an encoder adapted to different control conditions, the effective features of the control conditions are extracted, so that the model can learn the connection between the control conditions and the image, thereby quickly generating corresponding ultrasound images according to user needs.

[0031] The method provided by the present invention utilizes the powerful learning ability of the generative model, and has the characteristics of global consistency, local diversity and controllability of generated images. It can assist in the training of clinicians and provide a large amount of synthetic data support for training deep learning models.

Claims

1. A method for generating a multi-section prenatal ultrasound image model controlled by a layout frame, characterized in that: The method comprises the following steps: Step 1, obtaining a set of prenatal ultrasound images of different sections, and preprocessing each prenatal ultrasound image; Step 2: Use the pre-trained target detection algorithm to extract control conditions from each prenatal ultrasound image pre-processed in step 1, including extracting the layout box corresponding to the anatomical structure and the structure name contained therein; marking the region of interest in the layout box and making a corresponding region of interest mask; and saving the corresponding structural text information for each prenatal ultrasound image as a text control condition; Step 3: construct a diffusion generative model structure based on the U-Net network structure, add a control network structure to the diffusion generative model structure, and introduce scale space adaptive sampling at the decoding end of the U-Net network structure; Step 4, the prenatal ultrasound images obtained in step 1 and the corresponding control conditions extracted in step 2 constitute a training data set; based on the training data set, each prenatal ultrasound image is gradually denoised through a diffusion process to obtain a noisy prenatal ultrasound image, each noisy prenatal ultrasound image is used as an input of the U-Net network structure, the corresponding control conditions extracted are used as an input of the control network structure, and the synthetic prenatal ultrasound image generated by combining the control conditions is used as an output, and the diffusion generative model structure constructed based on the U-Net network structure is trained in combination with the loss function to obtain a trained model, i.e., a prenatal ultrasound image generation model; Step 5. The desired synthesized prenatal ultrasound image is gradually denoised through a diffusion process to obtain a noisy prenatal ultrasound image. The noisy prenatal ultrasound image is used as the input of the U-Net model, and the corresponding layout box and structural text information are used as the input of the control network model. The prenatal ultrasound image generation model is applied to generate a synthesized prenatal ultrasound image. The user can freely edit the block diagram of each anatomical structure in the layout box, reapply the prenatal ultrasound image generation model, and controllably generate a synthesized prenatal ultrasound image.

2. The method for generating a multi-section prenatal ultrasound image model controlled by a layout frame according to claim 1, characterized in that: The preprocessing of each prenatal ultrasound image in step 1 includes cropping and normalization, and the image is adjusted to 512×512 pixels.

3. The method for generating a multi-section prenatal ultrasound image model controlled by a layout frame according to claim 2, characterized in that: The control network structure includes encoding blocks of four resolutions and an intermediate block. The resolutions are 64×64, 32×32, 16×16, and 8×8 respectively. The encoding block of each resolution is repeated three times. The output results are added to the jump connections and intermediate blocks corresponding to the U-Net network structure through zero convolution, and are fused with the image features extracted by the U-Net network structure to guide the image generation process.

4. The method for generating a multi-section prenatal ultrasound image model controlled by a layout frame according to claim 3, characterized in that: The prenatal ultrasound image of the missing structure and the corresponding regional mask, the extracted layout frame, and the structural text information are encoded by respective adapted encoders and then input into the control network structure, and the structural text information is the global prompt information.

5. The method for generating a multi-section prenatal ultrasound image model controlled by a layout frame according to claim 4, characterized in that: A pre-alignment feature module is added to the input end of the control network structure, and the features of the layout frame after passing through its encoder and the features of the ultrasonic image combined with the regional mask after passing through its encoder are pre-aligned by the pre-alignment feature module and then input into the control network structure.

6. The method for generating a multi-section prenatal ultrasound image model controlled by a layout frame according to claim 1, characterized in that: The scale space adaptive sampling is based on the grayscale difference change of medical ultrasound images, adopts isotropic diffusion with random direction and anisotropic diffusion with diffusion intensity adjusted based on gradient information, retains the acoustic impedance difference of the organ edge in ultrasound imaging, and denoises the homogeneous area at the same time.

7. The method for generating a multi-section prenatal ultrasound image model controlled by a layout frame according to claim 6, characterized in that: After scale-space adaptive sampling, multi-scale mask sampling is introduced, and DDIM sampling is applied to the mask area of ​​each scale to extract image features. Features extracted from different scales are fused together to generate synthetic prenatal ultrasound images.

8. The method for generating a multi-section prenatal ultrasound image model controlled by a layout frame according to claim 1, characterized in that: When training the diffusion generative model structure based on the U-Net network structure, the loss functions used include the mean square error loss of the mask area and the structural similarity index loss; the specific expressions of the mean square error loss of the mask area and the structural similarity index loss are as follows: Where N is the number of samples, is the true value, is the generated value; in, x , y are real images and synthetic images respectively. , Is a real image x and synthetic images y The mean of , Is a real image x and synthetic images y The variance of Is a real image x and synthetic images y The covariance of , is a stability constant used to avoid the denominator being zero.

9. A multi-section prenatal ultrasound image generation model controlled by a layout frame according to claim 1 A method of this type, characterized in that In step 5, the user can freely edit the frame diagrams of each anatomical structure in the layout frame, including changing the relative position of the frame diagram, adjusting the size of the structure, constructing missing structures, and making masks of the corresponding missing structures. The changed layout frame and regional mask are used as new control conditions, and the prenatal ultrasound image generation model is applied to output the expected synthetic prenatal ultrasound image.

Images