HRCT image generation method and system based on conditional control diffusion model

Abstract

The invention discloses an HRCT image generation method and system based on a conditional control diffusion model, and relates to the technical field of image processing, and the method comprises the steps: collecting a lesion image to construct a data set, and employing an image processing algorithm to recognize and extract a to-be-detected region from the data set; determining a seed point by analyzing texture characteristics of the to-be-detected region, and inputting the seed point into the image segmentation model to obtain an image segmentation result; classifying and marking the segmented images by using an image analysis technology to obtain an image classification result; and based on an image classification result, optimizing a preset image diffusion model, and fusing the image diffusion model with the condition regulation and control network to generate a high-resolution image. According to the method, by establishing the data set and adopting the image processing technology, the high-quality image is generated while the labeling cost is reduced, a more reliable data basis is provided for a subsequent image segmentation task, and the detection efficiency of the image is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for generating HRCT images based on a conditional controlled diffusion model. Background Art

[0002] High-resolution computed tomography (HRCT) is of great significance for the prediction and prognosis assessment of interstitial lung disease (ILD). However, the imaging manifestations of ILD are complex, including various lesion types such as ground-glass shadows, honeycomb shadows, and lattice shadows. Different types of ILD lesions are difficult to distinguish. Traditional quantitative CT analysis methods do not include the spatial distribution characteristics of lesions, have poor sensitivity to mixed lesions, and have poor repeatability.

[0003] Due to the complex imaging manifestations of interstitial lung diseases, different types of ILD lesions are easily confused, and the lesions do not have obvious contour boundaries, which requires a large amount of high-quality training data for pixel-level deep learning segmentation algorithms. This leads to the following deficiencies in existing technologies:

[0004] 1. The high-quality segmentation and annotation of a large number of interstitial lung diseases is not only a time-consuming and labor-intensive task, but also prone to mislabeling or missing labels due to the significant subjectivity, inter-observer and intra-observer variability, and relatively low sensitivity of radiologists' visual assessment of HRCT, thus affecting the segmentation accuracy of the algorithm.

[0005] 2. Due to privacy, high labor costs for labeling, and a general lack of data samples for rare lesions, it is very difficult to obtain a large amount of high-quality labeled data in the medical field. The scarcity and imbalance of data lead to weak generalization capabilities of existing segmentation algorithms, making them difficult to implement and apply.

[0006] 3. Existing algorithms use technology based on generative adversarial networks (GANs) as a means of data enhancement. However, compared with large image generation models based on diffusion, GANs have weaker generation capabilities and it is more difficult to achieve conditional control of the generation results.

[0007] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0008] In view of the problems in the related art, the present invention proposes a HRCT image generation method and system based on a conditional controlled diffusion model to overcome the above technical problems existing in the existing related art.

[0009] To this end, the specific technical solution adopted by the present invention is as follows:

[0010] According to one aspect of the present invention, a method for generating HRCT images based on a conditional controlled diffusion model is provided, the method comprising the following steps:

[0011] S1, collecting lesion images to construct a data set, and using an image processing algorithm to identify and extract the area to be detected from the data set;

[0012] S2. Determine a seed point by analyzing the texture characteristics of the area to be detected, and input the seed point into the image segmentation model to obtain an image segmentation result;

[0013] S3. Use image analysis technology to classify and label the segmented images to obtain image classification results;

[0014] S4. Based on the image classification results, optimize the preset image diffusion model, and fuse the image diffusion model with the conditional regulation network to generate a high-resolution image.

[0015] Furthermore, seed points are determined by analyzing the texture characteristics of the area to be detected, and the seed points are input into the image segmentation model. Obtaining the image segmentation result includes the following steps:

[0016] S21, extracting texture features of the area to be detected, and generating seed points based on the texture features of the area to be detected;

[0017] S22: input the seed point into the image segmentation model, and output the image segmentation result through the image segmentation model.

[0018] Furthermore, the seed point is input into the image segmentation model, and outputting the image segmentation result through the image segmentation model includes the following steps:

[0019] S221, based on the image segmentation model, using the image encoder, converting the input seed point into an image embedding;

[0020] S222, generating a segmentation hint based on the texture features of the area to be detected, and processing the segmentation hint using a hint encoder;

[0021] S223, input the image embedded and processed segmentation hint into the mask decoder to obtain the image segmentation result.

[0022] Furthermore, the image analysis technology is used to classify and annotate the segmented images, and obtaining the image classification results includes the following steps:

[0023] S31, obtaining the type, form and texture features of the image;

[0024] S32, based on the type, morphology and texture features of the image, using image interpretation technology to analyze the image features in the image segmentation result to obtain a classification label in the image segmentation result;

[0025] S33. Based on the classification labels, the segmented images are classified and labeled to obtain image classification results.

[0026] Furthermore, the image classification result includes: segmentation result and segmentation type.

[0027] Furthermore, based on the image classification results, optimizing a preset image diffusion model, and fusing the image diffusion model with a conditional regulation network to generate a high-resolution image includes the following steps:

[0028] S41, extracting geometric shape features, category labels and mask information of the image from the image classification result;

[0029] S42, optimizing a pre-built image diffusion model using geometric shape features, category labels, and mask information of the image;

[0030] S43. Use the optimized image diffusion model to process the image classification results and combine them with the conditional control network to obtain a high-resolution image.

[0031] Furthermore, the image diffusion model includes: variational autoencoder, U-type network and CLIP text encoder.

[0032] Furthermore, optimizing the pre-built image diffusion model using the geometric shape features, category labels and mask information of the image includes the following steps:

[0033] S421, inputting the geometric shape features, category labels and mask information of the image into a pre-built image diffusion model, and using the variational autoencoder in the image diffusion model to encode the unmasked CT original image, the masked CT original image and the mask image to obtain hidden layer features;

[0034] S422, stacking the hidden layer features on the feature channel, and using the stacking result as the input of the U-type network;

[0035] S423. Use the CLIP text encoder to introduce the category label as a text prompt word into the image diffusion model, and adjust the image diffusion model in combination with the input of the U-type network.

[0036] Furthermore, the image classification results are processed using the optimized image diffusion model and combined with the conditional control network to obtain a high-resolution image, including the following steps:

[0037] S431, inputting the image classification result into the optimized image diffusion model to generate a repaired image;

[0038] S432, extracting geometric features of the inpainted image as spatial conditions, using the trained copy of the conditional control network, connecting the adjusted image diffusion model through a zero convolutional layer, and generating a high-resolution image.

[0039] According to another aspect of the present invention, there is also provided a HRCT image generation system based on a conditional controlled diffusion model, the system comprising a region extraction module, an image segmentation module, an image classification module and an image generation module;

[0040] A region extraction module is used to collect lesion images to construct a data set, and use an image processing algorithm to identify and extract the region to be detected from the data set;

[0041] An image segmentation module is used to determine a seed point by analyzing the texture characteristics of the area to be detected, and input the seed point into an image segmentation model to obtain an image segmentation result;

[0042] The image classification module is used to classify and label the segmented images using image analysis technology to obtain image classification results;

[0043] The image generation module is used to optimize a preset image diffusion model based on the image classification result, and fuse the image diffusion model with the conditional regulation network to generate a high-resolution image.

[0044] The beneficial effects of the present invention are:

[0045] 1. The present invention establishes a data set and adopts image processing technology to generate high-quality images while reducing the annotation cost, providing a more reliable data basis for subsequent image segmentation tasks and improving the detection efficiency of images.

[0046] 2. The present invention processes the image segmentation results through image interpretation technology, and can embed its synthesized realistic image into the image, thereby obtaining a large amount of enhanced data, which can greatly increase the amount of training data for the image segmentation model, not only improving the generalization ability of the image segmentation model, but also enhancing the segmentation performance of the image segmentation model.

[0047] 3. The present invention combines the image diffusion model with the conditional control network, which can not only further enhance the key features of the synthesized image, but also enrich the diversity of the synthesized image by controlling the spatial conditions, and further improve the generalization ability of the segmentation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 is a flow chart of a method for generating HRCT images based on a conditional controlled diffusion model according to an embodiment of the present invention;

[0050] Figure 2 is a flow chart of a method for generating HRCT images based on a conditional controlled diffusion model according to an embodiment of the present invention;

[0051] Figure 3 is a schematic diagram of generating seed points based on texture features in a HRCT image generation method based on a conditional controlled diffusion model according to an embodiment of the present invention;

[0052] Figure 4 is a schematic diagram of obtaining a segmentation result based on a seed point in a HRCT image generation method based on a conditional controlled diffusion model according to an embodiment of the present invention;

[0053] Figure 5 A flow chart of a segmentation model in a method for generating HRCT images based on a conditional controlled diffusion model according to an embodiment of the present invention;

[0054] Figure 6 It is a principle block diagram of a HRCT image generation system based on a conditional controlled diffusion model according to an embodiment of the present invention.

[0055] In the figure:

[0056] 1. Region extraction module; 2. Image segmentation module; 3. Image classification module; 4. Image generation module. DETAILED DESCRIPTION

[0057] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments. They can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. By referring to these contents, ordinary technicians in this field should be able to understand other possible implementation methods and advantages of the present invention.

[0058] According to an embodiment of the present invention, a method and system for generating HRCT images based on a conditional controlled diffusion model are provided.

[0059] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Figure 1-Figure 4As shown, according to the HRCT image generation method based on the conditional controlled diffusion model according to an embodiment of the present invention, the method comprises the following steps:

[0060] S1. Collect lesion images to construct a data set, and use image processing algorithms to identify and extract the area to be detected from the data set.

[0061] It should be noted that the image processing technology is a traditional image processing technology, such as threshold technology, etc., and deep learning methods can also be used to realize image processing.

[0062] S2. Determine the seed point by analyzing the texture characteristics of the area to be detected, and input the seed point into the image segmentation model to obtain the image segmentation result.

[0063] Specifically, the seed points are determined by analyzing the texture characteristics of the area to be detected, and the seed points are input into the image segmentation model. The image segmentation results are obtained by the following steps:

[0064] S21, extracting texture features of the area to be detected, and generating seed points based on the texture features of the area to be detected.

[0065] It should be noted that the generation of seed points based on the texture features of the area to be detected includes: calculating the contrast according to the grayscale co-occurrence matrix; dividing the area to be detected into multiple areas with lines and calculating the contrast of each area; screening out the five areas with the largest contrast; and randomly generating seed points in the five areas.

[0066] Among them, the elements in the gray-level co-occurrence matrix represent the joint distribution of the gray levels of two pixels with a certain spatial position relationship. Due to the large dimension of the gray-level co-occurrence matrix, it is generally not directly used as a feature to distinguish textures, but some statistics constructed based on it are used as texture classification features; the contrast reflects the clarity of the image and the depth of the texture grooves; the clearer the texture, the greater the contrast.

[0067] S22: input the seed point into the image segmentation model, and output the image segmentation result through the image segmentation model.

[0068] Specifically, inputting the seed point into the image segmentation model and outputting the image segmentation result through the image segmentation model includes the following steps:

[0069] S221, based on the image segmentation model, using the image encoder, converting the input seed point into an image embedding;

[0070] S222, generating a segmentation hint based on the texture features of the area to be detected, and processing the segmentation hint using a hint encoder;

[0071] S223, input the image embedded and processed segmentation hint into the mask decoder to obtain the image segmentation result.

[0072] It should be noted that the image segmentation model is the SAM (Segment Anything) model. SAM is a segmentation model proposed by Meta, which breaks through the traditional segmentation boundaries and greatly promotes the development of basic computer vision models. The model designs an image encoder and a hint encoder to process the input image and segmentation hints, as well as a mask decoder to process the output. In the specific model structure, the input of a picture is converted into a corresponding image embedding vector through the image encoder. If the input segmentation hint is a mask type, it will be added to the encoding through the convolutional layer and input into the mask decoder together with the image embedding. If there are other segmentation hints, they will be converted through the hint encoder and output to the mask decoder, and finally the mask generated by the model will be output through the mask decoder.

[0073] In addition, for the points generated based on texture features, the obtained mask does not require high accuracy and can only include part of the lesions. This is because the training set is constructed based on these masks and the diffusion model is used to learn the characteristics of the lesions.

[0074] S3. Use image analysis technology to classify and label the segmented images to obtain image classification results.

[0075] Specifically, using image analysis technology to classify and annotate the segmented images, obtaining image classification results includes the following steps:

[0076] S31, obtaining the type, form and texture features of the image;

[0077] S32, based on the type, morphology and texture features of the image, using image interpretation technology to analyze the image features in the image segmentation result to obtain a classification label in the image segmentation result;

[0078] S33. Based on the classification labels, the segmented images are classified and labeled to obtain image classification results.

[0079] It should be noted that the classification labels include grid shadow lesions and honeycomb shadow lesions.

[0080] Specifically, the image classification result includes: segmentation result and segmentation type.

[0081] It should be noted that the image interpretation technology is the GPT-4 model, which is a large multimodal model (accepting image and text input and issuing text output). It can understand and generate natural language in various contexts, learn from a large amount of text data, and has powerful image interpretation and analysis capabilities. It only needs to provide certain background knowledge, such as the type of lesion and the corresponding morphological and texture feature descriptions. GPT-4 can analyze the characteristics of the input image, select the corresponding lesion type, and realize the classification of lesions. Combined with the obtained image segmentation results, the corresponding lesion segmentation results and lesion types are automatically obtained for the original image, and automatic labeling is realized, so that a data set can be created.

[0082] S4. Based on the image classification results, optimize the preset image diffusion model, and fuse the image diffusion model with the conditional regulation network to generate a high-resolution image.

[0083] Specifically, based on the image classification results, optimizing a preset image diffusion model, and fusing the image diffusion model with a conditional regulation network to generate a high-resolution image includes the following steps:

[0084] S41. Extracting geometric shape features, category labels and mask information of the image from the image classification results.

[0085] S42. Optimize the pre-built image diffusion model using the geometric shape features, category labels and mask information of the image.

[0086] Specifically, the image diffusion model includes: variational autoencoder, U-type network and CLIP text encoder.

[0087] It should be noted that the image diffusion model includes a variational autoencoder, a U-net and a CLIP text encoder; the variational autoencoder (VAE) is used to transfer the diffusion process from the high-dimensional original pixel space to the low-dimensional latent space, thereby significantly reducing the amount of calculation; the U-net is trained to predict the noise added in each step of the diffusion process, so as to learn how to gradually remove the noise from the noisy image and restore a clear image; the CLIP text encoder (CLIP TextEncoder) is used to introduce text prompts to achieve text-guided generation.

[0088] Specifically, optimizing the pre-built image diffusion model using the geometric shape features, category labels, and mask information of the image includes the following steps:

[0089] S421, inputting the geometric shape features, category labels and mask information of the image into a pre-built image diffusion model, and using the variational autoencoder in the image diffusion model to encode the unmasked CT original image, the masked CT original image and the mask image to obtain hidden layer features.

[0090] It should be noted that the geometric shape features, category labels and lesion mask images of the CT image are input into a pre-built image diffusion model, and the lesion mask image is used to perform occlusion processing on the CT image, that is, the pixels in the mask area of ​​the CT image are set to zero to obtain the occluded CT image. The variational autoencoder in the image diffusion model is used to encode the CT image, the occluded CT image and the lesion mask image to obtain the hidden layer features.

[0091] S422, stacking the hidden layer features on the feature channel, and using the stacking result as the input of the U-type network;

[0092] S423. Use the CLIP text encoder to introduce the category label as a text prompt word into the image diffusion model, and adjust the image diffusion model in combination with the input of the U-type network.

[0093] It should be noted that the image diffusion model adopts a repair model based on stable diffusion. Its basic principle is to start from a known data distribution, gradually add noise, and then reverse the diffusion process to recover clean data from the noise.

[0094] S43. Use the optimized image diffusion model to process the image classification results and combine them with the conditional control network to obtain a high-resolution image.

[0095] Specifically, the image classification results are processed using the optimized image diffusion model and combined with the conditional control network to obtain a high-resolution image, including the following steps:

[0096] S431, inputting the image classification result into the optimized image diffusion model to generate a repaired image;

[0097] S432, extracting geometric features of the inpainted image as spatial conditions, using the trained copy of the conditional control network, connecting the adjusted image diffusion model through a zero convolutional layer, and generating a high-resolution image.

[0098] It should be noted that the conditional control network (ControlNet) is used to introduce additional spatial conditional control. It will lock the parameters of the image diffusion model to retain the powerful generation ability of the large model after pre-training on billions of images, and create trainable copies of its encoding blocks and intermediate blocks as a powerful backbone for learning control conditions; the conditional control network extracts the geometric shape features of interstitial lung disease lesions as spatial conditions input to the trainable copy, and its output is connected to the image diffusion model through a zero convolution layer to control the synthesis of lesion texture.

[0099] In addition, the segmentation model uses real data and generated enhanced data as training sets to learn the segmentation of lesions; for example, DeepLabv3 is used to train the real data set and the data set generated by the diffusion model, and then tested on the test set (the evaluation indicator is the Dice coefficient); after adding the generated data set, the effect (dice) on the test set is tested to see if it is improved compared to when the generated data set is not added, to verify the effectiveness of CT image generation. Figure 5 As shown in the figure, if the generated CT images are added to the training set, the final results (such as dice) on the same batch of test sets will increase, indicating that the method is effective.

[0100] like Figure 6 As shown, according to another embodiment of the present invention, a HRCT image generation system based on a conditional controlled diffusion model is also provided, the system comprising a region extraction module 1, an image segmentation module 2, an image classification module 3 and an image generation module 4;

[0101] The region extraction module 1 is used to collect lesion images to construct a data set, and use an image processing algorithm to identify and extract the region to be detected from the data set;

[0102] The image segmentation module 2 is used to determine the seed point by analyzing the texture characteristics of the area to be detected, and input the seed point into the image segmentation model to obtain the image segmentation result;

[0103] The image classification module 3 is used to classify and label the segmented images using image analysis technology to obtain image classification results;

[0104] The image generation module 4 is used to optimize a preset image diffusion model based on the image classification result, and fuse the image diffusion model with the conditional regulation network to generate a high-resolution image.

[0105] In summary, by means of the above technical solution of the present invention, by establishing a data set and using image processing technology, high-quality images are generated while reducing the annotation cost, providing a more reliable data basis for subsequent image segmentation tasks and improving the detection efficiency of images. By processing the image segmentation results through image interpretation technology, it is possible to embed its synthesized realistic image on the image, thereby obtaining a large amount of enhanced data, which can greatly increase the amount of training data for the image segmentation model, not only improving the generalization ability of the image segmentation model, but also enhancing the segmentation performance of the image segmentation model. By combining the image diffusion model with the conditional control network, not only the key features of the synthesized image can be further enhanced, but also the diversity of the synthesized image can be enriched by controlling the spatial conditions, further improving the generalization ability of the segmentation model.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating HRCT images based on a conditional controlled diffusion model, characterized in that: The method comprises the following steps: S1, collecting lesion images to construct a data set, and using an image processing algorithm to identify and extract the area to be detected from the data set; S2. Determine a seed point by analyzing the texture characteristics of the area to be detected, and input the seed point into the image segmentation model to obtain an image segmentation result; S3. Use image analysis technology to classify and label the segmented images to obtain image classification results; S4. Based on the image classification results, optimize the preset image diffusion model, and fuse the image diffusion model with the conditional regulation network to generate a high-resolution image.

2. The HRCT image generation method based on the conditional controlled diffusion model according to claim 1, characterized in that: The step of determining the seed point by analyzing the texture characteristics of the area to be detected and inputting the seed point into the image segmentation model to obtain the image segmentation result comprises the following steps: S21, extracting texture features of the area to be detected, and generating seed points based on the texture features of the area to be detected; S22: input the seed point into the image segmentation model, and output the image segmentation result through the image segmentation model.

3. The HRCT image generation method based on the conditional controlled diffusion model according to claim 2, characterized in that: The step of inputting the seed point into the image segmentation model and outputting the image segmentation result through the image segmentation model comprises the following steps: S221, based on the image segmentation model, using the image encoder, converting the input seed point into an image embedding; S222, generating a segmentation hint based on the texture features of the area to be detected, and processing the segmentation hint using a hint encoder; S223, input the image embedded and processed segmentation hint into the mask decoder to obtain the image segmentation result.

4. The HRCT image generation method based on the conditional controlled diffusion model according to claim 1, characterized in that: The image analysis technology is used to classify and label the segmented image to obtain the image classification result, which includes the following steps: S31, obtaining the type, form and texture features of the image; S32, based on the type, morphology and texture features of the image, using image interpretation technology to analyze the image features in the image segmentation result to obtain a classification label in the image segmentation result; S33. Based on the classification labels, the segmented images are classified and labeled to obtain image classification results.

5. The HRCT image generation method based on the conditional controlled diffusion model according to claim 4, characterized in that: The image classification result includes: a segmentation result and a segmentation type.

6. The HRCT image generation method based on the conditional controlled diffusion model according to claim 1, characterized in that: The method of optimizing a preset image diffusion model based on the image classification result and fusing the image diffusion model with a conditional regulation network to generate a high-resolution image comprises the following steps: S41, extracting geometric shape features, category labels and mask information of the image from the image classification result; S42, optimizing a pre-built image diffusion model using geometric shape features, category labels, and mask information of the image; S43. Use the optimized image diffusion model to process the image classification results and combine them with the conditional control network to obtain a high-resolution image.

7. The HRCT image generation method based on the conditional controlled diffusion model according to claim 6, characterized in that: The image diffusion model includes: a variational autoencoder, a U-type network and a CLIP text encoder.

8. The HRCT image generation method based on the conditional controlled diffusion model according to claim 7, characterized in that: The method of optimizing the pre-built image diffusion model by using the geometric shape features, category labels and mask information of the image comprises the following steps: S421, inputting the geometric shape features, category labels and mask information of the image into a pre-built image diffusion model, and using the variational autoencoder in the image diffusion model to encode the unmasked CT original image, the masked CT original image and the mask image to obtain hidden layer features; S422, stacking the hidden layer features on the feature channel, and using the stacking result as the input of the U-type network; S423. Use the CLIP text encoder to introduce the category label as a text prompt word into the image diffusion model, and adjust the image diffusion model in combination with the input of the U-type network.

9. The HRCT image generation method based on the conditional controlled diffusion model according to claim 6, characterized in that: The method of processing the image classification results by using the optimized image diffusion model and combining the conditional control network to obtain a high-resolution image includes the following steps: S431, inputting the image classification result into the optimized image diffusion model to generate a repaired image; S432, extracting geometric features of the inpainted image as spatial conditions, using the trained copy of the conditional control network, connecting the adjusted image diffusion model through a zero convolutional layer, and generating a high-resolution image.

10. A HRCT image generation system based on a conditional controlled diffusion model, used to implement the HRCT image generation method based on a conditional controlled diffusion model according to any one of claims 1 to 9, characterized in that: The system includes a region extraction module, an image segmentation module, an image classification module and an image generation module; The region extraction module is used to collect lesion images to construct a data set, and use an image processing algorithm to identify and extract the region to be detected from the data set; The image segmentation module is used to determine the seed point by analyzing the texture characteristics of the area to be detected, and input the seed point into the image segmentation model to obtain the image segmentation result; The image classification module is used to classify and label the segmented images using image analysis technology to obtain image classification results; The image generation module is used to optimize a preset image diffusion model based on the image classification result, and fuse the image diffusion model with the conditional regulation network to generate a high-resolution image.

Images