Text generation chest X-ray image system and method based on diffusion model

Abstract

The invention relates to a system and a method for generating a chest X-ray image based on a text of a diffusion model, and the system comprises a hidden space noise filtering module, a self-adaptive visual perception text learning module and a medical knowledge injection module for the diffusion model, performing hidden space mapping, manifold modeling and noise discrimination processing through a pre-trained image coding unit to obtain a denoised CXR image, and then obtaining a corresponding text according to a corresponding relationship between the CXR image and the text; the adaptive visual perception text learning module uses the noise filtered CXR image and text to train a large visual language model; a medical knowledge injection module aiming at the diffusion model is used for gradually injecting template text coding information generated according to the trained large visual language model into the diffusion model in a training state through a trainable control adapter, so that the perception intensity of the diffusion model on the text information in the training process is stably improved; therefore, the CXR image generation quality of the model is improved. According to the method, through an innovative method combining three stages of hidden space noise filtering, adaptive visual perception text learning and a medical knowledge injection module, the model is endowed with the capability of fully understanding characters and CXR images and generating high-quality CXR images of a specified type.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image generation, in particular to a system and method for generating chest X-ray images from text based on a diffusion model. Background Art

[0002] Due to the many limitations of medical imaging data in terms of collection, processing, and privacy protection, it is currently extremely challenging to build a large-scale, well-structured, and high-quality medical imaging dataset. Therefore, medical image generation models are expected to become a key method to solve this problem. Text-to-image (TTI) generation is of great significance for controllable and diverse image generation. With the characteristics of fast acquisition, non-invasiveness, and low radiation exposure, chest X-ray (CXR) images have become one of the most popular medical imaging modalities. At present, some studies have made progress in medical TTI tasks, but their generation performance may be limited by problems such as the high complexity of the intrinsic characteristics of medical data. Summary of the invention

[0003] In view of the problems in existing image generation methods in the medical field, such as the serious noise interference of the model and the difficulty in understanding the medical text features in the report, which leads to the poor authenticity and accuracy of the image generation, the present invention proposes a text generation system and method for chest X-ray images based on a diffusion model. By combining a three-stage innovative method of latent space noise filtering, adaptive visual perception text learning and medical knowledge injection module, the model is given the ability to fully understand text and CXR images and generate high-quality CXR images of specified types.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a text generation chest X-ray image system based on a diffusion model, comprising: a latent space noise filtering module, an adaptive visual perception text learning module and a medical knowledge injection module for the diffusion model, wherein: the latent space noise filtering module performs latent space mapping, manifold modeling and noise discrimination processing on an original CXR image through a pre-trained image encoding unit to obtain a CXR image and corresponding text after noise data is removed; the adaptive visual perception text learning module uses the CXR image and text after noise filtering to train a large visual language model; the medical knowledge injection module for the diffusion model gradually injects templated text encoding information generated by the trained large visual language model into the diffusion model in a training state through a trainable control adapter, so as to steadily improve the diffusion model's perception intensity of text information during the training process, so as to improve the model's CXR image generation quality.

[0006] The latent space noise filtering module includes: a pre-trained autoencoder image encoding unit, a latent space clustering unit, a manifold modeling unit and a noise discrimination unit, wherein: the pre-trained autoencoder image encoding unit maps the original CXR image to the latent space by encoding, and obtains the latent space representation of the original CXR image; the latent space clustering unit performs K-means clustering processing on the latent space representation to obtain the distribution of the latent space representation of the image in the latent space and multiple centers of distribution, and obtains the latent space representation of typical normal data and other data after filtering; the manifold modeling unit learns the latent space representation of typical normal data, constructs a learning and reconstruction model of the latent space features of typical normal data, reconstructs the typical normal data, and performs coarse-grained screening on the reconstructed data according to the size of the reconstruction error, removes the typical noise data from the data set, and obtains the remaining CXR image data except for the typical normal and typical noise data. The noise discrimination unit is trained based on typical noise data and typical normal data, and combines the reconstruction error calculated by the popular modeling unit to perform explicit fine-grained noise discrimination on the latent space feature representation of the remaining CXR images except for the typical normal and typical noise data, thereby completing the noise data filtering in the latent space.

[0007] The pre-trained autoencoder image encoding unit is implemented using but not limited to the technology recorded in Chambon, P., Blüthgen, C., Delbrouck, J., Sluijs, RV, Polacin, M., Chaves, JM, Abraham, T., Purohit, S., Langlotz, CP, & Chaudhari, A. RoentGen: Vision-Language Foundation Model for Chest X-ray Generation [J]. ArXiv, 2022. https: / / arxiv.org / abs / 2211.1273.

[0008] The latent space representation of typical normal data and other data refers to comparing the distance of each representation data from the nearest sample center with a preset threshold, and the representation data smaller than the threshold and its corresponding original CXR image are regarded as typical normal data.

[0009] The coarse-grained screening means that during the training process, the manifold modeling unit learns the characteristic representation of typical normal data, thereby reconstructing the typical normal data normally and obtaining a lower reconstruction error. For typical noise data, the manifold modeling unit has not learned its latent space characteristic representation, and a larger reconstruction error will be obtained during reconstruction. This reconstruction error is compared with a set value, and the original CXR image corresponding to the latent space representation with a reconstruction error greater than the set value is defined as typical noise data and removed from the original CXR image data set, thereby obtaining the remaining CXR image data excluding the typical normal and typical noise data.

[0010] The explicit fine-grained noise discrimination refers to: learning typical normal and noise data latent space features through a latent space noise discriminator and judging the category of the remaining data accordingly, and at the same time weighting the reconstruction error obtained in the manifold modeling stage of the remaining data to obtain the final discrimination score to judge whether the remaining data is noise data or normal data.

[0011] The adaptive visual perception text learning module includes: a text feature compression unit, a text-image alignment unit and a pre-trained large-scale visual language model in the field of medical imaging, wherein: the pre-trained large-scale visual language model in the field of medical imaging encodes text information of a text report corresponding to a filtered normal image through a text feature extraction unit, and after obtaining the encoding information corresponding to the text, encodes image information of the filtered normal image through an image feature extraction unit to obtain image encoding information; the text feature compression unit performs dimensionality reduction and compression processing on the text encoding information to obtain compressed image encoding information; the text-image alignment unit performs text-image contrast learning based on the image encoding information and the compressed text encoding information, and uses contrast loss to train the text feature extraction unit and the text feature compression unit of the large-scale visual language model in the field of medical imaging.

[0012] The medical knowledge injection module includes: a text templating unit, a trained text feature extraction unit and a text feature compression unit, a pre-trained autoencoder image encoding unit, a control adapter, a diffusion model learning unit enhanced by medical knowledge, and an autoencoder image decoding unit, wherein: the text templating unit processes the text reports corresponding to all normal images after latent space noise filtering to obtain a templated text report, the text feature extraction unit and the text feature compression unit trained by contrastive learning perform text information extraction and compression based on the templated text report to obtain accurate text encoding compression information, the pre-trained autoencoder image encoding unit performs image encoding based on the normal image filtered by latent space noise to obtain image encoding information, and the control adapter injects low-dimensional text encoding information into the diffusion model learning unit at a learnable ratio to gradually improve the diffusion model's perception intensity of the text. During the training phase, the medical knowledge-enhanced diffusion model learning unit first performs noise processing on the image coding information, and then gradually denoises the image coding in combination with the compressed text coding information, and finally obtains the noise-free original CXR image coding. The diffusion model learning unit learns the data distribution of the image coding in this denoising-denoising process. During the inference phase, the image coding information that conforms to the text report description is generated by receiving random noise and compressed text coding information. The autoencoder image decoding unit performs decoding operations based on the image coding information output by the diffusion model learning unit in the inference phase, and obtains a CXR image corresponding to the text report description.

[0013] The present invention relates to a method for generating CXR images based on text based on the above system. The method comprises: a latent space noise filtering module is used to gradually learn the general pattern of abnormal noise data from coarse granularity to fine granularity in a high-dimensional latent space and remove it, and after obtaining a denoised CXR image data set, an adaptive visual perception text learning module is used to use a large visual language model in the field of medical imaging to extract key information from a text report as guidance for providing CXR images; and a medical knowledge injection module is used to use a control adapter to gradually inject general medical knowledge into a text-image generation model in a fine-grained manner, and a complete diffusion generation model based on medical knowledge enhancement is constructed to realize the generation of CXR images containing specified types using text reports. Technical Effects

[0014] The present invention effectively removes noise data in the latent space from the CXR image dataset, especially near the decision boundary, and endows a large-scale visual language model (VLM) in a specific field with the ability to explicitly model the contextual relationship between different tags and dynamically learn concise and important report embedding information. Finally, the text information is injected into the diffusion process in a controllable manner in a fine-grained manner by controlling the adapter, further strengthening the importance of medical representation in text embedding and stabilizing and refining the diffusion process.

[0015] Under the premise of low computing resource consumption, the present invention can generate chest X-ray images with high clarity, fidelity and diversity. The test scores of the generation model evaluation indicators Fréchet Inception Distance (FID) and mean Area Under Curve (mAUC) on two commonly used medical CXR image datasets are better than the current best CXR image generation method, and has strong generation and generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of the training phase of the present invention;

[0017] Figure 2 Generate a phase flow chart for the present invention;

[0018] Figure 3 This is the flowchart of latent space noise filtering;

[0019] Figure 4 Flowchart for adaptive visually perceptual text learning;

[0020] Figure 5 To control the workflow diagram of the adapter and the flowchart of the medical knowledge injection module;

[0021] Figure 6 This is a schematic diagram of the latent space noise filtering module;

[0022] Figure 7 Schematic diagram of adaptive visual perception text learning module;

[0023] Figure 8 Schematic diagram of the module for injecting medical knowledge;

[0024] Fig. 9 To generate an image example;

[0025] Fig.10 Visual comparison of the generation effects of different generation methods on the same dataset. DETAILED DESCRIPTION

[0026] This embodiment relates to a method for generating CXR images based on text based on a diffusion model enhanced by medical knowledge, including: Figure 1 The training phase shown and Figure 2 The build phases are shown.

[0027] The training phase includes:

[0028] Step 1: Figure 6 As shown, latent space noise filtering specifically includes:

[0029] 1.1) It will contain N pairs of images and corresponding report pairs All images in the dataset The pre-trained autoencoder ε encoding is mapped to the latent space, that is, the encoding is

[0030] The pre-trained autoencoder refers to an open source image encoder that has been pre-trained for encoding-decoding on a large-scale dataset, and uses its efficient and accurate CXR image feature extraction and reconstruction functions to obtain an accurate representation of the CXR image in the latent space.

[0031] 1.2) Use the dimension concatenation method in the latent space to convert the encoding obtained in step 1.1 into Reconstructed into a one-dimensional vector

[0032] 1.3) Apply K-means clustering to cluster the one-dimensional vector obtained in step 1.2 Divide the data into different clusters, and use the distance between the sample data and the center of the sample cluster as the threshold to filter out typical normal data.

[0033] 1.4) Use the manifold modeling method based on the U-shaped autoencoder to capture non-abnormal data with clear imaging view, complete regional presentation, and no foreign body occlusion from the typical normal data filtered out in step 1.3 The total distribution in the latent space.

[0034] The U-shaped automatic encoder comprises: an encoding network and a decoding network, wherein: the encoding network performs encoding processing according to the latent space representation of the input typical normal data to obtain the latent space encoding result of the typical normal data, and the decoding network performs decoding processing according to the latent space encoding result of the typical normal data to obtain the reconstruction result of the latent space representation of the typical normal data.

[0035] The U-shaped autoencoder is trained in the following way:

[0036] Step 1.4.1: By encoding and decoding the latent space representation of typical normal data, the reconstruction result of the latent space representation of typical normal data is obtained.

[0037] Step 1.4.2: Calculate the mean square error between this result and the latent space representation of the input typical normal data to obtain the mean square error loss, i.e., the reconstruction error.

[0038] Step 1.4.3: Use this loss value to perform backpropagation to adjust the network parameters and iterate repeatedly until the number of training rounds reaches the set number and then stop training.

[0039] like Figure 6As shown, the latent space noise discriminator includes: a series of cascaded convolutional layers, normalization layers, activation layers for hierarchical feature extraction, adaptive average pooling layers for binary classification, and linear classifiers, wherein: the cascaded convolutional layers perform convolution processing according to the latent space image encoding information to obtain further feature representation; the normalization layer performs numerical normalization processing according to the previous stage feature representation to obtain a feature representation result with a total of unit value; the activation layer performs threshold activation according to the previous stage output to obtain information that plays a decisive role in noise discrimination; the adaptive average pooling layer performs numerical regional average processing on the previous stage information to reduce the computational complexity and obtains the representation after regional average processing; the linear classifier performs classification processing according to the previous stage output information to obtain the final classification result.

[0040] The noise discriminator is trained under the supervision of binary cross entropy loss, and determines whether the input is a noise pattern based on its output score. Finally, all noise images are detected by combining reconstruction error and discrimination score, and the filtered data set is obtained based on the detection results. Where: n represents the number of entries in the cleaned dataset.

[0041] Step 2: Perform adaptive visual perception text learning on the image after noise filtering in step 1, specifically including:

[0042] 2.1) The text feature extraction unit and image feature extraction unit of a pre-trained large-scale visual language model in the field of medical imaging are used to map the text and the noise-filtered image and the two modalities into a specific latent space to achieve feature alignment in the low-dimensional latent space.

[0043] The mapping refers to: using the text feature extraction unit and image feature extraction unit of the large visual language model in the field of medical imaging to extract features from the input text report and image, and obtain the corresponding one-dimensional image feature vector G ic and text feature vector G sxc For the text feature vector, the text feature compression unit is used to compress it to obtain the same ic The text feature compression vector G of the same length xc Then, the category label [CLS] of the image and text is directly concatenated into the feature vector as the first element to obtain the image and text feature vector G of the same length. xi ,G yi .

[0044] The text feature extraction unit and the image feature extraction unit of the large-scale visual language model in the field of medical imaging are implemented by the technology described in, but not limited to, Zhang, S., Xu, Y., Usuyama, N., Bagga, JK, Tinn, R., Preston, S., Rao, RN, Wei, M., Valluri, N., Wong, C., Lungren, MP, Naumann, T., & Poon, H. BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs [J]. ArXiv, 2023. https: / / doi.org / 10.48550 / arXiv.2303.00915.

[0045] The feature alignment is to guide the training process of the image and text feature extraction unit by maximizing the similarity between the image and text feature vectors of T positive sample pairs in the batch and minimizing the similarity of the negative sample pairs in the batch through the comparison target, so that the image and text feature extraction unit can weaken or ignore the parts of the report that are not related to the image during the feature extraction process, and extract key corresponding features from the medical report and the image. The overall average alignment loss function is Among them: The alignment loss function of a single image-text pair is (G xi ,G yi ) is the latent space feature vector corresponding to the i-th CXR and its corresponding report, and the [CLS] category label is used as the global category representation.

[0046] 2.2) In order to accurately capture the context related to the image and reduce the impact of text redundant information on text feature extraction, this method introduces a text feature compression unit at the back end of the text feature extraction unit. The compressed text information is compared with the output of the image encoder, so that the text information compression unit has the ability to compress the potential embedding information of the text in the latent space to a low dimension. In the generation stage, the latent space text embedding information output by the text feature compression unit will provide corresponding text guidance for the downstream generation process, such as Figure 2 shown.

[0047] Step 3: Inject domain-specific knowledge into the diffusion process in a controllable manner, highlight the key parts of the report information, and gradually refine the generation process, including:

[0048] 3.1) Latent space diffusion model training process. The diffusion process of the latent space diffusion model occurs in a more efficient low-dimensional latent space ( Figure 8 Middle Z t The model learns the feature representation of the original data from the process of noise injection and restoration of the original data. The noise injection process is called the forward process, which simulates a fixed Markov chain of length T, specifically: and Where: z0 is the latent space representation of normal CXR image z1~z T represents the continuously noisy image representation from step 1 to step T in the latent space. In the forward process, the variance β is gradually added to the latent space representation z0 of the image. t ∈(0,1) noise that varies with time steps, where: t∈1,...T. The training goal is to add a noisy version of z to time t≤T. t Noise prediction and removal are performed to reverse the noise addition process to obtain z t-1 The corresponding training loss function is expressed as: Among them: ,∈ θ (z t ,t,c) represents the prediction of the real noise ∈ by a series of noise estimation models with parameters θ. The noise estimation models of all time steps share the same model weights, and c is the text report information. When the number of training rounds reaches the set value, the training is stopped.

[0049] 3.2) Report text template: Use the CheXbert model to process medical reports and generate category annotations, and convert the results into template text descriptions. Specifically, the text report is input into the CheXbert model, and the model automatically captures the key information of the text and outputs the key medical information contained in the report, and converts it into a template text description that "contains 'type'". Then introduce the text feature extraction unit and compression unit obtained in step 2, receive the templated text description, extract features and compress it in the latent space, and obtain the latent space representation of the templated text.

[0050] The CheXbert model adopts but is not limited to the technical implementation recorded in Smit, Akshay et al. "CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT." Conference on Empirical Methods in Natural Language Processing (2020).

[0051] 3.3) During the training of the diffusion model, the trainable control adapter introduces templated text information into the pre-trained image generation model by gradually injecting medical knowledge into the network, further refining and highlighting the guiding role of key text information in the diffusion process. The specific implementation process is as follows: At the beginning of the training of the diffusion model learning unit, the noise estimation model in the unit is initialized using the pre-trained weights.

[0052] The structure of each noise estimation model is as follows: Figure 8 As shown, it consists of several encoder blocks, intermediate blocks and decoder blocks, denoted as E, M, D respectively. The encoder blocks and intermediate blocks are cloned into the trainable control adapter, designated as E′ and M′, and the decoder block D' is initialized with a zero linear layer. e(e′), m(m′) and d represent the outputs of a certain encoder block, intermediate block or decoder block of the trainable control adapter. Thereafter, during the decoding process of the diffusion model learning unit, the control adapter takes the features of the templated text as input, encodes it with its own encoder block E′ and intermediate block M′, and injects the output into the diffusion model learning unit. To ensure that when training starts, the input of the i-th block of the decoding block D is the same as the original denoising model so as to maintain the normal generation ability of the pre-trained model, the decoder block D' of the learnable control adapter is initialized with a zero linear layer. The input of the i-th decoder block of the diffusion model learning unit is: Where: zero represents a zero linear layer. When training starts, the parameters of the trainable control adapter are initialized to zero, which has no effect on the input of the decoding block D of the diffusion model learning unit. In the subsequent training process, the learnable control adapter gradually processes the feature information of the templated text and injects it into the diffusion model learning unit, and merges it with the output information of the decoder block D of the diffusion model learning unit, gradually strengthening the integration of medical knowledge in the diffusion process and enhancing the diffusion model's ability to understand text information.

[0053] like Figure 2As shown, the generation stage includes: inputting random noise and templated text reports of the same size as the expected image into the trained medical knowledge injection module to generate a CXR image described by text, such as Fig. 9 shown.

[0054] Through specific experiments, the Diff-CXR model of the present invention was trained using the MIMIC-CXR dataset, and the generation performance was verified on the MIMIC-CXR test set. The IU-Xray dataset was then used to further evaluate the performance of Diff-CXR for unseen texts.

[0055] In order to verify the effects of each method and measure the quality of generated data, the present invention uses Fréchet Inception Distance distance (FID) and mAUC scores for evaluation in ablation experiments. FID quantifies the distribution difference between generated data and real data. The lower the FID score, the better the visual fidelity. The mAUC score is used to evaluate the visual language alignment between the generated image and the input report. It is obtained by calculating the mAUC score of the generated image predicted by the pre-trained chest classification network densnet121-resnet224-all relative to the original label of MIMIC-CXR. The higher the mAUC score, the better the visual language alignment between the input report and the generated CXR. The experimental results are shown in Table 1. It can be seen that the overall FID score of the generated model after using the noise filtering method dropped from 28.84 to 20.93, and the mAUC score increased from 0.644 to 0.670, proving the significant impact of the noise image on the generation process. At the same time, the addition of the medical knowledge injection method reduced the FID value from 21.56 to 19.50 and increased the mAUC score from 0.676 to 0.678, indicating that this method successfully improved the authenticity and accuracy of the generated CXR.

[0056] In the horizontal comparison experiment, the present invention is compared with the best models of text generation images in the current CXR field, such as UnixGen, LLM-CXR and RoentGen, and the results are shown in Tables 2 and 3. In the above experiments, the present invention uses floating point operations (FLOPs) to evaluate the computational complexity of different methods. As shown in Table 3, the images generated by the present invention are always higher than LLM-CXR in terms of AUC scores of almost all 11 common chests, indicating that the generated images have the best alignment effect with the input report. As shown in Table 3, the present invention is also better than LLM-CXR in FID scores, with increases of 9.770 and 9.829 respectively, proving the advantages of the present invention in visual fidelity. In addition, the present invention also shows excellent generalization ability for unseen texts with different distributions. Compared with the performance of LLM-CXR on the IU-Xray dataset, the present invention shows better performance, with mAUC and FID scores increased by 56.4% and 23.8% respectively. As shown in Table 2, Diff-CXR also achieves the lowest computational complexity of 29.641 GFLOPs, which is almost one-third of RoentGen and one-ninth of LLM-CXR, successfully proving the effectiveness of the method. In addition, due to the optimization of computational complexity, the reasoning speed of the present invention is also significantly improved, which is 7.6 times and 4.0 times faster than LLM-CXR and RoentGen. At the same time, the actual generated images of the present invention and the current best models are visualized as follows Fig.10 As shown, the image generated by the present invention also has imaging features that are more similar to the standard image.

[0057] Table 1:

[0058] Table 2:

[0059] Table 3:

[0060] In summary, in order to overcome the challenges faced by the current TTI generation method based on the diffusion model in medical scenarios, especially the problem of overfitting noise data, the present invention mainly revolves around three directions: first, the latent space noise filtering method, by mapping the data into the latent space to identify the noise and data samples, reducing the negative impact of noise samples on the generation performance; second, improving the generation efficiency and accuracy, using an adaptive visual perception text learning method to extract key report information, reduce information irrelevant to the image, thereby improving the generation efficiency; finally, the quality of the generated image is improved by the medical knowledge injection method, by gradually integrating medical knowledge into the model, achieving fine control of the diffusion process, and further improving the quality of the generated image. The implementation of these methods not only significantly improves the FID and mAUC scores of the generated images, confirming the effectiveness of the present invention in improving the authenticity and accuracy of the generated results, but also transforms the medical TTI generation into a powerful data enhancement tool that can be applied to a variety of downstream tasks, thereby significantly improving the performance of the model with only a small amount of real data, showing the important role and broad application prospects of medical TTI generation in computer-aided diagnosis.

[0061] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A text-generated chest X-ray image system based on a diffusion model, characterized in that: include: Latent space noise filtering module, adaptive visual perception text learning module and medical knowledge injection module for diffusion model, wherein: the latent space noise filtering module performs latent space mapping, manifold modeling and noise discrimination processing on the original CXR image through the pre-trained image coding unit to obtain the CXR image and corresponding text after removing the noise data; The adaptive visual perception text learning module uses noise-filtered CXR images and texts to train a large visual language model. The medical knowledge injection module for the diffusion model gradually injects the templated text encoding information generated by the trained large visual language model into the diffusion model in the training state through a trainable control adapter, thereby robustly improving the diffusion model's perception of text information during the training process, thereby improving the model's CXR image generation quality.

2. The text-generated chest X-ray image system based on diffusion model according to claim 1, characterized in that: The latent space noise filtering module comprises: a pre-trained autoencoder image encoding unit, a latent space clustering unit, a manifold modeling unit and a noise discrimination unit, wherein: the pre-trained autoencoder image encoding unit maps the original CXR image to the latent space by encoding, and obtains the latent space representation of the original CXR image; the latent space clustering unit performs K-means clustering on the latent space representation, obtains the distribution of the latent space representation of the image in the latent space and multiple centers of the distribution, and obtains the latent space representation of typical normal data and other data through filtering; the manifold modeling unit learns the latent space representation of typical normal data After constructing a learning and reconstruction model of latent space features of typical normal data, the typical normal data is reconstructed, and the reconstructed data is coarsely screened according to the size of the reconstruction error, and the typical noise data is removed from the data set to obtain the remaining CXR image data except for the typical normal and typical noise data; the noise discrimination unit is trained based on the typical noise data and the typical normal data, and the reconstruction error calculated by the popular modeling unit is combined to perform explicit fine-grained noise discrimination on the latent space feature representation of the remaining CXR images except for the typical normal and typical noise data, and complete the noise data filtering in the latent space; The latent space representation of typical normal data and other data refers to comparing the distance of each representation data from the nearest sample center with a preset threshold, and the representation data smaller than the threshold and its corresponding original CXR image are regarded as typical normal data.

3. The text-generated chest X-ray image system based on diffusion model according to claim 2, characterized in that: The coarse-grained screening means that during the training process, the manifold modeling unit learns the characteristic representation of typical normal data, thereby reconstructing the typical normal data normally and achieving a lower reconstruction error; For typical noise data, the manifold modeling unit has not learned its latent space feature representation, and a large reconstruction error will be obtained during reconstruction. This reconstruction error is compared with the set value. If the reconstruction error is greater than the set value, the original CXR image corresponding to the latent space representation is defined as typical noise data and removed from the original CXR image dataset to obtain the remaining CXR image data excluding the typical normal and typical noise data.

4. The text-generated chest X-ray image system based on diffusion model according to claim 1, characterized in that: The explicit fine-grained noise discrimination refers to: learning typical normal and noise data latent space features through a latent space noise discriminator and judging the category of the remaining data accordingly, and at the same time weighting the reconstruction error obtained in the manifold modeling stage of the remaining data to obtain the final discrimination score to judge whether the remaining data is noise data or normal data.

5. The text-generated chest X-ray image system based on diffusion model according to claim 1, characterized in that: The adaptive visual perception text learning module includes: a text feature compression unit, a text-image alignment unit and a pre-trained large-scale visual language model in the field of medical imaging, wherein: the pre-trained large-scale visual language model in the field of medical imaging encodes text information of a text report corresponding to a filtered normal image through a text feature extraction unit, and after obtaining the encoding information corresponding to the text, encodes image information of the filtered normal image through an image feature extraction unit to obtain image encoding information; the text feature compression unit performs dimensionality reduction and compression processing on the text encoding information to obtain compressed image encoding information; the text-image alignment unit performs text-image contrast learning based on the image encoding information and the compressed text encoding information, and uses contrast loss to train the text feature extraction unit and the text feature compression unit of the large-scale visual language model in the field of medical imaging.

6. The text-generated chest X-ray image system based on diffusion model according to claim 1, characterized in that: The medical knowledge injection module includes: a text template unit, a trained text feature extraction unit and a text feature compression unit, a pre-trained autoencoder image encoding unit, a control adapter, a diffusion model learning unit enhanced by medical knowledge, and an autoencoder image decoding unit, wherein: the text template unit processes the text reports corresponding to all normal images after latent space noise filtering to obtain a templated text report, the text feature extraction unit and the text feature compression unit trained by contrastive learning perform text information extraction and compression according to the templated text report to obtain accurate text encoding compression information, the pre-trained autoencoder image encoding unit performs image encoding according to the normal image filtered by latent space noise to obtain image encoding information, and the control adapter injects low-dimensional text encoding information into the diffusion model learning unit at a learnable ratio to gradually improve the diffusion model's perception strength of the text; During the training phase, the medical knowledge-enhanced diffusion model learning unit first performs noise processing on the image coding information, and then gradually denoises the image coding in combination with the compressed text coding information, and finally obtains the noise-free original CXR image coding. The diffusion model learning unit learns the data distribution of the image coding in this denoising-denoising process. During the inference phase, the image coding information that conforms to the text report description is generated by receiving random noise and compressed text coding information. The autoencoder image decoding unit performs decoding operations based on the image coding information output by the diffusion model learning unit in the inference phase, and obtains a CXR image corresponding to the text report description.

7. A method for generating CXR images based on text based on the system according to any one of claims 1 to 6, characterized in that: The latent space noise filtering module gradually learns the general pattern of abnormal noise data from coarse granularity to fine granularity in the high-dimensional latent space and removes it. After obtaining the denoised CXR image dataset, the adaptive visual perception text learning module uses a large visual language model in the field of medical imaging to extract key information from the text report as a guide for CXR images. Then, through the medical knowledge injection module, the control adapter is used to gradually inject general medical knowledge into the text-image generation model in a fine-grained manner, and a complete diffusion generation model based on medical knowledge enhancement is constructed to realize the generation of CXR images of specified types using text reports.

8. The method for generating CXR images based on text according to claim 7, characterized in that: The latent space noise filtering specifically includes: 1.1) It will contain N pairs of images and corresponding report pairs All images in the dataset The pre-trained autoencoder ε encoding is mapped to the latent space, that is, the encoding is 1.2) Use the dimension concatenation method in the latent space to convert the encoding obtained in step 1.1 into Reconstructed into a one-dimensional vector 1.3) Apply K-means clustering to cluster the one-dimensional vector obtained in step 1.2 Divide into different clusters, and use the distance between the sample data and the center of the sample cluster as a threshold to filter out typical normal data; 1.4) Use the manifold modeling method based on the U-shaped autoencoder to capture non-abnormal data with clear imaging view, complete regional presentation, and no foreign body occlusion from the typical normal data filtered out in step 1.3 The total distribution in the latent space.

9. The method for generating CXR images based on text according to claim 7, characterized in that: The adaptive visual perception text learning specifically includes: 2.1) Using the text feature extraction unit and image feature extraction unit of a large visual language model pre-trained in the field of medical imaging, the text and the noise-filtered image and the two modalities are mapped to a specific latent space to achieve feature alignment in the low-dimensional latent space; 2.2) In order to accurately capture the context related to the image and reduce the impact of text redundant information on text feature extraction, this method introduces a text feature compression unit at the back end of the text feature extraction unit, and compares the compressed text information with the output of the image encoder. This allows the text information compression unit to compress the potential embedding information of the text in the latent space to a low dimension. In the generation stage, the latent space text embedding information output by the text feature compression unit will provide corresponding text guidance for the downstream generation process.

10. The method for generating CXR images based on text according to claim 7, characterized in that: The step of gradually injecting general medical knowledge into the text-image generation model in a fine-grained manner specifically includes: 3.1) Latent space diffusion model training process. The diffusion process of the latent space diffusion model occurs in a more efficient low-dimensional latent space constructed by the pre-trained image feature extraction unit. The model learns the feature representation of the original data from the process of noise injection and restoration of the original data. The noise injection process is called the forward process, which simulates a fixed Markov chain of length T. Specifically: and Where: z0 is the latent space representation of normal CXR image z1~z T Represents the continuous noisy image representation from step 1 to step T in the latent space. In the forward process, the variance β is gradually added to the latent space representation z0 of the image. t ∈(0,1) noise that changes with time steps, where: t∈1,...T, the training goal is to add a noisy version of z to time t≤T t Noise prediction and removal are performed to reverse the noise addition process to obtain z t-1 , the corresponding training loss function is expressed as: Among them: ,∈ θ (z t ,t,c) represents the prediction of the real noise ∈ by a series of noise estimation models with parameters θ. The noise estimation models of all time steps share the same model weights. c is the text report information. When the number of training rounds reaches the set value, the training stops; 3.2) Report text template: Use the CheXbert model to process medical reports and generate category annotations, and convert the results into template text descriptions. Specifically, the text report is input into the CheXbert model, and the model automatically captures the key information of the text and outputs the key medical information contained in the report, and converts it into a template text description that "contains 'type'". Then, the text feature extraction unit and compression unit obtained in step 2 are introduced to receive the templated text description and perform feature extraction and compression on it in the latent space to obtain the latent space representation of the templated text; 3.3) During the training of the diffusion model, the trainable control adapter introduces templated text information into the pre-trained image generation model by gradually injecting medical knowledge into the network, further refining and highlighting the guiding role of key text information in the diffusion process. The specific implementation process is as follows: at the beginning of the training of the diffusion model learning unit, the noise estimation model in the unit is initialized using the pre-trained weights.

Images