Longitudinal thoracic cavity image synthesis method based on local disease semantic guide editing
By using local disease semantic guidance method in longitudinal chest image synthesis, using the GAN model and the connection of potential vectors Link-LR, the problems of insufficient local control and non-smoothing image are solved, and a high-level can truly reflect the progress trend of the disease is generated. Quality longitudinal image sequence.
Patent Information
- Application Number
- CN202411148742.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-12-03
AI Technical Summary
The prior art local control is not fine enough in the synthesis of longitudinal image data, the synthetic images are not smooth, discontinuous, and cannot effectively capture information about disease progression over time.
Using a longitudinal chest image synthesis method based on local disease semantic guidance, the link-LR is introduced by generating the connection between the adversarial network (GAN) model and the latent vector and the local area of interest (ROI), and the perturbation and constraint terms are introduced, and principal component analysis is used. Identify feature directions and optimize potential vectors to generate smooth, continuous longitudinal image sequences.
Fine local control of specific lesion areas of the image is achieved, the accuracy and nature of image editing are improved, and the generated longitudinal image sequence can truly reflect the disease progress trend.
Smart Images

Figure CN119067940A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image processing, and more specifically, to a longitudinal chest image synthesis method. Background Art
[0002] With the continuous advancement of medical technology and the improvement of people's health awareness, the demand for early detection of diseases and medical imaging analysis in the healthcare field is increasing. In this context, the importance of longitudinal image data has gradually become prominent because it can show the chronological order of the development causes and effects of diseases.
[0003] However, the collection and use of longitudinal image data face a series of challenges, and technical barriers are particularly significant, especially when interpreting and analyzing data, such as identifying subtle changes in radiological images, which requires professional medical knowledge.
[0004] As generative artificial intelligence has gradually attracted attention, methods based on generative adversarial networks (GANs) have achieved remarkable success in multiple fields of natural and medical images, especially in generating and editing images. However, their application in medical imaging for local lesion detection remains an area of active research. One of the main challenges in synthesizing radiological images is the subtlety of lesions, which complicates the extraction of disease-specific details from non-lesion backgrounds and leads to unexpected alterations in non-lesion regions.
[0005] M. Yang et al. from the University of Toronto proposed a mask-guided optimization method for learning semantic manifolds in the StyleGAN2 model in the paper "Mask-guided Discovery of Semantic Manifolds in Generative Models". This method only changes within the masked area, while the rest of the area remains unchanged. However, this method is limited by the lack of local control and cannot achieve more refined local control, and its interpretability is relatively weak. In addition, existing controllable image synthesis methods usually produce a single output image based on the input, which cannot capture important information about the progression of the disease over time. Therefore, being able to locally edit images with disease-aware content and generate a series of images that reflect the trend of disease progression is an important and unresolved problem. Summary of the invention
[0006] The purpose of the present invention is to overcome the defects of the existing method for synthesizing longitudinal image data, such as insufficient local control, uneven and discontinuous synthesis of longitudinal image data, and to provide a longitudinal chest image synthesis method based on local disease semantic guided editing.
[0007] To achieve the above object, the technical solution of the present invention is as follows:
[0008] The present invention provides a longitudinal chest image synthesis method based on local disease semantic guided editing, comprising the following steps:
[0009] Obtain a human chest X-ray dataset and select samples containing only normal categories and a specific disease type as the training dataset;
[0010] Using the training data set to train a generative adversarial network (GAN) model, dividing a local region of interest (ROI) of a generated image of the GAN, and establishing a connection (Link-LR) between a potential vector and the ROI region;
[0011] Based on the Link-LR connection, a GAN model is pre-trained;
[0012] Performing principal component analysis on the pre-trained GAN model and manually selecting the feature directions most relevant to a specific disease;
[0013] The characteristic direction is used to guide the optimization of the latent vector, and a constraint term is imposed to limit the distance between adjacent latent vectors, and a synthesized longitudinal image sequence is output.
[0014] Furthermore, the connection Link-LR between the potential vector and the ROI region is specifically:
[0015] The generator of the Generative Adversarial Network (GAN) takes z~P(z) as input, maps the input from the d-dimensional latent space to the high-dimensional image space X, and generates an output image x=G(z), where z represents the input latent vector and x represents the generated image;
[0016] The pixels in the local region of interest ROI of the generated image are defined as v i , other irrelevant pixels are defined as v ir , divide the potential space of GAN, extract a part of the subspace and define it as z i , the remainder is defined as z ir , for z i Introducing a disturbance P i =[0,..,0,p i ,0,...,0], and get the perturbed image:
[0017] △x=G(z+αP i )
[0018] A loss function is introduced to constrain and the value of the loss function is made as small as possible to establish the connection Link-LR between the potential vector and the ROI area. The loss function expression is:
[0019]
[0020] Among them, M iris the user-defined region of pixels of no interest v ir The binary mask, that is, only in v ir The pixel area value is 1, while in v i The value in the pixel area is 0. Indicates the addition of disturbance P i After that, for the irrelevant region v ir The degree of influence, ||·||2 represents Norm.
[0021] Furthermore, the disturbance p i The value of is sampled from the standard Gaussian distribution N~(0,1), and the disturbance p i Only in w i has non-zero values inside and zero elsewhere, which is used for the ROI region v i Apply control while minimizing the impact on v ir impact.
[0022] Furthermore, during the training process of the pre-trained GAN model based on the Link-LR connection, an optimizer is used to perform iterative training.
[0023] Furthermore, the steps of performing principal component analysis on the pre-trained GAN model and manually selecting the feature direction most relevant to a specific disease are specifically as follows:
[0024] Performing principal component analysis (PCA) on the latent space of the pre-trained GAN model to decouple lesion-aware semantic information in the latent space and identify major change directions in directions corresponding to different semantic features;
[0025] Manually select the principal component direction that is most relevant to a specific disease and causes the most significant feature changes and record it as n, and denote the latent vector of the pre-trained GAN model as z L , z L The latent vector to be optimized in is denoted as z Li , z L and z Li The difference between them is recorded as n i ,
[0026]
[0027] After selecting the most relevant principal component directions, the following loss function is minimized:
[0028]
[0029] Where n is the manually selected principal component direction and θ′ is an adjustable tolerance parameter.
[0030] Furthermore, the use of the feature direction to guide the potential vector optimization is to guide the vector z Li Trajectories along a specific direction n are optimized to exert semantic control over the generated images, thus ensuring that the synthesized longitudinal image sequences exhibit features associated with a specific disease.
[0031] Furthermore, the imposed constraints are specifically:
[0032] The constraints are defined as:
[0033]
[0034] Among them, z Li and z L(i+k) is the adjacent potential vector to be optimized, k represents the interval between adjacent potential vectors, and d max is the maximum distance allowed;
[0035] The constraint term is minimized, and the minimized constraint term is imposed when the feature direction is used to guide the optimization of the latent vector to ensure smooth and consistent changes between the latent vectors.
[0036] Furthermore, the minimizing constraint term is to calculate the constraint term when k is equal to 1 and 2.
[0037] Furthermore, after the output of the synthesized longitudinal image sequence, a segmentation verification step is also included:
[0038] Preset parameters and use the preset parameters to analyze pixel-level differences between the longitudinal image sequence and original images in a training data set;
[0039] Areas with differences below the preset parameters are considered potential background patterns, while areas above the preset parameters show potential abnormalities and pathological changes;
[0040] These results were compared with the annotation results of medical experts to verify the effectiveness and practical value of the longitudinal chest image synthesis method based on local disease semantics-guided editing in locally controlling the generation of chest X-ray images.
[0041] Furthermore, after the output of the synthesized longitudinal image sequence, an AI model verification step is also included:
[0042] The longitudinal image sequence is input into an existing medical multimodal model to diagnose the disease type, lesion area and severity, so as to verify the authenticity and reliability of the longitudinal image sequence data generated by the longitudinal chest image synthesis method based on local disease semantic guided editing.
[0043] Compared with the prior art, the present invention has the following beneficial effects:
[0044] The present invention establishes a novel connection mechanism between latent vectors and ROI pixel areas, which can finely control specific lesion areas in images locally. By introducing targeted disturbances in the latent space, principal component analysis is used to identify the main semantic directions, and in the image generation process, the directions most relevant to the specific disease are manually selected to guide image generation. Constraints are introduced between adjacent latent vectors so that the generated longitudinal image sequence can change smoothly and continuously, ensuring that the pixel values of the ROI area are accurately adjusted while keeping other parts of the image unchanged, thereby improving the accuracy of image editing and making the generated images more natural and realistic. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 It is a flowchart of the longitudinal chest image synthesis method of the present application.
[0046] Figure 2 It is a flowchart of the Link-LR module and the Disease-guided PCA module of the embodiment of the present application.
[0047] Figure 3 This is a composite result diagram of cardiac hypertrophy, pleural effusion and edema in an embodiment of the present application.
[0048] Figure 4 This is a schematic diagram comparing pleural effusion images using the method LDEdit in this application and other image editing methods.
[0049] Figure 5 It is the segmentation verification result of the embodiment of the present application and the comparison with the annotation effect of human experts.
[0050] Figure 6 This is the verification result of the medical multimodal large model of the embodiment of the present application. DETAILED DESCRIPTION
[0051] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0053] Example 1
[0054] See also Figure 1The embodiment of the present invention provides a longitudinal chest image synthesis method based on local disease semantic guided editing, comprising the following steps:
[0055] S1: Get a human chest X-ray dataset and select samples that only contain normal categories and a specific disease type as the training dataset:
[0056] In the embodiment of the present invention, the human chest X-ray dataset obtained in S1 is derived from the CheXpert dataset, which includes 224,316 chest X-ray radiographs, divided into frontal and side X-ray images, including 14 categories such as health, mediastinal enlargement, cardiomegaly, lung opacity, lung lesions, edema, lung consolidation, pneumonia, atelectasis, pneumothorax, pleural effusion, other pleura, fracture and auxiliary equipment. This embodiment selects three disease types with significant pathological characteristics, namely cardiomegaly, pleural effusion and edema, selects all data of these three specific diseases from the CheXpert dataset, uses only samples taken from the front, and merges them with normal samples to form a training dataset.
[0057] S2: Use the training data set to train a generative adversarial network (GAN) model, divide the local region of interest (ROI) of the generated image of the GAN, and establish a connection (Link-LR) between the potential vector and the ROI region:
[0058] In step S2, the generative adversarial network GAN uses the StyleGAN2 network in Tero Karras' paper "Analyzing and Improving the Image Quality of StyleGAN". It should be noted that the method of the present invention can also be implemented using other GAN network models.
[0059] See also Figure 2 , step S2 is specifically as follows: put the training set data in step S1 into the StyleGAN2 network for training, and focus on establishing the connection between the GAN latent space and the ROI area in the synthetic image during the training process. First, customize the ROI area in the synthetic image for different disease types. For example, in the disease of cardiac hypertrophy, define the position of the heart as the ROI area, and select the first 64 dimensions of the GAN latent space to connect with the ROI area using the loss function constraint. The specific connection method is:
[0060] The generator of the Generative Adversarial Network (GAN) takes z~P(z) as input, maps it from the d-dimensional latent space to the high-dimensional image space X, and generates an output image x=G(z), where z represents the input latent code and x represents the generated image.
[0061] We define the pixels in the local region of interest ROI of the generated image as v i , other irrelevant pixels are defined as v ir , divide the potential space of GAN, extract a part of the subspace and define it as z i , and define the remainder as z ir , in the z i Minimize v when introducing slight perturbations ir , so that the ROI area v i Apply control without affecting region v ir .
[0062] In order to achieve the above purpose, the disturbance P is introduced i =[0,..,0,p i ,0,...,0], the perturbation is only i has non-zero values inside and zero elsewhere. i The value of is sampled from the standard Gaussian distribution N~(0,1). After adding the disturbance, the perturbed image can be obtained:
[0063] △x=G(z+αP i )
[0064] The following loss function is introduced:
[0065]
[0066] Among them, M or is the user-defined region of pixels of no interest v ir The binary mask, that is, only in v ir The value in the pixel area is 1, while in the pixel area v i The internal value is 0. Indicates the addition of disturbance P i After that, for the irrelevant region v ir The degree of influence, ||·||2 represents Norm.
[0067] make The value of is as small as possible to achieve precise local control of the ROI area without affecting other areas.
[0068] S3: Based on the Link-LR, pre-trained GAN model:
[0069] During the training process, each image was uniformly cropped to a 256×256 resolution image and compressed into a zip format. At the same time, Adam was used as the optimizer during the training process, with 5000 epochs. After the training was completed, the model parameters were saved.
[0070] S4: Perform principal component analysis on the pre-trained GAN model and manually select the feature directions most relevant to a specific disease:
[0071] In the latent space of GAN, rich disease-specific semantic information is embedded. By decoupling the semantic information within the latent space, we can discover the directions associated with specific disease characteristics, thereby guiding the optimization of the latent vector to generate images with corresponding characteristics.
[0072] See also Figure 2 , first use the loss function We build a latent space of the GAN model connected to Link-LR pre-training and perform principal component analysis (PCA) to identify the main directions of change, which correspond to different semantic features. Then, we manually select the most relevant main direction, denoted as n, which is most relevant to a specific disease and leads to the most significant feature change. For example, the presence of a small amount of pleural effusion leads to the feature direction of blunting the costophrenic angle. For cardiac hypertrophy, we select the heart size as the main semantic direction. We will use the loss function The potential vector after establishing Link-LR is represented as z L , denote the potential vector to be optimized as z Li . Let the difference between them be n i ,
[0073]
[0074] Then, the following loss function is minimized:
[0075]
[0076] Where n is the principal component direction we pre-selected and θ′ is an adjustable tolerance parameter.
[0077] In this example, 10 principal components were explored and then the semantic directions most relevant to the characteristic disease features were manually selected, with the tolerance parameter θ′ set to 35.
[0078] S5: Using the feature direction to guide the optimization of the latent vector, and applying constraints to limit the distance between adjacent latent vectors, and outputting a synthesized longitudinal image sequence:
[0079] See also Figure 2 , guide vector z Li Optimizing along a trajectory in a specific direction n enables semantic control over the generated images. In this way, image synthesis can be manipulated more precisely, ensuring that the synthesized longitudinal image sequences exhibit features associated with a specific disease.
[0080] During the disease progression, it is necessary to ensure smooth and consistent changes between latent vectors to generate natural and realistic longitudinal image sequence samples that represent the disease progression trend. To achieve this, a constraint term is introduced between latent vectors to limit the size of each pair of adjacent latent vectors z Li and z L(i+k) The constraint term is defined as follows:
[0081]
[0082] Among them, k represents the interval between adjacent vectors, d max is the maximum distance allowed.
[0083] Minimizing the constraint term ensures that the behaviors between adjacent latent vectors are constrained, thus maintaining smooth transitions during translation.
[0084] Calculate the constraints when k is equal to 1 and 2 respectively, and set the maximum distance d allowed between adjacent potential vectors max =0.3.
[0085] By optimizing the latent vector along the preset semantic direction and imposing constraints, a series of longitudinal image data containing characteristic information of specific disease progression can be generated and output.
[0086] See also Figure 3 , showing the result diagram of cardiac hypertrophy, pleural effusion and edema synthesis in Example 1 of the present invention,
[0087] See also Figure 4 , showing a schematic diagram comparing pleural effusion images using the method LDEdit of the present invention and other image editing methods.
[0088] Example 2
[0089] See also Figure 5 Based on Example 1, this embodiment further includes a segmentation verification step after generating and outputting a series of longitudinal image data containing characteristic information of specific disease progression:
[0090] Preset parameters and use the preset threshold parameters to analyze pixel-level differences between the longitudinal image sequence and original images in a training data set;
[0091] Areas with differences below the preset threshold parameters are considered potential background patterns, while areas above the preset threshold parameters show potential abnormalities and pathological changes;
[0092] These results were compared with the annotation results of medical experts to verify the effectiveness and practical value of the method of the present invention in locally controlling the generation of chest X-ray images.
[0093] The other steps of this embodiment 2 are the same as those of embodiment 1 and will not be repeated here.
[0094] Example 3
[0095] See also Figure 6 Based on Example 1, after generating and outputting a series of longitudinal image data containing characteristic information of specific disease progression, Example 3 further includes an AI model verification step:
[0096] The generated longitudinal image sequence data representing the early and late stages of the disease were input into the existing most advanced medical multimodal model XrayGLM proposed by Wang et al. in the paper “XrayGLM: Medical Diagnosis of Chest X-rays Based on Fine-tuning Training of VisualGLM-6B” to diagnose the disease type, lesion area and severity, so as to verify the authenticity and reliability of the longitudinal image sequence data generated by the method of the present invention.
[0097] The other steps of this embodiment 3 are the same as those of embodiment 1 and will not be repeated here.
[0098] It should be noted that the segmentation verification step and the AI model verification step in Example 2 and Example 3 can also be combined together.
[0099] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A longitudinal chest image synthesis method based on local disease semantics guided editing, characterized in that: The following steps are involved: Obtain a human chest X-ray dataset and select samples containing only normal categories and a specific disease type as the training dataset; Using the training data set to train a generative adversarial network (GAN) model, dividing a local region of interest (ROI) of a generated image of the GAN, and establishing a connection (Link-LR) between a potential vector and the ROI region; Based on the Link-LR connection, a GAN model is pre-trained; Performing principal component analysis on the pre-trained GAN model and manually selecting the feature directions most relevant to a specific disease; The characteristic direction is used to guide the optimization of the latent vector, and a constraint term is imposed to limit the distance between adjacent latent vectors, and a synthesized longitudinal image sequence is output.
2. According to claim 1, a longitudinal chest image synthesis method based on local disease semantics guided editing is characterized in that: The connection Link-LR between the potential vector and the ROI region is specifically: The generator of the Generative Adversarial Network (GAN) takes z~P(z) as input, maps the input from the d-dimensional latent space to the high-dimensional image space X, and generates an output image x=G(z), where z represents the input latent vector and x represents the generated image; The pixels in the local region of interest ROI of the generated image are defined as v i , other irrelevant pixels are defined as v ir , divide the potential space of GAN, extract a part of the subspace and define it as z i , the remainder is defined as z ir , for z i Introduce disturbance P i =[0,..,0,p i ,0,...,0], and get the perturbed image: △x=G(z+αPi) The loss function is introduced to constrain and the value of the loss function is made as small as possible to establish the connection Link-LR between the potential vector and the ROI area. The loss function expression is: Among them, M ir is the user-defined region of pixels of no interest v ir The binary mask, that is, only in v ir The pixel area value is 1, while in v i The value in the pixel area is 0. Indicates the addition of disturbance P i After that, for the irrelevant area v ir The degree of influence, ||·||2 represents Norm.
3. The method for synthesizing longitudinal thoracic images based on local disease semantics-guided editing according to claim 2, characterized in that: The disturbance p i The value of is sampled from the standard Gaussian distribution N~(0,1), and the disturbance p i Only in w i has non-zero values inside and zero elsewhere, which is used for the ROI region v i Apply control while minimizing the impact on v ir impact.
4. The longitudinal thoracic image synthesis method based on local disease semantics guided editing according to claim 1, characterized in that: During the training process of the pre-trained GAN model based on the Link-LR connection, an optimizer is used to perform iterative training.
5. The method for synthesizing longitudinal thoracic images based on local disease semantics-guided editing according to claim 1, characterized in that: The steps of performing principal component analysis on the pre-trained GAN model and manually selecting the feature directions most relevant to a specific disease are specifically as follows: Performing principal component analysis (PCA) on the latent space of the pre-trained GAN model to decouple lesion-aware semantic information in the latent space and identify major change directions in directions corresponding to different semantic features; Manually select the principal component direction that is most relevant to a specific disease and causes the most significant feature changes and record it as n, and denote the latent vector of the pre-trained GAN model as z L , z L The latent vector to be optimized in is denoted as z Li , z L and z Li The difference between them is recorded as n i , After selecting the most relevant principal component directions, the following loss function is minimized: Where n is the manually selected principal component direction and θ′ is an adjustable tolerance parameter.
6. A longitudinal chest image synthesis method based on local disease semantics guided editing according to any one of claims 1 or 5, characterized in that: The feature direction is used to guide the optimization of the potential vector, that is, the guide vector z Li The generated images are semantically controlled by optimizing the trajectory along a specific direction n, ensuring that the synthesized longitudinal image sequences exhibit features associated with a specific disease.
7. The method for longitudinal thoracic image synthesis based on local disease semantics guided editing according to claim 1, characterized in that: The constraints imposed are specifically: Define the constraints as: Among them, z Li and z L(i+k) is the adjacent potential vector to be optimized, k represents the interval between adjacent potential vectors, and d max is the maximum distance allowed; The constraint term is minimized, and the minimized constraint term is imposed when the feature direction is used to guide the optimization of the latent vector to ensure smooth and consistent changes between the latent vectors.
8. The method for synthesizing longitudinal thoracic images based on local disease semantics-guided editing according to claim 7, characterized in that: The minimization constraint term is to calculate the constraint term when k is equal to 1 and 2.
9. The longitudinal thoracic image synthesis method based on local disease semantics guided editing according to claim 1, characterized in that: After the output of the synthesized longitudinal image sequence, a segmentation verification step is also included: Preset parameters and use the preset parameters to analyze pixel-level differences between the longitudinal image sequence and original images in a training data set; Areas with differences below the preset parameters are considered potential background patterns, while areas above the preset parameters show potential abnormalities and pathological changes; These results were compared with the annotation results of medical experts to verify the effectiveness and practical value of the longitudinal chest image synthesis method based on local disease semantics-guided editing in locally controlling the generation of chest X-ray images.
10. The longitudinal thoracic image synthesis method based on local disease semantics guided editing according to claim 1, characterized in that: After the output of the synthesized longitudinal image sequence, an AI model verification step is also included: the longitudinal image sequence is input into an existing medical multimodal model to diagnose the disease type, lesion area and severity, so as to verify the authenticity and reliability of the longitudinal image sequence data generated by the longitudinal chest image synthesis method based on local disease semantic guided editing.
Citation Information
Patent Citations
Systems and methods for detecting an indication of malignancy in a sequence of anatomical images
US20190236782A1
Brain image-based quantitative brain disease prediction method and apparatus
US20240079139A1
Private synthetic time series data generation
US20240249021A1
Sequence transcription with deep neural networks
US8965112B1
Multimodal three-dimensional medical image fusion method and system, and electronic device
WO2021022752A1
