Method for converting back photos into spine X-ray images based on stable diffusion model
Through the technology based on stable diffusion model, the back photos are converted into high-quality spine X-ray images, which solves the problems of low X-ray images and radiation exposure in the existing technology, and achieves low-cost and efficient spine in remote areas Convex screening improves the accuracy and coverage of screening.
Patent Information
- Application Number
- CN202311748637.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-19
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2043-12-19
AI Technical Summary
When using deep learning for scoliosis screening, the generated X-ray images are not of high quality, making it difficult to accurately evaluate the severity of lateral convexity. Traditional X-ray examinations have problems with radiation exposure and scarcity of resources, making it difficult to be in remote areas. Effective screening.
Using a method based on a stable diffusion model, the back photos are converted into spine X-ray images, high-quality X-ray images are generated through encoder, decoder and UNet, and the posture and shape alignment are aligned with the cross attention mechanism and the YOLOv8 model, and the calculation is carried out. Cobb angles are used to assess lateral convex severity.
Achieve efficient and convenient scoliosis screening, reduce radiation exposure, improve screening coverage and accuracy, and generate clear and real X-ray images at low cost without special equipment, providing doctors with more accurate The basis for judgment.
Smart Images
Figure CN117689533B_ABST
Abstract
Description
Technical Field
[0001] The invention discloses a method for converting a back photograph into a spinal X-ray image based on a stable diffusion model, and belongs to the technical field of computer vision and medical image analysis. Background Art
[0002] Deep learning is currently the most widely used artificial intelligence technology. It simulates the workings of the human brain's neural networks, automatically extracting useful information from large amounts of data. Medicine, a science that concerns human health, faces a variety of complex data and clinical challenges. Combining deep learning with medicine can allow patients to enjoy hospital-based medical services in their communities and help doctors quickly and accurately assess patients, leading to more efficient and safer examination and treatment plans. Deep learning has already demonstrated remarkable potential and advantages in fields such as medical image analysis.
[0003] Adolescent idiopathic scoliosis (AIS) is a three-dimensional spinal deformity that occurs in children aged 10-18 years. Its etiology remains largely unknown. It is the most common cause of spinal health problems in adolescents and one of the three major health issues affecting primary and secondary school students in my country, second only to myopia and obesity in incidence. AIS not only affects patients' image and self-esteem but also compresses internal organs, impairs the respiratory, cardiovascular, and nervous systems, and can even be life-threatening. Therefore, early detection and intervention of AIS and slowing the progression of scoliosis are of great clinical importance. However, in remote areas, X-rays are scarce, and regular follow-up visits to hospitals are difficult. Furthermore, X-rays are time-consuming, costly, and carry the risk of radiation exposure, requiring specialized personnel for guidance. Therefore, reducing reliance on spinal X-rays with low-cost, radiation-free, and easily accessible back radiographs is a crucial yet challenging issue.
[0004] Most hospitals in my country can only perform AIS testing offline. Due to the uneven distribution of medical resources, patients struggle to find experienced orthopedic surgeons, potentially leading to delays in AIS diagnosis and treatment. Patients travel to hospitals for registration, waiting, and X-rays, all of which consume time and money, increasing transportation and radiation risks. Furthermore, communication between doctors and patients is often disrupted, with busy doctors and unclear patients, leading to misdiagnosis, missed diagnosis, and medical disputes. If we could perform an initial screening assessment online, we could save patients money, improve testing efficiency, and significantly improve healthcare in remote areas.
[0005] Existing patent applications, such as patent number 202110584759.9, innovate by automatically analyzing captured back photographs, extracting features from both the 2D back photograph and the 3D image information for classification, and determining the presence of scoliosis. However, this method has some drawbacks, such as being less intuitive and having low accuracy in assessing scoliosis severity. In contrast, our method is more intuitive, generating corresponding X-ray images from back photographs, thus providing more robust evidence for judgment.
[0006] Image generation technology is a technique that generates new images based on input data or conditions. Image generation has many applications in medicine, such as generating X-rays, MRIs, and CT images for purposes such as assessment, education, and research. However, image generation also faces numerous challenges, such as improving image quality, diversity, consistency, and controllability. Currently, some image generation models perform poorly on X-ray images, resulting in blurry, distorted, unnatural images, or images that do not conform to the input. These models may utilize techniques such as generative adversarial networks (GANs) or variational autoencoders (VAEs), which are latent variable-based generative models that learn an underlying probability distribution from input data and then generate new data from this distribution. While these techniques perform well on some image generation tasks, they may encounter difficulties with X-ray images, such as the limited amount of data, complex data distribution, and high noise levels.
[0007] The stable diffusion model is a new image generation technique that does not require learning the distribution of a latent variable. Instead, it uses a backward diffusion process to gradually transform the observed data into a simple prior distribution, such as Gaussian white noise. Then, through a forward denoising process, the prior distribution is gradually restored to the original data. This process can be divided into many small steps, each using a neural network to predict the denoised distribution for the next step. The stable diffusion model can generate high-quality, high-resolution, and highly diverse images, and can easily control the generated image attributes, such as text, category, and style. The stable diffusion model has demonstrated its potential and advantages in fields such as medical image analysis. Can we improve the stable diffusion model to accept paired inputs of back photographs and X-rays? This would allow it to better handle the noise, details, and structure of the data, more flexibly adapt to the distribution and variation of the data, and more fully utilize the input conditions?
[0008] To address this issue, we propose a feature map-based approach to stabilize the diffusion model. Summary of the Invention
[0009] The purpose of the present invention is to provide a method for converting back photos into spinal X-ray images based on a stable diffusion model to solve the problems raised in the above-mentioned background technology. It can convert back photos into X-ray images, thereby improving the screening quality of scoliosis. Our method assists in assessing the severity of diseases such as scoliosis by analyzing back photos. This method has the following advantages: efficient and convenient: compared with traditional X-ray examinations, this method does not require patients to go to the hospital to wait in line, and only requires taking a back photo to give results in less than a second; no radiation exposure hazards: compared with X-ray examinations, this method will not cause any radiation damage to patients, nor will it increase the risk of cancer in patients; improve screening coverage and accuracy: compared with manual screening, this method can greatly improve screening coverage and accuracy, especially for remote areas and resource-scarce areas, and can achieve low-cost and high-efficiency scoliosis screening.
[0010] To achieve the above object, the present invention provides the following technical solution: a method for converting a back photograph into a spinal X-ray image based on a stable diffusion model, comprising the following steps:
[0011] Step 1: Input the back photos of 2049 patients with adolescent idiopathic scoliosis (AIS) provided by the user. The images do not require special equipment or conditions and ensure the clarity and completeness of the back.
[0012] Step 2: The back photo is input as a prompt into the stable diffusion model, which consists of an encoder, a decoder, and a UNet, and can convert the back photo into a spinal X-ray image.
[0013] Step 3: During the conversion process, the back photo and X-ray image are first aligned to make the posture and shape of the two consistent.
[0014] Step 4: Input the X-ray image into the encoder to obtain the potential vector, which is mapped to an isotropic Gaussian distribution (pure noise) by gradually adding noise.
[0015] Step 5: Use the fully connected layer to extract the features of the back photo and map the back photo into the vector space.
[0016] Step 6: Use UNet to denoise the pure noise, and at the same time map the feature vector of the back photo to the middle layer of UNet through cross attention.
[0017] Step 7: Input the denoised latent vector into the decoder to obtain the X-ray image and predict the scoliosis condition.
[0018] Step 8: Calculate the Cobb angle using the generated virtual X-ray image and compare it with the Cobb angle of the real X-ray image to evaluate the effectiveness and accuracy of the method.
[0019] Furthermore, in step 1, the user provides a photo of their back. This image does not require special equipment or conditions, as long as the back is clear and intact. The purpose of this step is to obtain information about the user's back in preparation for subsequent steps. The user can take a photo of their back with a regular camera or mobile phone; no markers or sensors are required. The user should maintain an upright posture, avoiding obstruction or tilting of the back. The user can take the photo themselves or with assistance. The back photo should be clear and complete, with edges and details clearly visible, and should not be blurry, dark, or overexposed.
[0020] Furthermore, in step 2, the back photo is fed as a prompt into a stable diffusion model (SDM), which consists of an encoder, a decoder, and a unified neural network (UNet). This model converts the back photo into an X-ray image. The encoder encodes the spinal X-ray image into a latent vector, and the decoder decodes the latent vector into an X-ray image. The UNet, a convolutional neural network with skip connections, denoises the latent vector and maps the back photo features to the UNet's intermediate layers through cross-attention. The SDM generates a matching virtual spinal X-ray image based on the information in the back photo.
[0021] Furthermore, in step three, before conversion, the back photograph and X-ray image are aligned to make the poses and shapes of the two consistent. The purpose of this step is to eliminate the differences between the two images and improve the accuracy and consistency of the conversion. Since the two images are taken under different conditions, they may have differences in pose and shape, such as angle, size, etc. These differences will affect the conversion effect of the stable diffusion model, resulting in the generated X-ray image being inconsistent with the real X-ray image. To solve this problem, we used a target detection and instance segmentation method based on the YOLOv8 model, which can automatically detect and segment the key spinal parts from the two images. In this way, we can obtain a set of back photographs and X-ray images with consistent poses and shapes as the input and output of the stable diffusion model.
[0022] Furthermore, in step 4, the X-ray image is fed into the encoder to obtain a latent vector, which is then gradually mapped to an isotropic Gaussian distribution (pure noise) by gradually adding noise. This step aims to transform the observed data into a simple prior distribution, paving the way for subsequent denoising and restoration. The encoder encodes the X-ray image into a latent vector, then gradually adds noise to map it to an isotropic Gaussian distribution (pure noise). In this way, we can transform the complex X-ray image into a simple noisy image, which serves as an intermediate state for a stable diffusion model.
[0023] Furthermore, in step 5, fully connected layers are used to extract features from the back photo, mapping it into a vector space. This primarily improves the robustness and controllability of the conditional input, thereby enhancing the quality of the generated image. Fully connected layers combine the features extracted by the convolutional layers to produce higher-level feature representations. These feature representations can be mapped into a vector space, making them more easily processed and controlled. This approach can improve the quality of generated images, making them more realistic and natural.
[0024] Furthermore, in step 6, pure noise is denoised using a UNet, while features from the back photo are simultaneously mapped to the UNet's intermediate layers via cross-attention. This step aims to leverage information from the back photo to denoise pure noise and generate a matching X-ray image. Cross-attention is a mechanism for achieving feature interaction and fusion, allowing weighted transfer of back photo features to the UNet.
[0025] Furthermore, in step 7, the denoised latent vector is fed into the decoder to produce the X-ray image. This step aims to restore the denoised latent vector to its original form, the X-ray image. The decoder's structure is the opposite of the encoder's; it decodes the latent vector into the X-ray image.
[0026] Furthermore, in step eight, the Cobb angle is automatically calculated using the generated X-ray images and compared with the Cobb angle of real X-ray images to evaluate the effectiveness and accuracy of the method. The purpose of this step is to use the generated X-ray images to assist in assessing the severity of conditions such as scoliosis. The Cobb angle is a metric used to measure the degree of scoliosis. It refers to the angle formed by the upper and lower endplates of the two most inclined vertebrae. The size of the Cobb angle reflects the severity of scoliosis. Generally speaking, a Cobb angle greater than 10 degrees is considered scoliosis, and a Cobb angle greater than 40 degrees indicates the need for surgical treatment. We used a deep learning-based object detection method that automatically detects vertebral centers from X-ray images and then calculates the Cobb angle. We calculated the Cobb angle using the generated X-ray images and compared it with the Cobb angle of real X-ray images to evaluate the effectiveness and accuracy of our method. We used several evaluation metrics, such as mean squared error (MSE).
[0027] The present invention inputs a back photograph into a stable diffusion model, aligns the back photograph and X-ray image, inputs the X-ray image into an encoder, uses UNet to denoise pure noise, and simultaneously extracts the feature vector of the back photograph using a fully connected layer and maps it to the middle layer of UNet through cross-attention. Finally, the generated X-ray image is used to automatically calculate the Cobb angle and predict the condition of scoliosis.
[0028] The technical solution provided by the present invention has the following technical effects:
[0029] 1. The stable diffusion model utilizes conditional inputs to guide the generation process, ensuring image authenticity and consistency, and generating corresponding X-ray images from back photographs. Compared to models such as pix2pix and CycleGAN, the stable diffusion model better preserves image detail and contrast, resulting in clearer, more realistic X-ray images. This provides more information and evidence for AIS assessment, helping doctors determine whether a patient's spine exhibits scoliosis.
[0030] 2. Patients can receive professional medical services at home, saving time and money. Doctors can also remotely review the patient's X-ray images through the online platform. This approach not only provides convenience for AIS patients but also reduces pressure on hospitals and improves medical efficiency. Furthermore, through the online platform, doctors can communicate and consult with other experts, gaining more professional advice and suggestions, thereby improving the quality and effectiveness of AIS patient treatment. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 This is a flowchart of an embodiment of the present invention;
[0032] Figure 2 This is the model architecture of an embodiment of the present invention;
[0033] Figure 3 The encoder and decoder of the embodiment of the present invention;
[0034] Figure 4 A UNet diagram according to an embodiment of the present invention;
[0035] Figure 5 This is a conditional encoder according to an embodiment of the present invention. DETAILED DESCRIPTION
[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0037] like Figure 1-Figure 5 As shown, the present invention provides a method for converting a back photograph into a spinal X-ray image based on a stable diffusion model, comprising the following steps:
[0038] 1. The user provides a photo of their back. This image does not require special equipment or conditions, as long as the back is clear and intact. The purpose of this step is to obtain information about the user's back in preparation for subsequent steps. The user can use a home camera or smartphone to take a photo of their back; no markers or sensors are required. The user should maintain an upright posture, avoiding obstruction or tilting of the back. The user can take the photo themselves or with assistance. The back photo should be clear and intact, with edges and details clearly visible. It should not be blurry, dark, or overexposed.
[0039] 2. The back photo is fed as a prompt into a stable diffusion model, consisting of an encoder, decoder, and Unified Network (UNet), which converts the back photo into an X-ray image. The encoder encodes the X-ray image into a latent vector, and the decoder decodes the latent vector into the X-ray image. The UNet, a convolutional neural network with skip connections, denoises the latent vector and maps the back photo features to the UNet's intermediate layers through cross-attention. The stable diffusion model generates a matching virtual spinal X-ray image based on the information in the back photo.
[0040] 2. Before conversion, the back photos and X-ray images are aligned to make them consistent in pose and shape. The purpose of this step is to eliminate the differences between the two images and improve the accuracy and consistency of the conversion. Since the two images are taken under different conditions, they may have differences in pose and shape, such as angle, size, etc. These differences will affect the conversion effect of the stable diffusion model, resulting in the generated X-ray image being inconsistent with the real X-ray image. To solve this problem, we use a method of target detection (X-rays) and instance segmentation (back photos), which can automatically detect and segment key parts from the two images. In this way, we can obtain a set of back photos and X-ray images with consistent pose and shape as the input and output of the stable diffusion model.
[0041] 4. The X-ray image is fed into the encoder to obtain a latent vector. This latent vector is then gradually mapped to an isotropic Gaussian distribution (pure noise) by gradually adding noise. This step aims to transform the observed data into a simple prior distribution, preparing for subsequent denoising and restoration. The encoder encodes the X-ray image into a latent vector and then gradually maps it to an isotropic Gaussian distribution (pure noise) by gradually adding noise. This allows us to transform the complex X-ray image into a simple noisy image, serving as an intermediate state for a stable diffusion model.
[0042] 5. Use the UNet to denoise the pure noise. Simultaneously, cross-attention is used to map the back photo features extracted using the fully connected layer to the UNet's intermediate layers. This step aims to better utilize the information from the back photo to denoise the pure noise and generate a matching X-ray image. Cross-attention is a mechanism for achieving feature interaction and fusion, which allows weighted transfer of back photo features to the UNet.
[0043] 6. The denoised latent vector is fed into the decoder to produce the X-ray image. This step aims to restore the denoised latent vector to its original form, the X-ray image. The decoder's structure is the opposite of the encoder's; it decodes the latent vector into the X-ray image.
[0044] 7. Automatically calculate the Cobb angle using the generated virtual X-ray images and compare it with the Cobb angle from real X-ray images to evaluate the effectiveness and accuracy of the method. The goal of this step is to use the generated X-ray images to assist in assessing the severity of conditions such as scoliosis. The Cobb angle is a metric used to assess scoliosis severity. It is calculated by identifying the two most tilted vertebrae on the scoliotic area on the X-ray image and measuring the angle between the extended lines of the upper and lower endplates of these two vertebrae. Generally speaking, a Cobb angle greater than 10 degrees is considered scoliosis; a Cobb angle greater than 20 degrees recommends conservative bracing; and a Cobb angle greater than 40 degrees suggests consideration of spinal correction surgery. We used a deep learning-based object detection method that automatically detects vertebral centers from X-ray images and then calculates the Cobb angle. We calculated the Cobb angle using the generated virtual X-ray images and compared it with the Cobb angle from real X-ray images to evaluate the effectiveness and accuracy of our method. We used several evaluation metrics, such as mean squared error (MSE).
Claims
1. A method for converting back photographs into spinal X-ray images based on a stable diffusion model, characterized in that: The following steps are involved: Step 1: Input the back photo provided by the user to ensure the clarity and completeness of the back; Step 2: The back photo is input as a prompt into the stable diffusion model, which consists of an encoder, a decoder, and a UNet, and can convert the back photo into a spinal X-ray image; Step 3: During the conversion process, the back photo and X-ray image are first aligned to make the posture and shape of the two consistent; Step 4: Input the X-ray image into the encoder to obtain the latent vector, which is mapped to an isotropic Gaussian distribution by gradually adding noise; Step 5: Use the fully connected layer to extract the features of the back photo and map the back photo into the vector space; Step 6: Use UNet to denoise the pure noise, and map the feature vector of the back photo to the middle layer of UNet through cross attention; Step 7: Input the denoised latent vector into the decoder to obtain the X-ray image and predict the condition of scoliosis; Step 8: Automatically calculate the Cobb angle using the generated X-ray image and compare it with the Cobb angle of the real X-ray image to evaluate the effectiveness and accuracy of the method.
2. The method for converting back photographs into spinal X-ray images based on a stable diffusion model according to claim 1, characterized in that: In step 1, the user uses an ordinary camera or mobile phone to take a photo of the back without wearing a marker or sensor; the user should maintain an upright posture to avoid blocking or tilting the back; the user can take a selfie or ask someone to help; the back photo should be clear and complete, with edges and details visible, and should not be blurred, dim or overexposed.
3. The method for converting back photographs into spinal X-ray images based on a stable diffusion model according to claim 2, characterized in that: In the step 2, the back photo is input as a prompt into a stable diffusion model, which consists of an encoder, a decoder and a UNet, and can convert the back photo into an X-ray image; the encoder encodes the X-ray image into a latent vector, and the decoder decodes the latent vector into an X-ray image; UNet is a convolutional neural network with jump connections, which denoises the latent vector and maps the features of the back photo to the middle layer of UNet through cross attention; the stable diffusion model generates a matching X-ray image based on the information of the back photo.
4. The method for converting back photographs into spinal X-ray images based on a stable diffusion model according to claim 3, characterized in that: In step three, the back photo and the X-ray image are aligned before conversion to make the posture and shape of the two images consistent; the purpose is to eliminate the difference between the two images and improve the accuracy and consistency of the conversion.
5. The method for converting back photographs into spinal X-ray images based on a stable diffusion model according to claim 4, characterized in that: In the step 4, the X-ray image is input into the encoder to obtain a latent vector, which is mapped to an isotropic Gaussian distribution by gradually adding noise, with the aim of converting the observed data into a simple prior distribution in preparation for subsequent denoising and recovery; the encoder encodes the X-ray image into a latent vector, and then maps it to an isotropic Gaussian distribution by gradually adding noise; the complex X-ray image is converted into a simple noise image as an intermediate state of the stable diffusion model.
6. The method for converting back photographs into spinal X-ray images based on a stable diffusion model according to claim 5, characterized in that: In the step 5, the back photo features are extracted using a fully connected layer, and the back photo is mapped into a vector space; the robustness and controllability of the conditional input are improved, thereby improving the quality of the generated image; the fully connected layer integrates the features extracted by the convolutional layer, thereby obtaining a higher-level feature representation; These feature representations are mapped into a vector space so that they can be better processed and controlled.
7. The method for converting back photographs into spinal X-ray images based on a stable diffusion model according to claim 6, characterized in that: In the step six, UNet is used to denoise the pure noise, and the features of the back photo are mapped to the middle layer of UNet through cross attention; the pure noise is denoised using the information of the back photo to generate a matching X-ray image; cross attention is a mechanism for achieving feature interaction and fusion, which transfers the features of the back photo to UNet in a weighted manner.
8. The method for converting back photographs into spinal X-ray images based on a stable diffusion model according to claim 7, characterized in that: In the step seven, the denoised latent vector is input into the decoder to obtain an X-ray image; the denoised latent vector is restored to the original data, i.e., the X-ray image; the structure of the decoder is opposite to that of the encoder, and it decodes the latent vector into an X-ray image.
9. The method for converting back photographs into spinal X-ray images based on a stable diffusion model according to claim 8, characterized in that: In step eight, the Cobb angle is calculated using the generated X-ray image and compared with the Cobb angle of the real X-ray image to evaluate the effectiveness and accuracy of the method; the generated X-ray image is used to assist in evaluating the severity of spinal diseases such as scoliosis; the Cobb angle is an indicator for measuring the angle of scoliosis, which refers to the angle formed by the upper and lower end plates of the two most inclined vertebrae; the size of the Cobb angle reflects the severity of scoliosis, when the Cobb angle is greater than 10 degrees, it is evaluated as scoliosis, when the Cobb angle is greater than 20 degrees, it is recommended to receive conservative treatment with a brace, and when the Cobb angle is greater than 40 degrees, it is recommended to receive surgical treatment.