Method and apparatus for generating non-radiative X-ray images based on ultrasonic coronal plane images

Abstract

The present invention discloses a method and device for generating non-radiative X-ray images based on ultrasonic coronal plane images. A generation attention network for spinal ultrasonic images and spinal X-ray images is constructed and trained. The generation attention network for spinal ultrasonic images and spinal X-ray images includes a first generator network, a second generator network, a first discriminator, and a second discriminator. The first generator network and the second generator network have the same structure, including an image encoder, an image decoder, and an attention mechanism module. The spinal ultrasonic coronal plane image obtained by ultrasonic scanning is input into the trained generation attention network for spinal ultrasonic images and spinal X-ray images to generate a corresponding spinal X-ray image. The present invention realizes imaging in the corresponding spinal X-ray state in a non-radiative manner and can be used in the regular clinical screening of adolescent idiopathic scoliosis, thereby avoiding the cumulative influence of X-ray radiation on the growth and development of adolescents.

Description

Technical Field

[0001] The present application belongs to the fields of medicine and deep learning technology, and specifically relates to a method and device for generating radiation-free X-ray images based on ultrasonic coronal images. Background Art

[0002] Adolescent idiopathic scoliosis (AIS) is the most prevalent form of scoliosis and is defined as a three-dimensional torsional deformity of the spine and trunk. Scoliosis primarily develops during adolescence and causes lateral curvature, axial rotation, and sagittal normal curvature, lordosis, and kyphosis. Because of the risk of curvature progression associated with rapid growth in adolescents, adolescents with AIS should be monitored regularly for early detection and further intervention.

[0003] The gold standard for judging spinal deformity relies on standing X-rays, which show clear spinal structures to determine whether the spine has deformed. However, repeated X-ray monitoring will lead to cumulative X-ray exposure, which may increase the risk of cancer development, especially for adolescents and children. Compared with X-rays, ultrasound is a radiation-free imaging method that is cost-effective and real-time. Therefore, using ultrasound scanning to judge scoliosis has been a research hotspot in recent years. Unfortunately, the advantage of ultrasound imaging lies in the soft tissue structure. The bone component has a strong attenuation and scattering effect on ultrasound. Therefore, ultrasound bone images have the following characteristics: 1) less image information. Since ultrasound cannot penetrate the bone structure, only the linear strong echo on the bone surface can be clearly shown in the image; 2) more image noise. Ultrasound scanning of bone structure is prone to diffraction and noise, resulting in blurred, burr-like and unclear content in the image. Therefore, the application of ultrasound scanning for bone structure imaging is relatively limited, and its structural problems are particularly prominent in spinal images. The human spine is a long and narrow structure composed of multiple irregular bony vertebral structures. Each vertebra contains many anatomical structures such as spinous processes, transverse processes, articular processes, lamina, interlaminar spaces, and intervertebral foramina. These characteristics lead to challenges in the clarity, recognition, and accuracy of ultrasound images of spinal bone structures, and X-ray imaging makes up for this deficiency.

[0004] In recent years' research and invention patents, deep neural networks have been used for computer vision and medical imaging tasks. Among them, medical image synthesis is becoming an active research topic in medical imaging. Many existing works focus on synthesis across imaging modalities. For example, there are studies on mapping MRI scans to their CT modality, and also studies on mapping certain MRIs to other MRI modalities. In the algorithms of image synthesis, generative adversarial networks (GANs) are an emerging deep learning technology for modeling high-dimensional data distributions and have been widely used in natural image processing and medical image analysis. Least Squares GAN (LSGAN) can generate high-quality images through the discriminator, and the loss is smoother and unsaturated. Image-to-image translation is one of the important applications of GANs. In the absence of paired training data, CycleGAN first proposed the cyclic consistency loss to implement one-to-one mapping. MUNIT synthesizes the separated content and style to generate the final image, and improves the image quality by using adaptive instance normalization. Therefore, relying on deep learning technology and related algorithms of image synthesis, it has practical application significance to achieve imaging in the state of spinal X-ray without radiation based on the original spinal ultrasound image. Summary of the Invention

[0005] The purpose of this application is to propose a method and device for generating radiation-free X-ray images based on ultrasonic coronal plane images, and to use an original generation attention network model to train a generation network to retain the detailed features of spinal ultrasound and X-ray images to the greatest extent, so as to generate corresponding spinal X-ray images through spinal ultrasound coronal plane images without radiation.

[0006] To achieve the above purpose, the technical solution of this application is as follows:

[0007] A method for generating radiation-free X-ray images based on ultrasonic coronal plane images, comprising:

[0008] Preprocess the spinal ultrasound coronal plane images obtained by ultrasonic scanning and the spinal X-ray images obtained by standing X-ray photography, and construct a training set. The training samples in the training set include spinal ultrasound coronal plane images and spinal X-ray images;

[0009] Construct and train a generation attention network for spinal ultrasound images and spinal X-ray images. The generation attention network for spinal ultrasound images and spinal X-ray images includes a first generator network, a second generator network, a first discriminator, and a second discriminator. The first generator network generates a corresponding synthetic X-ray image from the input spinal ultrasound coronal plane image, and then sends the synthetic X-ray image and the input spinal X-ray image to the first discriminator for comparison. The synthetic X-ray image is also input into the second generator network to generate a corresponding cyclic verification ultrasound image. The second generator network also generates a corresponding synthetic ultrasound image from the input spinal X-ray image, and then sends the synthetic ultrasound image and the input spinal ultrasound coronal plane image to the second discriminator for comparison. The synthetic ultrasound image is also input into the first generator network to generate a corresponding cyclic verification X-ray image.

[0010] Input the spinal ultrasound coronal plane image obtained by ultrasound scanning into the trained generation attention network for spinal ultrasound images and spinal X-ray images to generate a corresponding spinal X-ray image.

[0011] Furthermore, the first generator network and the second generator network have the same structure, including: an image encoder, an image decoder, and an attention mechanism module.

[0012] Furthermore, the image encoder includes a downsampling module and a residual block, and the image decoder includes a residual block and an upsampling module.

[0013] Furthermore, the attention mechanism module inputs the image features extracted by the image encoder into the image decoder for upsampling to obtain a score-cam image, multiplies it with the input image to obtain an input image mask, inputs it into the image encoder to obtain masked features, performs weighted summation of the masked features and the image features to obtain attention features with attention distribution, and then inputs the attention features into the image decoder to obtain a synthetic output.

[0014] Furthermore, the first discriminator and the second discriminator have the same structure, including four convolutional blocks connected in sequence, and each convolutional block includes a convolutional layer, a batch normalization layer, and an activation function layer.

[0015] This application also proposes a radiation-free X-ray image generation device based on an ultrasound coronal plane image, including a processor and a memory storing a number of computer instructions. When the computer instructions are executed by the processor, the steps of the radiation-free X-ray image generation method based on an ultrasound coronal plane image are implemented.

[0016] A radiation-free X-ray image generation method and device based on an ultrasound coronal plane image proposed by this application have the following technical effects:

[0017] Clinically acquired spinal ultrasound images are used to achieve imaging corresponding to the state of spinal X-rays in a non-radiative manner, which can be used in the regular clinical screening of adolescent idiopathic scoliosis, thus avoiding the cumulative radiation of X-rays on adolescents and affecting their growth and development. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flowchart of a method for generating non-radiative X-ray images based on ultrasonic coronal plane images in this application;

[0019] Figure 2 This is a schematic diagram of the attention network structure for generating spinal ultrasound images and spinal X-ray images in this application;

[0020] Figure 3 This is a schematic diagram of the generator network structure in an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] In order to make the objectives, technical solutions and advantages of this application clearer, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0022] In one embodiment, as Figure 1 shown, a method for generating non-radiative X-ray images based on ultrasonic coronal plane images is proposed, including:

[0023] Step S1: Preprocess the spinal ultrasound coronal plane images obtained by ultrasonic scanning and the spinal X-ray images obtained by standing X-ray photography, and construct a training set. The training samples in the training set include spinal ultrasound coronal plane images and spinal X-ray images.

[0024] This embodiment takes the ultrasonic coronal plane images and X-ray images of the spine as examples for illustration, and is also applicable to the generation of X-ray images of other parts.

[0025] In this step, spinal ultrasound image data of scoliosis patients is collected, and their spinal X-ray images are collected after a certain period of ultrasonic detection. A total of 202 AIS patients were recruited (average age: 16.2 ± standard deviation (SD) 3.9 years; BMI: 18.7 ± 3.0 kg / m 2 )). The exclusion criteria are: 1. Patients with metal implants; 2. Patients who have received stent or surgical treatment; 3. Patients with a BMI index higher than 25.0 kg / m 2Patients. Before the ultrasound examination, all metal objects of the subjects were required to be removed to avoid affecting the electromagnetic space equipment. The subjects stood on the platform in the required posture, wearing a gown and exposing their backs to the operator. The operator adjusted the buttock plate, chest plate and four supports to keep the subjects in the required posture. After applying the appropriate ultrasonic gel to the spinal area, the operator held the probe on the subject's back to set system parameters such as the depth and frequency of the ultrasound. In this study, the scan covered the spinal area from the first thoracic vertebra (T1) to the fifth lumbar vertebra (L5). The probe was placed on T1 and L5 to start the scan. Finally, the operator manipulated the probe to vertically cover the spinal area. During the scan, ultrasonic images and spatial data were captured into the computer. When the probe reached the recorded highest point, the data collection would automatically stop. The whole scan usually took about 30 seconds and provided spinal coronal images through the volume projection imaging (VPI) method. All patients received full-spine standing posteroanterior X-ray images from the EOS imaging system ( Imaging, Paris, France) within one month after the 3D ultrasound examination.

[0026] Preprocessing operations were performed on the obtained spinal ultrasound coronal images and spinal X-ray images, including cropping, scaling and normalizing the spinal ultrasound coronal images and spinal X-ray images using Photoshop tools. A training set of spinal ultrasound images and spinal X-ray images with a pixel size of 256*256 was obtained.

[0027] Step S2: Construct and train a generative attention network for spinal ultrasound images and spinal X-ray images. The generative attention network for spinal ultrasound images and spinal X-ray images includes a first generator network, a second generator network, a first discriminator and a second discriminator.

[0028] In this step, a generative attention network for spinal ultrasound images and spinal X-ray images was constructed, as Figure 2As shown in the figure, the network mainly consists of two parts. One is the generator for generating images. According to the input image, it generates images of corresponding modalities, including the first generator network (generator u), which generates the input ultrasound image into the corresponding spinal X-ray image. At the same time, a second generator (generator r) is set up to regenerate the input X-ray image back into an ultrasound image. Based on the discrimination and verification of the target X-ray image, and then through the comparison and verification of the backtracked ultrasound image, the learning ability of the generator for sample images is improved to obtain a better X-ray image generation effect. The other is the discriminator network. As a referee, the discriminator compares the images generated by the generator with the real images in the dataset. The discriminator first takes the real image in the training sample as a benchmark sample, and its function setting value is 1. Each time the corresponding image generated by the generator is input into the discriminator for comparison with the real sample. The discriminator uses the corresponding loss function to score and assign values to the generated image between 0 and 1. Different differences will be returned to the generator to guide the next generation effect, so that the generator generates pictures identical to the real images. The discriminator includes the first discriminator (discriminator r) and the second discriminator (discriminator u). The first discriminator is used to compare the synthesized X-ray image with the input X-ray image, and the second discriminator is used to compare the synthesized ultrasound image with the input ultrasound image.

[0029] Specifically, as Figure 2 shown, the first generator network generates the input spinal ultrasound coronal plane image into the corresponding synthesized X-ray image, and then sends the synthesized X-ray image and the input spinal X-ray image to the first discriminator for comparison; the synthesized X-ray image is also input into the second generator network to generate the corresponding loop verification ultrasound image. The second generator network also generates the input spinal X-ray image into the corresponding synthesized ultrasound image, and then sends the synthesized ultrasound image and the input spinal ultrasound coronal plane image to the second discriminator for comparison; the synthesized ultrasound image is also input into the first generator network to generate the corresponding loop verification X-ray image.

[0030] In a specific embodiment, the first generator network and the second generator network have the same structure. As Figure 3 shown, it includes: an image encoder, an image decoder, and an attention mechanism module Score-CAM.

[0031] The image encoder Encoder includes a downsampling module and a residual block, which perform downsampling on the input image to enhance feature extraction. The downsampling module in the image encoder uses three layers of Instance Normalization-ReLU convolutional layers to perform convolution operations on the input image. The kernel sizes of the convolutional layers are (7×7), (3×3), and (3×3) respectively. Among them, the convolutional stride of the (7×7) kernel is 1, the padding is 3, and the output channels are 64. The convolutional stride of the (3×3) kernel is 2, the padding is 1, and the output channels are 128. The convolutional stride of the third layer (3×3) kernel is 2, the padding is 1, and the output channels are 256. The residual block is composed of four layers of Instance Normalization-ReLU residual layers, with a convolutional kernel of (3×3), a stride of 1, a padding of 1, and output channels of 256.

[0032] The image decoder Decoder includes a residual block and an upsampling module for feature extraction and upsampling. The residual block in the image decoder is composed of four layers of Adaptive-Instance Normalization-ReLU residuals, with a convolutional kernel of (3×3), a stride of 1, a padding of 1, and output channels of 256. The upsampling module is composed of two layers of InstancelayerNormalization-ReLU transposed convolutional layers (with a convolutional kernel of (3×3), a stride of 1, a padding of 1, and output channels of 128, 64) and a convolutional layer with a convolutional kernel of (7×7, a stride of 1, a padding of 3, and output channels of 3), and the activation function is the Tanh function.

[0033] The attention mechanism module Score-CAM inputs the image features extracted by the encoder into the decoder for upsampling to obtain the score-cam image, multiplies it pointwise with the input image to obtain the input image mask, inputs it into the encoder to obtain the mask feature (mask feature map), performs weighted summation with the original image features to obtain the attention features with attention distribution, and then inputs them into the decoder to obtain the synthetic output.

[0034] In this embodiment, the attention mechanism of Score-CAM is added to the generator. During the process of generating images, the attention to the details and textures of the images is enhanced to obtain more realistic and detailed generated images.

[0035] Use the spinal ultrasound images obtained in step 1 as input with a size of 256*256 pixels to train the constructed generative attention network. During the training process, the generator continuously generates realistic Fake image patches, and the Fake image patches and the real X-ray image patches in the training set are continuously pitted against each other in the discriminator. Eventually, until the network model converges and the discriminator can no longer distinguish between Fake-X-ray images and real X-ray images, the trained generative attention network model UXGAN is obtained. This model can convert the input spinal ultrasound images into corresponding spinal X-ray images with a size of 256*256 pixels.

[0036] In a specific embodiment, the first discriminator and the second discriminator have the same structure. The discriminator includes four convolutional blocks connected in sequence, and each convolutional block includes a convolutional layer, a batch normalization layer, and an activation function layer. The convolutional kernels of the discriminator all have a size of (3×3), strides of 2, 2, 2, 1, and padding of 1, and the output channels are 64, 128, 256, 512.

[0037] During training, each training sample includes a spinal ultrasound coronal plane image and the corresponding spinal X-ray image. The spinal ultrasound coronal plane image is input into the first generator network to generate the corresponding synthetic X-ray image, and then the synthetic X-ray image and the input spinal X-ray image are sent to the first discriminator for comparison to calculate the loss; the synthetic X-ray image is also input into the second generator network to generate the corresponding cycle verification ultrasound image, which is compared with the input spinal ultrasound coronal plane image to calculate the corresponding loss. The spinal X-ray image is also input into the second generator network to generate the corresponding synthetic ultrasound image, and then the synthetic ultrasound image and the input spinal ultrasound coronal plane image are sent to the second discriminator for comparison to calculate the loss; the synthetic ultrasound image is also input into the first generator network to generate the corresponding cycle verification X-ray image, which is compared with the input spinal X-ray image to calculate the corresponding loss. After calculating the loss, the gradient descent method is used to optimize the parameters of the model. The training of neural network models is a relatively mature technology in the art and will not be elaborated here.

[0038] In a specific embodiment, the loss function of the network model of the present application is:

[0039]

[0040] where L lsgan is the least squares GAN loss, L cycle is the cycle consistency loss, L cam is the attention loss L cam .

[0041] The meaning of min and max under this condition is that, first, find the discriminator that maximizes the loss (the best discrimination effect), and second, find the generator that minimizes the loss under this condition.

[0042] Among them, G u→r , G r→u represents generator u and generator r, and η u , η r represents the attention features of generator u and generator r.

[0043] Among them, D u , D r represents discriminator u and discriminator r, represents the attention feature data corresponding to the judgment results of discriminator u and discriminator r.

[0044] The least squares GAN loss L lsgan :

[0045]

[0046] Among them represents the mean of generator r, G u→r (x) represents the synthetic X-ray image data generated by generator u, and D r (·) represents the judgment value of discriminator r on the synthetic X-ray image.

[0047] The cycle consistency loss L cycle :

[0048]

[0049] Among them, G r→u (x) represents the cycle verification ultrasound image data generated by generator r, and ||·||1 represents the Euclidean norm.

[0050] The attention loss L cam :

[0051]

[0052] Among them and represent the means of generator u and generator r, and η u (x) represents the attention feature data.

[0053] Among them, λ1, λ2, and λ3 are weight coefficients. In a preferred embodiment, λ1 = 1, λ2 = 10, and λ3 = 100 in the loss function.

[0054] After training, a number of sets of training parameters are obtained; the training parameters are loaded into the generator, and the peak signal-to-noise ratio, structural similarity, and normalized mean squared error value are used as image metrics to test the image generation quality of the trained generator. The training parameters with the best metric evaluation are selected and loaded into the generator as the final generator model.

[0055] The method for fine-tuning the model is as follows: set the learning rate to 1e-4, set the batch-size to 1, there are a total of 3321 image patches, train for 10 epochs, and use the Adam optimizer for iterative training, where the exponential decay rates of the Adam optimizer are β1 = 0.9 and β2 = 0.99.

[0056] Step S3: Input the spinal ultrasound coronal plane image obtained by ultrasound scanning into the trained generation attention network of spinal ultrasound images and spinal X-ray images to generate the corresponding spinal X-ray image.

[0057] After training the generation attention network of spinal ultrasound images and spinal X-ray images, it can be put into actual application. During application, the spinal ultrasound coronal plane image obtained by ultrasound scanning is input into the trained generation attention network of spinal ultrasound images and spinal X-ray images, and the first generator network outputs the corresponding synthetic X-ray image, which is used as the spinal X-ray image.

[0058] After the network training of this application is completed, the best model parameters are selected to verify the trained spinal X-ray image generator. In order to verify the effectiveness of the network model, 20 ultrasound images (not included in the training set) are selected for verification. Table 1 shows the results of averaging the objective evaluation metrics of PSNR (peak signal-to-noise ratio), SSIM (structural similarity), and NMSE (normalized mean squared error) for the verification set composed of the selected 20 images. The obtained results are compared with methods such as CycleGAN. It can be seen from Table 1 that this application also achieves good results in objective evaluation.

[0059]

[0060]

[0061] Table 1

[0062] As can be observed from Table 1, compared with the traditional CycleGAN method, the average PSNR of the UXGAN in the technical solution of this application is 6.56 higher than that of the CycleGAN method. The higher the value, the higher the image quality. The NMSE and SSIM represent the error index of the image in pixel values and the index of image structure change respectively. The lower the NMSE value, the smaller the error in pixel values, and the higher the SSIM value, the better the image structure. It can be seen from Table 1 that the NMSE and SSIM of UXGAN are 0.092 less and 0.086 more respectively than the corresponding evaluation results of CycleGAN.

[0063] In another embodiment, this application also provides a non-radiative X-ray image generation device based on ultrasonic coronal plane images, including a processor and a memory storing a number of computer instructions. When the computer instructions are executed by the processor, the steps of the non-radiative X-ray image generation method based on ultrasonic coronal plane images are implemented.

[0064] For the specific limitations of the non-radiative X-ray image generation device based on ultrasonic coronal plane images, reference can be made to the limitations of the non-radiative X-ray image generation method based on ultrasonic coronal plane images in the above text, which will not be elaborated here. The above non-radiative X-ray image generation device based on ultrasonic coronal plane images can be implemented in whole or in part through software, hardware, and their combination. It can be embedded in the processor of a computer device in hardware form or be independent of it, or be stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations above.

[0065] The memory and the processor are directly or indirectly electrically connected to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The memory stores a computer program that can run on the processor. The processor realizes the network topology layout method in the embodiments of the present invention by running the computer program stored in the memory.

[0066] Among them, the memory may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electric Erasable Programmable Read-Only Memory (EEPROM), etc. Among them, the memory is used to store a program, and the processor executes the program after receiving an execution instruction.

[0067] The processor may be an integrated circuit chip with data processing capabilities. The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0068] The above embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several deformations and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A method for generating a non-radiative X-ray image based on an ultrasonic coronal plane image, characterized in that, The method for generating a non-radiative X-ray image based on an ultrasonic coronal plane image includes: Preprocessing the spinal ultrasonic coronal plane image obtained by ultrasonic scanning and the spinal X-ray image obtained by standing X-ray photography, and constructing a training set. The training samples in the training set include the spinal ultrasonic coronal plane image and the spinal X-ray image; Constructing and training a generation attention network for spinal ultrasonic images and spinal X-ray images. The generation attention network for spinal ultrasonic images and spinal X-ray images includes a first generator network, a second generator network, a first discriminator, and a second discriminator. The first generator network generates a corresponding synthetic X-ray image from the input spinal ultrasonic coronal plane image, and then sends the synthetic X-ray image and the input spinal X-ray image to the first discriminator for comparison; the synthetic X-ray image is also input into the second generator network to generate a corresponding cyclic verification ultrasonic image; the second generator network also generates a corresponding synthetic ultrasonic image from the input spinal X-ray image, and then sends the synthetic ultrasonic image and the input spinal ultrasonic coronal plane image to the second discriminator for comparison; the synthetic ultrasonic image is also input into the first generator network to generate a corresponding cyclic verification X-ray image; Inputting the spinal ultrasonic coronal plane image obtained by ultrasonic scanning into the trained generation attention network for spinal ultrasonic images and spinal X-ray images to generate a corresponding spinal X-ray image; Among them, the first generator network and the second generator network have the same structure, including: an image encoder, an image decoder, and an attention mechanism module; The image encoder includes a downsampling module and a residual block, and the image decoder includes a residual block and an upsampling module; The attention mechanism module inputs the image features extracted by the image encoder into the image decoder for upsampling to obtain a score-cam image, multiplies it with the input image to obtain an input image mask map, inputs it into the image encoder to obtain a mask feature, performs weighted summation of the mask feature and the image feature to obtain an attention feature with an attention distribution, and then inputs the attention feature into the image decoder to obtain a synthetic output; The first discriminator and the second discriminator have the same structure, including four convolutional blocks connected in sequence, and each convolutional block includes a convolutional layer, a batch normalization layer, and an activation function layer.

2. A non-radiative X-ray image generation device based on ultrasonic coronal plane images, comprising a processor and a memory storing a number of computer instructions, characterized in that, When the computer instructions are executed by a processor, the steps of the method described in claim 1 are implemented.

Images