An X-ray image registration method and device based on ultrasonic coronal plane images

Abstract

The present invention discloses an X-ray image registration method and device based on ultrasonic coronal plane images, constructs and trains a registration network for spinal ultrasonic coronal plane images and spinal X-rays. The registration network includes a first generator network, a first spatial transformation network, a second generator network, a second spatial transformation network and a discriminator. The first generator network is used to generate a corresponding deformation field according to the input floating image, floating image segmentation mask, reference image and reference image segmentation mask. The first spatial transformation network applies the deformation field to the floating image and its segmentation mask to obtain a registered image and a registered image segmentation mask. Based on the original spinal ultrasonic images and X-ray images, this application realizes the registration of X-ray images, which can not only reduce the time spent by patients, but also reduce the time patients are exposed to radiation.

Description

Technical Field

[0001] The present application relates to the fields of medicine and deep learning technology, and in particular to an X-ray image registration method and device based on ultrasonic coronal images. Background Art

[0002] Adolescent idiopathic scoliosis (AIS) is the most prevalent form of scoliosis and is defined as a three-dimensional torsional deformity of the spine and trunk. Scoliosis primarily develops during adolescence and causes lateral curvature, axial rotation, and sagittal normal curvature, lordosis, and kyphosis. Because of the risk of curvature progression associated with rapid growth in adolescents, adolescents with AIS should be monitored regularly for early detection and further intervention.

[0003] The standard for judging spinal deformity relies on standing X-rays, which show the clear spinal structure in the X-rays to determine whether the spine has deformed. However, repeated X-ray monitoring will lead to cumulative X-ray exposure, which may increase the risk of cancer development, especially for adolescents and children. Compared with X-rays, ultrasound is a radiation-free imaging method that is cost-effective and real-time. Therefore, the use of ultrasound scanning to judge scoliosis has been a research hotspot in recent years.

[0004] In recent years, deep learning technology has accounted for an increasing proportion in research and invention patents. Among them, medical image registration is becoming an active research topic in medical imaging. Based on spinal ultrasound images and X-ray images, realizing X-ray image registration can not only reduce the time spent by patients, but also reduce the time patients are exposed to radiation. It is an important research topic for technicians in this field. Summary of the invention

[0005] The purpose of this application is to provide an X-ray image registration method and device based on ultrasonic coronal images, construct a registration network model, train the registration network, and extract the features of the spinal X-ray image and the spinal ultrasonic coronal image in combination with the corresponding segmentation mask, and generate a deformation field. After passing through a spatial transformation network, the generated deformation field is applied to the spinal X-ray image to obtain a registered image.

[0006] In order to achieve the above purpose, the technical solution of this application is as follows:

[0007] A method for X-ray image registration based on ultrasonic coronal images, comprising:

[0008] The spinal ultrasound coronal images obtained by ultrasound scanning and the spinal X-ray images obtained by standing X-rays were preprocessed to construct a training set;

[0009] Construct and train a registration network for spinal ultrasound coronal plane images and spinal X-rays. The registration network includes a first generator network, a first spatial transformation network, a second generator network, a second spatial transformation network, and a discriminator. The first generator network is used to generate a corresponding deformation field based on the input floating image, floating image segmentation mask, reference image, and reference image segmentation mask. The first spatial transformation network applies the deformation field to the floating image and its segmentation mask to obtain a registered image and a registered image segmentation mask. The second generation network generates the registered image and its segmentation mask into a deformation field and then inputs it into the second spatial transformation network to obtain a cyclic verification image and its segmentation mask;

[0010] Input the spinal ultrasound image to be registered, spinal ultrasound image segmentation mask, spinal X-ray image, and spinal X-ray image segmentation mask into the trained registration network for spinal ultrasound coronal plane images and spinal X-rays, and output the registered image and its segmentation mask.

[0011] Furthermore, the first generator network has the same structure as the second generator network, including: an image encoder, an image decoder, and a skip connection. The image encoder and the image decoder fuse the features of the corresponding number of channels through the skip connection.

[0012] Furthermore, the first spatial transformation network is the same as the second spatial transformation network, including a parameter prediction network, a coordinate mapper, and a pixel collector.

[0013] Furthermore, the discriminator includes a convolutional layer, an activation function layer, and a batch normalization layer.

[0014] Furthermore, the loss function of the registration network for spinal ultrasound coronal plane images and spinal X-rays is:

[0015]

[0016] where: X, Y, X seg , Y seg are the floating image, reference image, floating image segmentation mask, and reference image segmentation mask respectively, G X represents the first generator network, G Y represents the second generator network, D X represents the discriminator, and λ1 is a weight parameter;

[0017]

[0018]

[0019]

[0020] Among them, L MI is the mutual information, and L SSIM is the structural similarity, and L smooth is the regularization term of the deformation field, is the deformation field, is the registered image after deformation, is the segmentation mask of the registered image after deformation; is the cyclic verification image and its segmentation mask.

[0021] The present application also provides an X-ray image registration device based on an ultrasonic coronal plane image, including a processor and a memory storing a number of computer instructions, and when the computer instructions are executed by the processor, the steps of the X-ray image registration method based on the ultrasonic coronal plane image are implemented.

[0022] An X-ray image registration method and device based on an ultrasonic coronal plane image proposed by the present application, relying on deep learning technology and related algorithms of image registration, realizes the registration of X-ray images on the basis of the original spine ultrasonic images and X-ray images, which can not only reduce the time spent by patients, but also reduce the time patients are exposed to radiation, and has practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flowchart of the X-ray image registration method based on the ultrasonic coronal plane image of the present application;

[0024] Figure 2 is a schematic diagram of the spine ultrasonic coronal plane image and the spine X-ray registration network of the present application;

[0025] Figure 3 is a schematic diagram of the generation network structure of the embodiment of the present application;

[0026] Figure 4 is a schematic diagram of the spatial transformation network of the embodiment of the present application;

[0027] Figure 5 is the dice curve of the registration network on the test set during the training process. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] In one embodiment, as Figure 1 shown, an X-ray image registration method based on an ultrasonic coronal plane image of the present application includes:

[0030] Step S1: Preprocess the spinal ultrasound coronal plane images obtained by ultrasound scanning and the spinal X-ray images obtained by standing X-ray photography, and then construct a training set.

[0031] In this embodiment, the spinal ultrasound coronal plane images and X-ray images are taken as examples for illustration, and the image registration of other parts is also applicable.

[0032] In this step, spinal ultrasound image data of scoliosis patients is collected, and their spinal X-ray images are acquired at a certain time after ultrasound detection. A total of 202 AIS patients were recruited (average age: 16.2 ± standard deviation (SD) 3.9 years; BMI: 18.7 ± 3.0 kg / m 2 ). The exclusion criteria were as follows: 1. Patients with metal implants; 2. Patients who had received stent or surgical treatment; 3. Patients with a BMI index higher than 25.0 kg / m 2 . Before the ultrasound examination, all metal items of the subjects were required to be removed to avoid affecting the electromagnetic space equipment. The subjects stood on the platform in the required posture, wearing a gown and exposing their backs to the operator. The operator adjusted the hip board, chest board and four supports to keep the subjects in the required posture. After applying the appropriate ultrasonic gel to the spinal area, the operator held the probe on the back of the subject to set system parameters such as the depth and frequency of the ultrasound. In this study, the scan covered the spinal area from the first thoracic vertebra (T1) to the fifth lumbar vertebra (L5). The probe was placed on T1 and L5 to start the scan. Finally, the operator manipulated the probe to vertically cover the spinal area. During the scan, the ultrasound images and spatial data were captured into the computer. When the probe reached the recorded highest point, the data collection would automatically stop. The whole scan usually took about 30 seconds and provided spinal coronal images through the volume projection imaging (VPI) method.

[0033] All patients received full-spine standing posteroanterior X-ray images by the EOS imaging system ( imaging, Paris, France) within one month after the 3D ultrasound examination.

[0034] Perform preprocessing operations on the obtained spinal ultrasound coronal plane images and spinal X-ray images, including using Photoshop tools to crop the size, scale the images, and perform normalization processing on the spinal ultrasound coronal plane images and spinal X-ray images, etc. A training set of spinal ultrasound images and spinal X-ray images with a pixel size of 256*256 is obtained, and the corresponding segmentation masks are obtained through segmentation.

[0035] Step S2: Construct and train a registration network for spinal ultrasound coronal plane images and spinal X-rays. The registration network includes a first generator network, a first spatial transformation network, a second generator network, a second spatial transformation network, and a discriminator.

[0036] Construct a registration network for spinal ultrasound images and spinal X-ray images, as Figure 2 shown. The registration network includes a first generator network (G X ), a first spatial transformation network (STN1), a second generator network (G Y ), a second spatial transformation network (STN2), and a discriminator (D X ). The first generator network is used to generate a corresponding deformation field according to the input floating image, floating image segmentation mask, reference image, and reference image segmentation mask The first spatial transformation network applies the deformation field to the floating image and its segmentation mask to obtain a registered image and a registered image segmentation mask. In the figure, the segmentation mask is simply referred to as the mask, which will not be elaborated below. The second generation network generates the registered image and its segmentation mask into a deformation field and then inputs it into the second spatial transformation network to obtain a cyclic verification image and its segmentation mask

[0037] It should be noted that in this application, when registering the ultrasound coronal plane image and the X-ray image, the ultrasound coronal plane image can be used as the floating image and the X-ray image as the reference image for registration. It is also possible to use the X-ray image as the floating image and the ultrasound coronal plane image as the reference image for registration. The following takes the X-ray image as the floating image and the ultrasound coronal plane image as the reference image for example.

[0038] In a specific embodiment, the first generator network and the second generator network have the same structure, including: an image encoder, an image decoder, and skip connections. As Figure 3 shown, where:

[0039] The image encoder Encoder includes a convolutional module for feature extraction and downsampling. The convolutional module in the image encoder uses four convolutional layers to perform convolutional operations on the input image. The convolutional kernel size is (3×3); the convolutional stride of the convolutional kernel is 2, the padding is 1, and the number of output channels from the first layer to the fourth layer is 32, 64, 128, and 256 respectively.

[0040] The image decoder Decoder includes a deconvolutional module for upsampling and introducing feature information of the corresponding scale into the upsampling. The deconvolutional module in the image decoder consists of four deconvolutional layers (convolutional kernel (3×3), stride 2, padding 1, output channel 256, 128, 64, 32) and one convolutional layer with a convolutional kernel of (3×3), stride 1, padding 2, output channel 2, and activation function Relu function.

[0041] In this embodiment, the image encoder and the image decoder fuse the characteristics of the corresponding number of channels through skip connections, improving the reusability of features.

[0042] In a specific embodiment, the first spatial transformation network and the second spatial transformation network are the same. As Figure 3 shown, it includes a parameter prediction network (Localisation Net), a coordinate mapper (Grid Generator), and a pixel sampler (Sampler). First, a set of matrix parameters is predicted through the parameter prediction network. Then, the Grid Generator maps the pixel coordinates of the target matrix to the pixel coordinates of the original image. Finally, the sampler determines the calculation method for the values of the target image pixel points.

[0043] In a specific embodiment, the discriminator network includes a convolutional layer, an activation function layer, and a batch normalization layer.

[0044] As a referee, the discriminator compares the pictures generated by the generator with the real images in the dataset, and constrains the generator through the loss function of the discriminator, making the deformation fields generated by the generator more and more accurate.

[0045] Taking the spinal ultrasound image, the spinal ultrasound image segmentation mask, the spinal X-ray image, and the spinal X-ray image segmentation mask obtained in step S1 as inputs with a size of 256 * 256 pixels, the constructed registration network is trained. During the training process, the generator continuously generates deformation fields, and then through the spatial transformation network, the registered images are generated. The registered images and the reference images continuously confront each other in the discriminator until the network model converges, and the trained registration network model is obtained. Finally, end-to-end registration work can be performed through this model.

[0046] During the training process, taking the X-ray image as the floating image and the ultrasound coronal plane image as the reference image as an example, the X-ray image and its segmentation mask, and the ultrasound coronal plane image and its segmentation mask are input into the first generator network to generate a deformation field Then the first spatial transformation network applies the deformation field to the X-ray image and its segmentation mask to obtain the registered image and its segmentation mask. One path of the registered image and its segmentation mask is input into the discriminator to be compared with the ultrasound coronal plane image and its segmentation mask to calculate the adversarial loss. The other path of the registered image and its segmentation mask is input into the second generator network to output a deformation field Then, through the second spatial transformation network, the loop verification image and its segmentation mask are output, and the loop verification loss is calculated.

[0047] The total loss function of the registration network in this embodiment is:

[0048]

[0049] Among them, X, Y, X seg , Y seg are the floating image, the reference image, the floating image segmentation mask, and the reference image segmentation mask respectively, and G X represents the first generator network, and G Y represents the second generator network, and D X represents the discriminator, and λ1 is the weight parameter.

[0050] The loss function includes the registration loss L regist , the cycle consistency loss L cycle , and the adversarial loss L adv .

[0051] The registration loss L regist :

[0052]

[0053] Among them, L MI is the mutual information, L SSIM is the structural similarity, L smooth is the regularization term of the deformation field, is the deformation field, is the registered image after deformation, is the registered image segmentation mask after deformation.

[0054] The cycle consistency loss L cycle :

[0055]

[0056] Among them is the cycle verification image and its segmentation mask.

[0057] The adversarial loss L adv :

[0058]

[0059] By minimizing -log(D(Y, Y·Y seg )) and to achieve the best registration effect.

[0060] After calculating the loss function, use the gradient descent method to optimize the generator and discriminator during training to obtain a trained network model. During the training process, the model is tested, and the similarity of the set (dice coefficient), structural similarity, and Jacobian determinant are used as indicators to test the effect of the registration network, and the model with the best effect is saved.

[0061] Step S3: Input the spine ultrasound image to be registered, the spine ultrasound image segmentation mask, the spine X-ray image, and the spine X-ray image segmentation mask into the trained registration network for the spine ultrasound coronal plane image and the spine X-ray image, and output the registered image and its segmentation mask.

[0062] After training the registration network, input the spine ultrasound image, the spine ultrasound image segmentation mask, the spine X-ray image, and the spine X-ray image segmentation mask into the trained registration network for the spine ultrasound coronal plane image and the spine X-ray image, and output the registered image.

[0063] To verify the effectiveness of the network model, 10 groups of test images (not included in the training set) are selected for verification. Figure 5 It is the mean curve of the dice coefficient on the validation set during the training process. The higher the value of dice, the better the registration effect. As can be seen from Figure 5 it, through the registration network, the dice coefficient has been improved to a certain extent, indicating that the registration network has a good effect.

[0064] In one embodiment, the present application further provides an X-ray image registration device based on an ultrasound coronal plane image, including a processor and a memory storing a number of computer instructions. When the computer instructions are executed by the processor, the steps of the X-ray image registration method based on the ultrasound coronal plane image are implemented.

[0065] For the specific limitations on the X-ray image registration device based on the ultrasound coronal plane image, reference can be made to the limitations on the X-ray image registration method based on the ultrasound coronal plane image in the above text, which will not be elaborated here. The above X-ray image registration device based on the ultrasound coronal plane image can be implemented in whole or in part by software, hardware, and their combination. It can be embedded in the processor of the computer device in hardware form or be independent of it, or be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations above.

[0066] The memory and the processor are directly or indirectly electrically connected to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The memory stores a computer program that can run on the processor. The processor realizes the network topology layout method in the embodiments of the present invention by running the computer program stored in the memory.

[0067] Among them, the memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electric Erasable Programmable Read-Only Memory (EEPROM), etc. Among them, the memory is used to store programs, and the processor executes the programs after receiving execution instructions.

[0068] The processor may be an integrated circuit chip with data processing capabilities. The above-mentioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0069] The above embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. An X-ray image registration method based on an ultrasonic coronal plane image, characterized in that, The X-ray image registration method based on ultrasonic coronal plane images includes: Preprocessing the spinal ultrasonic coronal plane images obtained by ultrasonic scanning and the spinal X-ray images obtained by standing X-ray photography, and constructing a training set; Construct and train a registration network for spinal ultrasound coronal plane images and spinal X-rays. The registration network includes a first generator network, a first spatial transformation network, a second generator network, a second spatial transformation network, and a discriminator. The first generator network is used to generate a corresponding deformation field according to the input floating image, floating image segmentation mask, reference image, and reference image segmentation mask. The first spatial transformation network applies the deformation field to the floating image and its segmentation mask to obtain a registered image and a registered image segmentation mask. The second generation network generates the registered image and its segmentation mask into a deformation field and then inputs it into the second spatial transformation network to obtain a loop-verified image and its segmentation mask. Inputting the spinal ultrasonic image to be registered, the spinal ultrasonic image segmentation mask, the spinal X-ray image and the spinal X-ray image segmentation mask into the trained registration network for spinal ultrasonic coronal plane images and spinal X-rays, and outputting the registered image and its segmentation mask; Among them, the loss function of the registration network for spinal ultrasonic coronal plane images and spinal X-rays is: Where: X, Y, X seg , Y seg are the floating image, the reference image, the floating image segmentation mask, and the reference image segmentation mask respectively, and G X represents the first generator network, and G Y represents the second generator network, and D X represents the discriminator, and λ1 is the weight parameter; Among them, L MI is the mutual information, L SSIM is the structural similarity, L smooth is the regularization term of the deformation field, is the deformation field, is the registered image after deformation, is the segmentation mask of the registered image after deformation; is the loop verification image and its segmentation mask.

2. The X-ray image registration method based on the ultrasonic coronal plane image according to claim 1, characterized in that, The first generator network and the second generator network have the same structure, including: an image encoder, an image decoder and skip connections, and the image encoder and the image decoder fuse the features of corresponding channel numbers through skip connections.

3. The X-ray image registration method based on the ultrasonic coronal plane image according to claim 1, characterized in that, The first spatial transformation network and the second spatial transformation network are the same, including a parameter prediction network, a coordinate mapper and a pixel collector.

4. The X-ray image registration method based on the ultrasonic coronal plane image according to claim 1, characterized in that, The discriminator includes a convolutional layer, an activation function layer and a batch normalization layer.

5. An X-ray image registration device based on an ultrasonic coronal plane image, comprising a processor and a memory storing a number of computer instructions, characterized in that, When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 4 are implemented.

Images