Liver three-dimensional ultrasound and CT image registration initialization method, device, and electronic device

Abstract

The present invention discloses a liver three-dimensional ultrasound and CT image registration initialization method, device, and electronic device, wherein the registration initialization method includes: training to obtain a 3D neural network CT segmentation network model and a 3D neural network ultrasound segmentation network model; inputting a liver CT test image into the 3D neural network CT segmentation network model to obtain a CT segmentation image; extracting its vascular part, calculating the vascular branch points and branch planes; inputting a real-time liver three-dimensional ultrasound image into the 3D neural network ultrasound segmentation network model to obtain an ultrasound segmentation image; extracting its vascular part, calculating the vascular branch points and branch planes; transforming the two branch planes; calculating the matching measurement score of each pair of transformed images; finding the plane pair with the highest score, calculating the initialization transformation matrix, and completing the image registration initialization. The present invention can realize the automatic registration initialization of liver 3D ultrasound and CT images, and the image does not need any manual setting parameters, and the automatic registration success rate is high and the registration speed is fast.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing, and in particular relates to a liver three-dimensional ultrasound and CT image registration initialization method, device, and electronic equipment. Background Art

[0002] Liver cancer is a major disease in my country. According to the "China Cancer Registry Annual Report 2018", the incidence of liver cancer ranks third and the mortality rate ranks second.

[0003] For early small hepatocellular carcinoma, interventional therapy is the preferred minimally invasive treatment method, and computed tomography (CT) images have become one of the important routine means in preoperative clinical diagnosis and treatment.

[0004] During interventional surgery, real-time ultrasound images are often used as a guide for interventional puncture. However, the disadvantage of ultrasound images is that they are two-dimensional images with a very limited field of view, and are easily blocked by fat ribs and the airflow in the body, resulting in uncertain imaging quality and uncertain tumor display. 3D ultrasound can more completely display the contours of the tumor and the information of the surrounding blood vessels, but doctors often find it difficult to judge the overall distribution of blood vessels, which can easily cause vascular damage during ablation.

[0005] Therefore, the fusion of CT images and 3D ultrasound images can utilize the information of both modalities and improve the success rate of surgery. Due to the uncertainty of ultrasound scanning angle, the initialization of 3D ultrasound and CT image registration is the biggest challenge.

[0006] Real-time CT ultrasound fusion technology is divided into tracking-based navigation system and image-based registration:

[0007] Commercial electromagnetic navigation or optical navigation systems require electromagnetic sensing devices and optical positioning systems to attach external markers to the patient during CT scanning.

[0008] The non-tracking image registration method is based on the vascular information of the ultrasound image. It uses the geometric relationship between the vascular branches to define the similarity, and uses the Viterbi shortest path algorithm to find the corresponding points of the vascular branches, or uses the branch planes of the blood vessels to transform the CT and ultrasound images to their respective branch planes. After the branch points are aligned, the similarity of the corresponding plane images is calculated to find the best branch plane pair, thereby calculating the initial transformation.

[0009] In summary, many commercial tracking-based navigations such as electromagnetic tracking technology can be aligned in real time, but they require markers, and electromagnetic sensor devices are easily affected by metal and unstable. Most of the current image-based technologies focus on the precise alignment stage after initialization. Only a small number of automatic initialization methods use vascular information and image grayscale information as image similarity measures, which require a large number of manual parameter settings and are greatly affected by the parameters, resulting in a low success rate for initialization alignment technology. Summary of the invention

[0010] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and provide a method, device, and electronic device for initializing the registration of liver three-dimensional ultrasound and CT images, which can realize automatic initialization of the registration of liver 3D ultrasound and CT images without any manual setting of parameters, with a high automatic registration success rate and a fast registration speed.

[0011] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0012] A liver three-dimensional ultrasound and CT image registration initialization method, which is characterized by including:

[0013] Train and obtain a 3D neural network CT segmentation network model;

[0014] Train and obtain a 3D neural network ultrasound segmentation network model;

[0015] Input the liver CT test image into the trained 3D neural network CT segmentation network model for segmentation to obtain a CT segmentation image;

[0016] Extract the blood vessel part in the CT segmentation image;

[0017] Calculate the vascular branch points and branch planes based on the vascular parts in the CT segmentation image;

[0018] Inputting the real-time three-dimensional liver ultrasound image into the trained 3D neural network ultrasound segmentation network model for segmentation to obtain an ultrasound segmentation image;

[0019] Extracting blood vessel parts from ultrasound segmentation images;

[0020] Calculate the vascular branch points and branch planes based on the vascular portion in the ultrasound segmentation image;

[0021] Transforming the branch plane obtained based on the ultrasound segmentation image and the branch plane obtained based on the CT segmentation image;

[0022] Calculate the matching measure score of each pair of transformed ultrasound segmentation images and CT segmentation images;

[0023] Find the plane pair with the highest matching measure score, calculate the initialization transformation matrix, and complete the initialization of liver 3D ultrasound and CT image registration.

[0024] As a preferred method, the 3D neural network CT segmentation network model training process includes:

[0025] Obtain liver CT training images and preprocess them;

[0026] The preprocessed liver CT training image is cropped to a set size and input into the 3D neural network CT segmentation network for training and the trained 3D neural network CT segmentation network model is saved.

[0027] As a preferred method, the 3D neural network ultrasound segmentation network model training process includes:

[0028] Obtaining liver three-dimensional ultrasound training images and preprocessing them;

[0029] The preprocessed liver ultrasound training image is cropped to a set size and then input into the 3D neural network ultrasound segmentation network for training and the trained 3D neural network ultrasound segmentation network model is saved.

[0030] As a preferred method, before inputting the liver CT test image into the trained 3D neural network CT segmentation network model, the following steps are also included:

[0031] Acquire liver CT test images and preprocess them;

[0032] The preprocessed liver CT test image is cropped to a set size and then input into the trained 3D neural network CT segmentation network model.

[0033] As a preferred method, before inputting the real-time liver three-dimensional ultrasound image into the trained 3D neural network ultrasound segmentation network model, the method further includes:

[0034] Preprocessing the acquired real-time three-dimensional liver ultrasound images;

[0035] The preprocessed real-time liver three-dimensional ultrasound image is cropped to a set size and then input into the trained 3D neural network ultrasound segmentation network model.

[0036] As a preferred embodiment, the pretreatment process includes:

[0037] Calculate the spatial spacing of all images to be preprocessed, select the median of all spatial spacings and resample all images to be preprocessed to the spatial spacing corresponding to the median;

[0038] Calculate the grayscale mean and variance of the resampled image and normalize the grayscale.

[0039] As a preferred method, the blood vessel part in the CT segmentation image is extracted, and the skeleton line is extracted using a blood vessel thinning filter.

[0040] As a preferred method, the blood vessel part in the ultrasonic segmentation image is extracted, and the skeleton line is extracted using a blood vessel thinning filter.

[0041] Based on the same inventive concept, the present invention also provides a liver three-dimensional ultrasound and CT image registration initialization device, which is characterized by comprising:

[0042] The first training module: used to train and obtain a 3D neural network CT segmentation network model;

[0043] The second training module: used to train and obtain a 3D neural network ultrasound segmentation network model;

[0044] CT segmentation image acquisition module: used to input the liver CT test image into the trained 3D neural network CT segmentation network model for segmentation to obtain a CT segmentation image;

[0045] The first extraction module: used to extract the blood vessel part in the CT segmentation image;

[0046] The first calculation module is used to calculate the blood vessel branch points and branch planes based on the blood vessel part in the CT segmentation image;

[0047] Ultrasound segmentation image acquisition module: used to input the real-time liver three-dimensional ultrasound image into the trained 3D neural network ultrasound segmentation network model for segmentation to obtain the ultrasound segmentation image;

[0048] The second extraction module is used to extract the blood vessel part in the ultrasonic segmentation image;

[0049] The second calculation module is used for transforming the branch plane obtained based on the ultrasound segmentation image and the branch plane obtained based on the CT segmentation image;

[0050] The third calculation module is used to calculate the matching measurement score of each pair of transformed ultrasound segmentation images and CT segmentation images;

[0051] The fourth calculation module is used to calculate the initialization transformation matrix based on the plane pair with the highest matching measurement score, and complete the initialization of liver three-dimensional ultrasound and CT image registration.

[0052] Based on the same inventive concept, the present invention further provides an electronic device, characterized in that the electronic device comprises:

[0053] one or more processors;

[0054] a storage device for storing one or more programs,

[0055] When the one or more programs are executed by the one or more processors, the one or more processors implement the liver three-dimensional ultrasound and CT image registration initialization method.

[0056] Compared with the prior art, the present invention firstly segments the gallbladder and blood vessels of preoperative CT and intraoperative ultrasound based on a 3D neural network (U-Net), then calculates the bifurcation points and branch plane spaces of the blood vessels, and for any set of bifurcation plane pairs of CT-US, transforms the images into their respective branch plane spaces and aligns them at the bifurcation points, then calculates the matching score measure based on the gallbladder and blood vessel segmentation, selects the plane pair with the largest score, obtains the initialization transformation matrix, completes automatic initialization, and realizes automatic registration initialization of liver 3D ultrasound and CT images, without any manual setting of parameters, with high automatic registration success rate and fast registration speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a technical flow chart of the present invention.

[0058] Figure 2 Grayscale image of the quantitative and visual results of the segmentation of the gallbladder and blood vessels in ultrasound images.

[0059] Figure 3 Grayscale images of the ultrasound image (first row), the registered CT image (second row), and the overlay of the two (third row). DETAILED DESCRIPTION

[0060] The technical problem to be solved by the present invention is to initialize the automatic registration of liver 3D ultrasound and CT.

[0061] The difficulty of multimodal image registration is the definition of the similarity measurement function, while the automatic segmentation technology based on deep convolutional neural networks is relatively mature, and the segmentation of weak labels as a measurement function can solve this problem. The present invention proposes to use deep learning to segment multiple organs of ultrasound, and proposes a similarity measurement function suitable for local ultrasound and CT based on the segmented image, which does not require any parameters and can achieve fast and robust registration of liver 3D ultrasound and CT.

[0062] like Figure 1 As shown, the liver three-dimensional ultrasound and CT image registration initialization method includes:

[0063] Step 1: Obtain liver CT training images and preprocess them.

[0064] The pretreatment process includes:

[0065] Calculate the spatial spacing of all images to be preprocessed, select the median of all spatial spacings and resample all images to be preprocessed to the spatial spacing corresponding to the median;

[0066] Calculate the grayscale mean and variance of the resampled image and normalize the grayscale.

[0067] Step 2: The preprocessed liver CT training image is cropped to a set size (128×160×180) and then input into a 3D neural network CT segmentation network (3D U-Net gallbladder and blood vessel segmentation network) for training and the trained 3D neural network CT segmentation network model is saved.

[0068] Step 3, obtaining a liver three-dimensional ultrasound training image and preprocessing it; the preprocessing method is the same as in step 1.

[0069] Step 4: crop the preprocessed liver ultrasound training image to a set size (128×160×96) and input it into a 3D neural network ultrasound segmentation network (3D U-Net gallbladder and blood vessel segmentation network) for training and save the trained 3D neural network ultrasound segmentation network model.

[0070] Step 5: For a pair of test CT and ultrasound images of the same patient, the CT is calculated before online testing to obtain a liver CT test image and preprocess it; the preprocessing method is the same as in step 1.

[0071] Step 6: crop the preprocessed liver CT test image to a set size and input it into the trained 3D neural network CT segmentation network model for segmentation to obtain a CT segmentation image.

[0072] Step 7: extract the blood vessel part in the CT segmentation image, and use the blood vessel thinning filter itkBinaryThinningImageFilter of Insight Toolkit (www.itk.org) to extract the skeleton line.

[0073] Step 8, calculating the vascular branch points and branch planes based on the vascular portion in the CT segmented image:

[0074] First, extract all the bifurcation points. For the skeleton line image, select the point with the sum of pixels in the neighborhood of 1 being 3 as the branch point. Then calculate the three branches connected by the support point as a branch plane segment. Select the parent node p and child branch according to the angle. The two child branches are u and v. w is the normal vector perpendicular to the plane where u and v are located, q is the normal vector perpendicular to the plane where w and u are located, p is the parent node coordinate, and the plane space transformation T represented by it is:

[0075] w=u×v,q=w×u

[0076] T=[u,q,w,p]

[0077] Save branch points and branch information to a file.

[0078] Step 9: preprocess the collected on-site real-time liver three-dimensional ultrasound image; the preprocessing method is the same as that in step 1.

[0079] Step 10, the pre-processed real-time liver three-dimensional ultrasound image is cropped to a set size and then input into a trained 3D neural network ultrasound segmentation network model for segmentation to obtain an ultrasound segmentation image, such as Figure 2 shown.

[0080] Step 11, extract the blood vessel part in the ultrasonic segmentation image, and use the blood vessel thinning filter itkBinaryThinningImageFilter of Insight Toolkit (www.itk.org) to extract the skeleton line.

[0081] The vascular branch points and branch planes are calculated based on the vascular portion in the ultrasound segmentation image; the calculation method is the same as that in step 8.

[0082] Step 12, read in the branch plane information of CT, and transform the branch plane obtained based on the ultrasound segmentation image and the branch plane obtained based on the CT segmentation image.

[0083] Step 13, calculate the matching measure Ls score of each pair of transformed ultrasound segmentation images and CT segmentation images:

[0084]

[0085] Among them, A is the CT segmentation image, B is the ultrasound segmentation image, i is the pixel point, l represents the segmentation background label, R is the ultrasound area, and N is the number of all labels.

[0086] Step 14: Find the plane pair T with the highest matching measure score CT and T US , calculate the initial transformation matrix T CT-US , complete the liver 3D ultrasound and CT image registration initialization:

[0087]

[0088] The registration effect is as follows Figure 3 shown.

[0089] The present invention also provides a liver three-dimensional ultrasound and CT image registration initialization device, comprising:

[0090] The first training module: used to train and obtain a 3D neural network CT segmentation network model;

[0091] The second training module: used to train and obtain a 3D neural network ultrasound segmentation network model;

[0092] CT segmentation image acquisition module: used to input the liver CT test image into the trained 3D neural network CT segmentation network model for segmentation to obtain a CT segmentation image;

[0093] The first extraction module: used to extract the blood vessel part in the CT segmentation image;

[0094] The first calculation module is used to calculate the blood vessel branch points and branch planes based on the blood vessel part in the CT segmentation image;

[0095] Ultrasound segmentation image acquisition module: used to input the real-time liver three-dimensional ultrasound image into the trained 3D neural network ultrasound segmentation network model for segmentation to obtain the ultrasound segmentation image;

[0096] The second extraction module is used to extract the blood vessel part in the ultrasonic segmentation image;

[0097] The second calculation module is used for transforming the branch plane obtained based on the ultrasound segmentation image and the branch plane obtained based on the CT segmentation image;

[0098] The third calculation module is used to calculate the matching measurement score of each pair of transformed ultrasound segmentation images and CT segmentation images;

[0099] The fourth calculation module is used to calculate the initialization transformation matrix based on the plane pair with the highest matching measurement score, and complete the initialization of liver three-dimensional ultrasound and CT image registration.

[0100] The present invention also provides an electronic device, comprising:

[0101] one or more processors;

[0102] a storage device for storing one or more programs,

[0103] When the one or more programs are executed by the one or more processors, the one or more processors implement the liver three-dimensional ultrasound and CT image registration initialization method.

[0104] The present invention was tested on 18 CT-ultrasound data sets from 14 patients collected in a hospital, with an initialization success rate of 94%, a TRE of 6 mm, an initialization time of 16 seconds, and Dice similarity coefficients of ultrasonic gallbladder and blood vessel segmentation of 0.86 and 0.78 respectively.

[0105] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, which all belong to the protection scope of the present invention.

Claims

1. A liver three-dimensional ultrasound and CT image registration initialization method, characterized in that: include: Train and obtain a 3D neural network CT segmentation network model; Train and obtain a 3D neural network ultrasound segmentation network model; Input the liver CT test image into the trained 3D neural network CT segmentation network model for segmentation to obtain a CT segmentation image; Extract the blood vessel part in the CT segmentation image; Calculate the vascular branch points and branch planes based on the vascular parts in the CT segmentation image; Inputting the real-time three-dimensional liver ultrasound image into the trained 3D neural network ultrasound segmentation network model for segmentation to obtain an ultrasound segmentation image; Extracting blood vessel parts from ultrasound segmentation images; Calculate the vascular branch points and branch planes based on the vascular portion in the ultrasound segmentation image; Transforming the branch plane obtained based on the ultrasound segmentation image and the branch plane obtained based on the CT segmentation image; Calculate the matching measure score of each pair of transformed ultrasound segmentation images and CT segmentation images; Find the plane pair with the highest matching measure score, calculate the initialization transformation matrix, and complete the initialization of liver 3D ultrasound and CT image registration.

2. The liver three-dimensional ultrasound and CT image registration initialization method according to claim 1, characterized in that: The 3D neural network CT segmentation network model training process includes: Obtain liver CT training images and preprocess them; The preprocessed liver CT training image is cropped to a set size and input into the 3D neural network CT segmentation network for training and the trained 3D neural network CT segmentation network model is saved.

3. The liver three-dimensional ultrasound and CT image registration initialization method according to claim 1, characterized in that: The 3D neural network ultrasound segmentation network model training process includes: Obtaining liver three-dimensional ultrasound training images and preprocessing them; The preprocessed liver ultrasound training image is cropped to a set size and then input into the 3D neural network ultrasound segmentation network for training and the trained 3D neural network ultrasound segmentation network model is saved.

4. The liver three-dimensional ultrasound and CT image registration initialization method according to claim 1, characterized in that: Before inputting the liver CT test image into the trained 3D neural network CT segmentation network model, the following steps are also included: Acquire liver CT test images and preprocess them; The preprocessed liver CT test image is cropped to a set size and then input into the trained 3D neural network CT segmentation network model.

5. The liver three-dimensional ultrasound and CT image registration initialization method according to claim 1, characterized in that: Before inputting the real-time liver three-dimensional ultrasound image into the trained 3D neural network ultrasound segmentation network model, the following steps are also included: Preprocessing the acquired real-time three-dimensional liver ultrasound images; The preprocessed real-time liver three-dimensional ultrasound image is cropped to a set size and then input into the trained 3D neural network ultrasound segmentation network model.

6. The liver three-dimensional ultrasound and CT image registration initialization method according to any one of claims 2 to 5, characterized in that: The pretreatment process includes: Calculate the spatial spacing of all images to be preprocessed, select the median of all spatial spacings and resample all images to be preprocessed to the spatial spacing corresponding to the median; Calculate the grayscale mean and variance of the resampled image and normalize the grayscale.

7. The liver three-dimensional ultrasound and CT image registration initialization method according to any one of claims 1 to 5, characterized in that: The blood vessels in the CT segmented image are extracted, and the bone line is extracted using the blood vessel thinning filter.

8. The liver three-dimensional ultrasound and CT image registration initialization method according to any one of claims 1 to 5, characterized in that: The blood vessels in the ultrasonic segmentation image are extracted, and the skeleton line is extracted using the blood vessel thinning filter.

9. A liver three-dimensional ultrasound and CT image registration initialization device, characterized in that: include: The first training module: used to train and obtain a 3D neural network CT segmentation network model; The second training module: used to train and obtain a 3D neural network ultrasound segmentation network model; CT segmentation image acquisition module: used to input the liver CT test image into the trained 3D neural network CT segmentation network model for segmentation to obtain a CT segmentation image; The first extraction module: used to extract the blood vessel part in the CT segmentation image; The first calculation module is used to calculate the blood vessel branch points and branch planes based on the blood vessel part in the CT segmentation image; Ultrasound segmentation image acquisition module: used to input the real-time liver three-dimensional ultrasound image into the trained 3D neural network ultrasound segmentation network model for segmentation to obtain the ultrasound segmentation image; The second extraction module is used to extract the blood vessel part in the ultrasonic segmentation image; The second calculation module is used for transforming the branch plane obtained based on the ultrasound segmentation image and the branch plane obtained based on the CT segmentation image; The third calculation module is used to calculate the matching measurement score of each pair of transformed ultrasound segmentation images and CT segmentation images; The fourth calculation module is used to calculate the initialization transformation matrix based on the plane pair with the highest matching measurement score, and complete the initialization of liver three-dimensional ultrasound and CT image registration.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the liver three-dimensional ultrasound and CT image registration initialization method according to any one of claims 1 to 8.

Images