CT (Computed Tomography) guided needle biopsy method based on recursive deformable non-rigid registration network

Abstract

The invention discloses a CT (Computed Tomography) guided needle biopsy method based on a recursive deformable non-rigid registration network. The CT guided needle biopsy method comprises the following steps: S1, collecting and enhancing a training data set; s2, training a recursive deformable non-rigid pyramid registration network; s3, training a target tissue segmentation network; s4, deploying a registration network and preparing the network before the operation; and S5, performing real-time registration in the operation. By means of the highlight display technology for the target tissue in the target focus, whether the taken tissue is the target tissue of the preoperative plan or not can be helped to be confirmed after an operation, so that the success rate and efficiency of puncture are improved, and unnecessary secondary biopsy is reduced.

Description

Technical Field

[0001] The present invention relates to a CT-guided puncture biopsy method, and in particular to a CT-guided puncture biopsy method based on a recursive deformable non-rigid registration network. Background Art

[0002] Before CT-guided puncture, enhanced CT examination is usually performed. Based on the principle of safety and obtaining the most effective tissue, the puncture path is ideally designed with the most obvious enhancement or radionuclide uptake as the target. However, for lesions with necrosis as the main focus or less biopsy tissue, the operator needs to repeatedly compare the preoperative enhanced image or nuclear concentration image with the plain scan as the image background during the biopsy, which is time-consuming and labor-intensive, increases the puncture time, and has a strong uncertainty as to whether the effective target tissue can be biopsied. Objective factors also include the free movement of organs and the positional deviation caused by the ejection of the biopsy slot. Whether the biopsied tissue is the target tissue set before the operation cannot be confirmed and fed back after the operation.

[0003] Generally, contrast agents can be injected during the puncture process to observe the target lesions, but this has the following disadvantages: ① CT-guided puncture is not a continuous dynamic process. Before adjusting the needle angle, a local scan is required for confirmation, because multiple large-scale injections of contrast agents are not in line with ethical standards and impose a huge burden on the patient's body for contrast agent excretion, damaging renal function; ② After the contrast agent is injected, experienced technicians are required to grasp the phase when the contrast agent enters the target organ, and there are many uncertainties; ③ Large amounts of contrast agents are expensive.

[0004] Therefore, for target lesions with less biopsy tissue, how to clearly display the effective target tissue during surgery without the need for contrast agent enhancement and provide confirmation and feedback after surgery is of great clinical value. Summary of the invention

[0005] Purpose of the invention: In view of the fact that the operator needs to repeatedly compare the preoperative enhanced image or nuclear concentration image during the existing biopsy operation, and the free movement of the organs and the ejection of the biopsy slot cause positional displacement, the present invention provides a CT-guided puncture biopsy method based on a recursive deformable non-rigid registration network, which can realize real-time registration of intraoperative plain scan images with preoperative enhanced images and improve the success rate and efficiency of puncture.

[0006] Technical solution: The CT-guided puncture biopsy method based on a recursive deformable non-rigid registration network described in the present invention comprises the following steps:

[0007] (1) Collection and enhancement of training datasets;

[0008] (2) Recursive deformable non-rigid pyramid registration network training;

[0009] (3) Target tissue segmentation network training;

[0010] (4) Deployment of the registration network and preparation of the preoperative network;

[0011] (5) Real-time registration during surgery.

[0012] Furthermore, the step S1 method: adopts a data enhancement strategy to obtain preoperative and intraoperative CT image pairs as training data for the present method, firstly collects preoperative enhanced CT images, uses a traditional intensity-based image registration method, designs n deformation fields φ, and the deformation field used for data enhancement is arbitrarily selected from these n deformation fields, and uses thin plate spline interpolation to simulate small changes in a specific stage, and finally inputs the preoperative enhanced image and the intraoperative deformed image obtained by data enhancement as a registration image pair into the registration network.

[0013] Furthermore, the presence or absence of target tissue in the image to be registered is manually marked to obtain a segmentation mask corresponding to the target tissue.

[0014] Furthermore, the method of step S2: during the training process, the preoperative image x f And the image x produced during the operation m are respectively input into the designed dual-branch feature encoder ε f (·), and ε m (·) generates features with different scale information, which are denoted as F1 and M respectively. l , 1 = {1, 2, 3, 4}, and then input it into the designed recursive deformable decoder to continuously learn and generate the deformation field θ1 under the scale feature, and recursively obtain the deformation field θ2 and deformation field θ4 of the second-level scale feature. Then, the Spatial TransformerNetwork is used to apply the learned deformation field θ4 to the preoperative image x f , generate the postoperative image x n , and finally the normalized cross-correlation loss L is used NCC For evaluation of preoperative images x f And the predicted postoperative image x n The similarity between NCC The expression is as follows:

[0015]

[0016] Where pi represents a patch in the image, whose size is 9x9. as well as Represents the average pixel value within this patch.

[0017] Furthermore, the encoder has a dual-branch feature and adopts a pyramid structure consisting of four encoding layers.

[0018] Furthermore, the decoder includes four recursive decoding layers, which correspond to the four feature encoding layers designed in the encoder. Each decoding module includes three core modules: feature fusion, deformation estimation, and deformation aggregation.

[0019] Furthermore, the features F4 and M4 obtained by the fourth encoding layer are first input into the first decoding layer of the decoder. The decoding layer consists of a feature fusion module, a deformation estimation module and a deformation aggregation module. The deformation field θ1 under the scale feature is continuously learned in a cyclic recursive manner. The deformation field θ1 and the features F3 and M3 obtained by the third encoder are further input into the second decoder, and the deformation field θ2 of the second scale feature is obtained cyclically and recursively. This step is repeated until the deformation field θ4 learned by the fourth decoding layer is generated.

[0020] Furthermore, the method of step S3 is as follows: for the registered intraoperative plain scan image x n , a U-net based segmentation network is used to identify the target tissue in the image.

[0021] Furthermore, the encoder of U-net is combined with the encoder ε of the registration network f (·) Shared weights, the loss of the segmentation mask after decoding prediction is calculated for the true label, and the binary cross entropy loss function is used to implement supervised learning. In addition, a focal loss is added as a constraint term. The process is expressed as:

[0022] L seg =L BCE (y,p)+αL Focal (y, p)

[0023] Where y is the pixel value of the true label, p is the predicted value of the model, and α is the weight coefficient used to balance the contribution of the Focal loss function.

[0024] Furthermore, the method of step S5 is as follows: inputting the intraoperative plain scan image and the preoperative fixed image into the trained registration network, registering the intraoperative plain scan image with the preoperative enhanced image, and displaying the lesion area in color.

[0025] The system network architecture of the present invention includes the following modules:

[0026] Dual-branch feature encoder. To achieve efficient registration of preoperative and intraoperative images, this method designs a dual-branch feature encoder. Each branch of the encoder is responsible for feature extraction of preoperative or intraoperative images to fully explore the potential features between different registered images. This method denotes the feature encoder of the preoperative static image as ε f (·), the feature encoder of intraoperative movement is denoted as ε m(·).

[0027] Recursive deformable decoder. The decoder aims to decode the features extracted by the encoder and realize the registration of preoperative and intraoperative images in a novel recursive pyramid way. Specifically, the encoder designed by this method includes four layers of network decoding layers, and each decoding module contains three core modules: feature fusion, deformation estimation, and deformation aggregation:

[0028] 1) Feature fusion module: This module aims to aggregate the deep features extracted by the dual-branch encoder and achieve coupled learning of the two features through two convolutional layers.

[0029] 2) Deformation estimation module: This module uses convolutional layers and differential homomorphic layers to represent and learn the fused features to predict the deformation field between the registered images.

[0030] 3) Deformation aggregation module: This module uses the Spatial Transformer Network (STN) to aggregate the deformation fields predicted in adjacent decoding layers to improve the registration performance in non-rigid registration scenarios.

[0031] Target tissue segmentation network. This network directly uses the U-Net network. Unlike the traditional segmentation network, its feature encoder shares weights with the encoder of the recursive registration network, so that the method of the present invention satisfies both the image registration function and the target tissue segmentation function.

[0032] The method of the present invention is divided into four steps: Figure 1 As shown:

[0033] Step S1: Collection and enhancement of training data sets. The data enhancement strategy is used to obtain preoperative and intraoperative CT image pairs as training data for this method. First, preoperative CT enhanced images are collected. In order to obtain more morphologically diverse images, the traditional intensity-based image registration method is used to design n deformation fields φ. The deformation field used for data enhancement is arbitrarily selected from these n deformation fields. Considering the free movement of organs and the positional offset caused by the ejection of the biopsy slot, thin plate spline (TPS) interpolation is used to simulate small changes in specific stages.

[0034] Step S2: Recursive deformable non-rigid registration network training. The network framework is as follows Figure 2 shown.

[0035] During the training process, the preoperative image x f And the image x produced during the operation m are respectively input into the designed dual-branch feature encoder ε f (·), and ε mThe encoder adopts a pyramid structure, which consists of four encoding layers. It performs multi-scale feature learning on the input image and generates features with different scale information, which are denoted as F l and M l , l = {1, 2, 3, 4}. Then it is input into the designed recursive deformable decoder. The decoder has four recursive decoding layers, which correspond to the four feature encoding layers designed in the encoder. This method first inputs the features F4 and M4 obtained from the fourth encoding layer into the first decoding layer of the decoder. The decoding layer consists of a feature fusion module, a deformation estimation module, and a deformation aggregation module, and continuously learns to generate the deformation field θ1 under the scale feature in a cyclic recursive manner. After that, the obtained θ1 and the features F3 and M3 obtained by the third encoder are further input into the decoder of the second layer, and the deformation field θ2 of the second-layer scale feature is obtained cyclically and recursively, and this step is repeated until the deformation field θ4 learned by the fourth decoding layer is generated. Subsequently, the Spatial Transformer Network (STN) is used to apply the learned deformation field θ4 to the preoperative image x f , generating the postoperative image x estimated by this method n .Finally, the normalized cross-correlation loss L is used NCC For evaluation of preoperative images x f And the predicted postoperative image x n The similarity between NCC The expression is as follows:

[0036]

[0037] Where pi represents a patch in the image, whose size is 9x9. as well as Represents the average pixel value within this patch.

[0038] Step S3: Target tissue segmentation network training. The present invention can not only realize the real-time registration of intraoperative plain scan images, but also detect and highlight the target tissue in the target lesion in real time. Specifically, for the registered intraoperative plain scan image x n , we maintain a segmentation network based on U-net to identify the target tissue in the image. Different from the traditional segmentation network, the encoder of U-net is combined with the encoder of the registration network ε f(·) Shared weights to achieve both the registration and segmentation functions of the network, meeting the needs of lightweight and high efficiency in clinical scenarios. The loss of the segmentation mask after decoding prediction is calculated for the true label, and supervised learning is achieved using a binary cross entropy loss function. Considering that the content of lesion tissue in the entire image is small, in order to balance the quantitative relationship between positive samples and secondary samples, an additional Focal Loss is added as a constraint to fully detect the lesion tissue.

[0039] Step S4: Deployment of the registration network and preparation of the preoperative network. Deploy the trained network to prepare for real-time image registration during surgery.

[0040] Step S5: Intraoperative registration. The intraoperative plain scan image and the preoperative fixed image are input into the trained registration network, the intraoperative plain scan image is registered with the preoperative enhanced image, and the lesion area is displayed in color to assist the doctor in the operation.

[0041] The ultimate goal of CT-guided puncture surgery is to obtain effective tissue samples for pathological examination, and its method is mainly based on planning the puncture path based on the target area determined by the preoperative enhanced examination. However, for target lesions that can be sampled less, they cannot be clearly displayed in the plain scan images during surgery. At the same time, the free movement of the organs and the possible positional deviation caused by the ejection of the biopsy slot make it impossible to ensure that the tissue taken is the target tissue in the preoperative plan. The present invention provides a technology that highlights the target tissue in the target lesion, which can help confirm whether the biopsy tissue is the target in the preoperative plan after surgery, thereby further improving the success rate and efficiency of the puncture and reducing unnecessary secondary biopsies.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. For target lesions with less biopsy tissue, the present invention overcomes the disadvantage that it is difficult to clearly identify the target tissue during CT scan during biopsy without the need for contrast agent enhancement. The target tissue can be segmented and highlighted based on the registration of preoperative enhanced images or radionuclide concentration images, which helps to confirm whether it is the target tissue in the preoperative plan after the biopsy, saving time, improving the success rate and efficiency of puncture, and being simple and practical.

[0044] 2. The present invention uses a technology to highlight the target tissue in the target lesion, which can help confirm whether the tissue taken is the target tissue planned before the operation, thereby improving the success rate and efficiency of the puncture and reducing unnecessary secondary biopsies. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic diagram of the method flow chart;

[0046] Figure 2Schematic diagram of the recursive deformable non-rigid registration network structure;

[0047] Figure 3 Schematic diagram of the key components of the recursive deformable non-rigid registration network, including: (a) the structure of the first layer of the encoding layer; (b) the structure of layers 2-4 of the encoding layer; (c) the feature fusion module of the decoding layer; (d) the deformation estimation module and the deformation aggregation module of the decoding layer. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be further described below.

[0049] Step S1: Collection and enhancement of training data sets. A data enhancement strategy is adopted to use preoperative and intraoperative CT image pairs as training data for this method. First, preoperative enhanced CT images are collected. In order to obtain more morphologically diverse images, a traditional intensity-based image registration method is adopted, and n=9 is designed, but not limited to 9 deformation fields φ. The deformation field used for data enhancement is randomly selected from these 9 deformation fields. In order to consider the free movement of organs and the positional deviation caused by the ejection of biopsy slots, subtle changes in specific stages are simulated by thin plate spline (TPS) interpolation. Finally, the preoperative enhanced image and the intraoperative deformed image obtained by data enhancement are input as a registration image pair into the registration network designed by the present invention. In addition, the presence or absence of target tissue in the image to be registered is manually marked, and the segmentation mask of the corresponding target tissue is obtained to train the target tissue recognition and segmentation network.

[0050] The registration network satisfies the registration and segmentation functions of the network. For the registration network, it adopts a dual-branch, pyramid coding structure to encode the image pairs to be registered respectively, and extract multi-scale feature information. Then the feature information from the dual branches is sent to the designed recursive deformable decoder, and the deformation field between the image pairs to be registered is gradually optimized in a cyclic recursive manner. We use the normalized cross-correlation (NCC) loss function to supervise the deformation between the preoperative and intraoperative image pairs. In addition, in order to highlight the target tissue during surgery, the registration network is equipped with a U-net segmentation sub-network. The segmentation network shares weights with the encoder of the quasi-network to meet the requirements of lightweight and high efficiency in clinical scenarios. The sub-network uses the binary cross entropy (BCE) loss function and the focal loss function for supervised learning.

[0051] Step S2: Recursive deformable non-rigid registration network training. The network framework is as follows Figure 2 As shown. During the training process, the preoperative image x f And the intraoperative plain scan image xm are respectively input into the designed dual-branch feature encoder ε f (·), and ε m (·). The encoder adopts a pyramid structure, which consists of four encoding layers. It learns multi-scale features of the input image and generates features with different scale information. Figure 3 As shown, for the first encoding layer, it includes {Multi-Head Attention Layer (MHSA)-Layer Norm Layer (LNorm)-LeakyRelu Activation Layer (LRelu)}, as Figure 3 (a) The subsequent three layers include {down-sampling convolution layer (DownConv)-multi-head attention layer (MHSA)-layer norm layer (LNorm)-leakyRelu activation layer (LRelu)}, where the down-sampling convolution layer (DownConv) connected at the beginning of the module converts the feature F of the previous layer into l-1 (M l-1 ) The feature scale is reduced and the number of channels is doubled, such as Figure 3 (b) is shown. And the last three layers all use residual connections after the multi-head attention layer to enhance the feature representation ability. The multi-scale features of each encoding layer of the dual-branch encoder are extracted to obtain F l and M l , l = {1, 2, 3, 4}.

[0052] Then the multi-scale feature F l and M l Input into the designed recursive deformable decoder. The decoder has four recursive decoding layers, which correspond to the four feature encoding layers designed in the encoder. The decoding layer designed by the present invention consists of a feature fusion module, a deformation estimation module and a deformation aggregation module, which continuously learns to generate the deformation field θ1 under the scale feature in a cyclic recursive manner.

[0053] Specifically, in the fourth layer decoder, features F4 and M4 are sent to the feature fusion module, which contains two {multi-head attention layer (MHSA)-Layer Norm layer (LNorm)-LeakyRelu activation layer (LRelu)}, and the fused feature E4 is obtained, as shown in Figure 3 (c). The process is expressed as:

[0054] E4=f([F4,M4])

[0055] Where f((.) represents the feature fusion module, and [.] represents the cascading of the two along the channel dimension.

[0056] Then, the fused feature E4 is sent to the deformation estimation module. This module consists of a multi-head attention layer (MHSA) and a differential homomorphic layer (Diff), which aims to characterize and learn the fused features to generate the deformation field θ4, as shown in Figure 3 (d) is shown. The process can be expressed as:

[0057] θ4=g(E4)

[0058] In addition, we use a recursive loop to continuously optimize the predicted deformation field. The process is expressed as:

[0059]

[0060]

[0061]

[0062] In the first layer decoder, D4 is E4, i represents the number of cycles, i=1, 2, ..k. k=5 but not limited to 5. * represents the deformation operation, which is implemented by the STN network. fusion(.) represents the designed deformation aggregation module, such as Figure 3 (d), which is expressed as:

[0063]

[0064] Then we send the final fusion feature E4 and θ4 of this layer to the decoder of the next layer. For the following three layers, we use 1 (1ayer) to represent the number of layers. The process of recursively predicting the deformation field of each layer can be expressed as:

[0065]

[0066]

[0067]

[0068] Where D l+1 , 1=3, 2, 1 represents the aggregated feature E obtained by the previous decoding layer l+1 fusion(.) is expressed as:

[0069]

[0070] This method establishes a pyramid hierarchical structure to gradually characterize and model the multi-scale information of preoperative and intraoperative plain scan images and gradually decode them, and uses recursive optimization to continuously perform fine-grained optimization on the deformation field. Finally, we apply the deformation field θ4 learned by the last decoding layer to the intraoperative image X m, and the NCC loss function is used to supervise the deformation field θ4 between the preoperative enhanced image and the intraoperative plain scan image. The NCC loss function is expressed as follows:

[0071]

[0072] Where pi represents a patch in the image, whose size is 9x9. as well as It represents the average value within this patch.

[0073] Step S3: Target tissue segmentation network training. For the registered intraoperative plain scan image x n , a U-net-based segmentation network is used to identify the target tissue in the image. Specifically, the encoder of U-net is combined with the encoder ε of the registration network f (·) Shared weights to achieve the network's registration and segmentation functions at the same time, meeting the needs of lightweight and high efficiency in clinical scenarios. Finally, the loss of the segmentation mask after decoding prediction is calculated for the true label, and the binary cross entropy loss function is used to achieve supervised learning. In addition, considering that the content of lesion tissue in the entire image is small, in order to balance the quantitative relationship between positive samples and secondary samples, an additional FocalLoss is added as a constraint to fully detect the lesion tissue. The process can be expressed as:

[0074] L seg =L BCE (y,p)+αL Focal (y, p)

[0075] Among them, y is the pixel value of the true label, p is the predicted value of the model, and α is the weight coefficient used to balance the contribution of the Focal loss function.

[0076] Step S4: Deployment of the registration network and preparation of the preoperative network. Deploy the trained network to prepare for real-time image registration during surgery.

[0077] Step S5: Intraoperative registration and lesion display. The intraoperative plain scan image and the preoperative fixed image are input into the trained registration network, the intraoperative plain scan image is registered with the preoperative enhanced image, and the lesion area is displayed in color to assist the doctor in the operation.

[0078] (1) A recursive deformable non-rigid registration network invented by preoperative data collection;

[0079] (2) After the trained network is configured, the intraoperative network will realize the registration of the intraoperative plain scan image and the preoperative enhanced image to assist the doctor in the operation.

[0080] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A CT-guided biopsy method based on a recursive deformable non-rigid registration network, characterized in that: The steps include: S1. Collection and enhancement of training datasets; S2, recursive deformable non-rigid pyramid registration network training; S3, target tissue segmentation network training; S4, deployment of the registration network and realization of preoperative network preparation; S5. Real-time registration during surgery.

2. The CT-guided biopsy method based on a recursive deformable non-rigid registration network according to claim 1, characterized in that: The method of step S1: adopting a data enhancement strategy to obtain preoperative and intraoperative CT image pairs as training data for the present method, firstly collect preoperative CT enhanced images, use the traditional intensity-based image registration method to design n deformation fields φ, the deformation field used for data enhancement is arbitrarily selected from these n deformation fields, and thin plate spline interpolation is used to simulate small changes in a specific stage, and finally the preoperative enhanced image and the intraoperative deformation image obtained by data enhancement are input into the registration network as a registration image pair.

3. The CT-guided biopsy method based on a recursive deformable non-rigid registration network according to claim 2, characterized in that: Manually mark whether there is target tissue in the image to be registered and obtain the segmentation mask of the corresponding target tissue.

4. The CT-guided biopsy method based on a recursive deformable non-rigid registration network according to claim 3, characterized in that: The method of step S2: during the training process, the preoperative image x f And the image x produced during the operation m are respectively input into the designed dual-branch feature encoder ε f (·), and ε m (·), features with different scale information are generated, denoted as F k and M k , k = {1, 2, 3, 4}, and then input it into the designed recursive deformable decoder to continuously learn and generate the deformation field θ1 under the scale feature, and recursively obtain the deformation field θ2 and deformation field θ4 of the second-level scale feature. Subsequently, the Spatial Transformer Network is used to apply the learned deformation field θ4 to the preoperative image x f , generate the postoperative image x n , and finally the normalized cross-correlation loss L is used NCC For evaluation of preoperative images x f And the predicted postoperative image x n The similarity between NCC The expression is as follows: Where pi represents a patch in the image, whose size is 9x9. as well as It represents the average pixel value in this patch.

5. The CT-guided biopsy method based on a recursive deformable non-rigid registration network according to claim 4, characterized in that: The encoder has dual-branch features and adopts a pyramid structure consisting of four encoding layers.

6. The CT-guided biopsy method based on a recursive deformable non-rigid registration network according to claim 4, characterized in that: The decoder consists of four recursive decoding layers, which correspond to the four feature encoding layers designed in the encoder. Each decoding module includes three core modules: feature fusion, deformation estimation, and deformation aggregation.

7. The CT-guided biopsy method based on a recursive deformable non-rigid registration network according to claim 6, characterized in that: First, the features F4 and M4 obtained from the fourth encoding layer are input into the first decoding layer of the decoder. The decoding layer consists of a feature fusion module, a deformation estimation module, and a deformation aggregation module. The deformation field θ1 under the scale feature is continuously learned in a cyclic recursive manner. The deformation field θ1 and the features F3 and M3 obtained from the third encoder are further input into the second decoder. The deformation field θ2 of the second scale feature is obtained cyclically and recursively, and this step is repeated until the deformation field θ4 learned by the fourth decoding layer is generated.

8. The CT-guided biopsy method based on a recursive deformable non-rigid registration network according to claim 7, characterized in that: The method of step S3 is: for the registered intraoperative plain scan image x n , a U-net based segmentation network is used to identify the target tissue in the image.

9. The CT-guided biopsy method based on a recursive deformable non-rigid registration network according to claim 8, characterized in that: The encoder of U-net is combined with the encoder ε of the registration network f (·) Shared weights, the loss of the segmentation mask after decoding prediction is calculated for the true label, and the binary cross entropy loss function is used to implement supervised learning. In addition, a focal loss is added as a constraint term. The process is expressed as: L seg =L BCE (y,p)+αL Focal (y,p) Where y is the pixel value of the true label and p is the predicted value of the model.

10. The CT-guided biopsy method based on a recursive deformable non-rigid registration network according to claim 9, characterized in that: The method of step S5 is to: The image is input into the trained registration network to register the intraoperative plain scan image with the preoperative enhanced image. The lesion area is displayed in color.

Images