A CT-guided biopsy method based on recursive deformable non-rigid registration network

Abstract

This invention discloses a CT-guided biopsy method based on a recursive deformable non-rigid registration network, comprising the following steps: S1, collection and enhancement of a training dataset; S2, training of a recursive deformable non-rigid pyramid registration network; S3, training of a target tissue segmentation network; S4, deployment of the registration network and preoperative network preparation; and S5, real-time intraoperative registration. This invention utilizes a technique for highlighting target tissue within the target lesion, helping to confirm whether the tissue being harvested is the preoperatively planned target tissue after surgery, thereby improving the success rate and efficiency of the puncture and reducing unnecessary secondary biopsies.

Description

Technical Field

[0001] The present invention relates to a CT-guided puncture biopsy method, and in particular to a CT-guided puncture biopsy method based on a recursive deformable non-rigid registration network. Background Art

[0002] CT-guided puncture is typically preceded by an enhanced CT scan. Ideally, the puncture path is designed to target sites with the most pronounced enhancement or radionuclide uptake, based on the principles of safety and maximum effective tissue acquisition. However, for lesions with predominantly necrotic lesions or limited biopsy tissue, the operator must repeatedly compare preoperative enhanced or radionuclide uptake images with the plain scan as the background image during the biopsy procedure. This is time-consuming and labor-intensive, increasing puncture time and significantly uncertainting whether the target tissue will be biopsied. Other objective factors include free movement of organs and positional deviation caused by ejection of the biopsy slot. Postoperative confirmation and feedback on whether the biopsied tissue represents the preoperative target tissue are lacking.

[0003] Generally, contrast agents can be injected during the puncture process to observe the target lesions, but this has the following disadvantages: ① CT-guided puncture is not a continuous dynamic process. Before adjusting the needle angle, a local scan is required for confirmation, because multiple large-scale injections of contrast agents are not in line with ethical standards and impose a huge contrast agent excretion burden on the patient's body, damaging renal function; ② After the contrast agent is injected, experienced technicians are required to grasp the phase when the contrast agent enters the target organ, and there are many uncertainties; ③ Large amounts of contrast agents are expensive.

[0004] Therefore, for target lesions with relatively little biopsy tissue, how to clearly display the effective target tissue during surgery without the need for contrast agent enhancement and provide confirmation and feedback after surgery is of great clinical value. Summary of the Invention

[0005] Purpose of the invention: To address the problem that during existing biopsy operations, the operator needs to repeatedly compare preoperative enhanced images or nuclear concentration images, and that free movement of organs and ejection of biopsy slots cause positional deviations, the present invention provides a CT-guided puncture biopsy method based on a recursive deformable non-rigid registration network, which can achieve real-time registration between intraoperative plain scan images and preoperative enhanced images, thereby improving the success rate and efficiency of puncture.

[0006] Technical solution: The CT-guided biopsy method based on a recursive deformable non-rigid registration network of the present invention comprises the following steps:

[0007] (1) Collection and enhancement of training datasets;

[0008] (2) Recursive deformable non-rigid pyramid registration network training;

[0009] (3) Target tissue segmentation network training;

[0010] (4) Deployment of the registration network and implementation of preoperative network preparation;

[0011] (5) Real-time registration during surgery.

[0012] Furthermore, the step S1 method: adopts a data enhancement strategy to obtain preoperative and intraoperative CT image pairs as training data for this method, first collects preoperative CT enhanced images, uses the traditional intensity-based image registration method, designs n deformation fields φ, and the deformation field used for data enhancement is arbitrarily selected from these n deformation fields, and uses thin plate spline interpolation to simulate small changes in a specific stage. Finally, the preoperative enhanced image and the intraoperative deformed image obtained by data enhancement are input into the registration network as a registration image pair.

[0013] Furthermore, the presence or absence of target tissue in the image to be registered is manually marked to obtain a segmentation mask corresponding to the target tissue.

[0014] Furthermore, the method of step S2: during the training process, the preoperative image x f And the image x that is deformed during surgery m are input into the designed dual-branch feature encoder ε f (·), and ε m (·) produces features with different scale information, which are recorded as F1 and M respectively. l , 1 = {1, 2, 3, 4}, and then input it into the designed recursive deformable decoder to continuously learn and generate the deformation field θ1 under the scale feature, and recursively obtain the deformation field θ2 and deformation field θ4 of the second-level scale feature. Subsequently, the learned deformation field θ4 is applied to the preoperative image x using Spatial TransformerNetwork. f , generate postoperative image x n , and finally adopt the normalized cross-correlation loss L NCC For evaluation of preoperative images x f And the predicted postoperative image x n The similarity between L NCC The expression is as follows:

[0015]

[0016] Where pi represents a patch in the image, whose size is 9x9. as well as It represents the average pixel value within this patch.

[0017] Furthermore, the encoder has a dual-branch feature and adopts a pyramid structure consisting of four encoding layers.

[0018] Furthermore, the decoder includes four recursive decoding layers, which correspond to the four feature encoding layers designed in the encoder. Each decoding module includes three core modules: feature fusion, deformation estimation, and deformation aggregation.

[0019] Furthermore, the features F4 and M4 obtained by the fourth encoding layer are first input into the first decoding layer of the decoder. The decoding layer consists of a feature fusion module, a deformation estimation module, and a deformation aggregation module. The deformation field θ1 under the scale feature is continuously learned in a cyclic recursive manner. The deformation field θ1 and the features F3 and M3 obtained by the third encoder are further input into the second decoder, and the deformation field θ2 of the second-layer scale feature is obtained cyclically and recursively. This step is repeated until the deformation field θ4 learned by the fourth decoding layer is generated.

[0020] Furthermore, the method of step S3 is as follows: for the registered intraoperative plain scan image x n , a U-net based segmentation network is used to identify target tissues in images.

[0021] Furthermore, the encoder of U-net is combined with the encoder ε of the registration network f (·) Shared weights, the loss is calculated for the segmentation mask after decoding and prediction by the true label, and the binary cross entropy loss function is used to implement supervised learning. In addition, a focal loss is added as a constraint term. The process is expressed as:

[0022] L seg =L BCE (y, p)+αL Focal (y, p)

[0023] Where y is the pixel value of the true label, p is the predicted value of the model, and α is the weight coefficient used to balance the contribution of the Focal loss function.

[0024] Furthermore, the method of step S5 is as follows: the intraoperative plain scan image and the preoperative fixed image are input into the trained registration network, the intraoperative plain scan image and the preoperative enhanced image are registered, and the lesion area is displayed in color.

[0025] The system network architecture of the present invention includes the following modules:

[0026] To achieve efficient registration of preoperative and intraoperative images, this method designs a dual-branch feature encoder. Each branch of the encoder is responsible for feature extraction of preoperative or intraoperative images, so as to fully exploit the potential features between different registered images. This method denotes the feature encoder of the preoperative still image as ε f (·), the feature encoder of intraoperative movement is denoted as ε m(·).

[0027] Recursive Deformable Decoder. This decoder aims to decode the features extracted by the encoder and achieve preoperative and intraoperative image registration using a novel recursive pyramid approach. Specifically, the encoder designed in this method includes four network decoding layers, each of which contains three core modules: feature fusion, deformation estimation, and deformation aggregation:

[0028] 1) Feature fusion module: This module aims to aggregate the deep features extracted by the dual-branch encoder and achieve coupled learning of the two features through two convolutional layers.

[0029] 2) Deformation estimation module: This module uses convolutional layers and differential homomorphic layers to represent and learn the fused features to predict the deformation field between the registered images.

[0030] 3) Deformation Aggregation Module: This module uses the Spatial Transformer Network (STN) to aggregate the deformation fields predicted in adjacent decoding layers to improve the registration performance in non-rigid registration scenarios.

[0031] Target tissue segmentation network. This network directly uses a U-Net network. Unlike traditional segmentation networks, its feature encoder shares weights with the encoder of the recursive registration network, allowing the method of the present invention to simultaneously meet the image registration function and target tissue segmentation function.

[0032] The method of the present invention is divided into four steps, such as Figure 1 As shown:

[0033] Step S1: Collection and enhancement of the training dataset. Data augmentation strategies are used to obtain preoperative and intraoperative CT image pairs as training data for this method. First, preoperative enhanced CT images are collected. To obtain more morphologically diverse images, traditional intensity-based image registration methods are used to design n deformation fields φ. The deformation field used for data augmentation is arbitrarily selected from these n deformation fields. Thin plate spline (TPS) interpolation is used to simulate small changes at specific stages, taking into account the free movement of organs and positional offsets caused by biopsy slot ejection.

[0034] Step S2: Recursive deformable non-rigid registration network training. The network framework is as follows Figure 2 shown.

[0035] During the training process, the preoperative image x f And the image x that is deformed during surgery m are input into the designed dual-branch feature encoder ε f (·), and ε m(·). The encoder adopts a pyramid structure, which consists of four coding layers. It learns multi-scale features of the input image and generates features with different scale information, which are denoted as F l and M l , l = {1, 2, 3, 4}. It is then input into the designed recursive deformable decoder. The decoder has four recursive decoding layers, which correspond to the four feature encoding layers designed in the encoder. This method first inputs the features F4 and M4 obtained from the fourth encoding layer into the first decoding layer of the decoder. The decoding layer consists of a feature fusion module, a deformation estimation module, and a deformation aggregation module, and continuously learns to generate the deformation field θ1 under the scale feature in a cyclic recursive manner. After that, the obtained θ1 and the features F3 and M3 obtained by the third encoder are further input into the decoder of the second layer, and the deformation field θ2 of the second-layer scale feature is obtained cyclically and recursively, and this step is repeated until the deformation field θ4 learned by the fourth decoding layer is generated. Subsequently, the Spatial Transformer Network (STN) is used to apply the learned deformation field θ4 to the preoperative image x f , generate the postoperative image x estimated by this method n .Finally, the normalized cross-correlation loss L is used NCC For evaluation of preoperative images x f And the predicted postoperative image x n The similarity between L NCC The expression is as follows:

[0036]

[0037] Where pi represents a patch in the image, whose size is 9x9. as well as It represents the average pixel value within this patch.

[0038] Step S3: Target tissue segmentation network training. The present invention can not only realize the real-time registration of intraoperative plain scan images, but also detect and highlight the target tissue in the target lesion in real time. Specifically, for the registered intraoperative plain scan image x n , we maintain a U-net based segmentation network to identify the target tissue in the image. Different from the traditional segmentation network, the encoder of U-net is combined with the encoder of the registration network ε f(·) Shared weights enable simultaneous network registration and segmentation, meeting the requirements of lightweight and efficient clinical scenarios. Loss is calculated between the true label and the decoded predicted segmentation mask, and supervised learning is achieved using a binary cross-entropy loss function. Considering the small amount of lesion tissue in the entire image, an additional Focal Loss is added as a constraint to balance the number of positive and negative samples to ensure sufficient detection of lesions.

[0039] Step S4: Deployment of the registration network and preparation of the preoperative network. Deploy the trained network to prepare for real-time image registration during surgery.

[0040] Step S5: Intraoperative registration. The intraoperative plain scan image and the preoperative fixed image are input into the trained registration network, and the intraoperative plain scan image and the preoperative enhanced image are registered, and the lesion area is displayed in color to assist the doctor in the operation.

[0041] The ultimate goal of CT-guided puncture surgery is to obtain effective tissue samples for pathological examination. Its method is mainly based on planning the puncture path based on the target area determined by the preoperative enhanced examination. However, for target lesions that can be sampled less, they cannot be clearly displayed in the plain scan images during surgery. At the same time, the free movement of the organs and the ejection of the biopsy slot may cause positional deviation, resulting in the inability to ensure that the tissue taken is the target tissue in the preoperative plan. The present invention provides a technology that highlights the target tissue in the target lesion, which can help confirm whether the biopsy tissue is the target in the preoperative plan after surgery, thereby further improving the success rate and efficiency of the puncture and reducing unnecessary secondary biopsies.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. For target lesions with limited biopsy tissue, this method overcomes the difficulty of clearly identifying target tissue on plain CT scans during biopsy, without the need for contrast enhancement. Based solely on the registration of preoperative enhanced images or radionuclide enrichment images, the target tissue can be segmented and highlighted. This helps confirm whether the biopsy is the target tissue planned after the procedure, saving time, improving puncture success rate and efficiency, and being simple and practical.

[0044] 2. The present invention uses a technology to highlight the target tissue in the target lesion, which can help confirm whether the tissue taken is the target tissue planned before the operation, thereby improving the success rate and efficiency of the puncture and reducing unnecessary secondary biopsies. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the method flow chart;

[0046] Figure 2Schematic diagram of the recursive deformable non-rigid registration network structure;

[0047] Figure 3 Schematic diagram of the key components of the recursive deformable non-rigid registration network, including: (a) the structure of the first layer of the encoding layer; (b) the structure of layers 2-4 of the encoding layer; (c) the feature fusion module of the decoding layer; (d) the deformation estimation module and deformation aggregation module of the decoding layer. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be further described below.

[0049] Step S1: Collection and enhancement of training data sets. A data enhancement strategy is adopted, and preoperative and intraoperative CT image pairs are used as training data for this method. First, preoperative CT enhanced images are collected. In order to obtain more morphologically diverse images, a traditional intensity-based image registration method is adopted, and n=9, but not limited to 9 deformation fields φ, are designed. The deformation field used for data enhancement is randomly selected from these 9 deformation fields. In order to take into account the free movement of the organs and the positional offset caused by the ejection of the biopsy slot, the subtle changes in a specific stage are simulated by thin plate spline (TPS) interpolation. Finally, the preoperative enhanced image and the intraoperative deformed image obtained by data enhancement are input as a registration image pair into the registration network designed by the present invention. In addition, the presence or absence of target tissue in the image to be registered is manually marked, and the segmentation mask of the corresponding target tissue is obtained to train the target tissue recognition and segmentation network.

[0050] The registration network satisfies the registration and segmentation functions of the network. For the registration network, it adopts a dual-branch, pyramid coding structure to encode the image pairs to be registered respectively, and extract multi-scale feature information. The feature information from the dual branches is then fed into the designed recursive deformable decoder, which gradually optimizes the deformation field between the image pairs to be registered in a cyclic recursive manner. We use the normalized cross-correlation (NCC) loss function to supervise the deformation between the preoperative and intraoperative image pairs. In addition, in order to highlight the target tissue during surgery, the registration network is equipped with a U-net segmentation sub-network. The segmentation network shares weights with the encoder of the quasi-network to meet the requirements of lightweight and high efficiency in clinical scenarios. The sub-network uses the binary cross entropy (BCE) loss function and the Focal loss function for supervised learning.

[0051] Step S2: Recursive deformable non-rigid registration network training. The network framework is as follows Figure 2 As shown. During the training process, the preoperative image x f And the intraoperative plain scan image xm are input into the designed dual-branch feature encoder ε f (·), and ε m (·). The encoder adopts a pyramid structure, which consists of four coding layers. It learns multi-scale features of the input image and generates features with different scale information. Figure 3 As shown, for the first encoding layer, it contains {Multi-head Attention Layer (MHSA)-Layer Norm Layer (LNorm)-LeakyRelu Activation Layer (LRelu)}, as Figure 3 (a) The subsequent three layers include {downsampling convolution layer (DownConv)-multi-head attention layer (MHSA)-layer norm layer (LNorm)-leakyrelu activation layer (LRelu)}, where the downsampling convolution layer (DownConv) connected at the beginning of the module converts the feature F of the previous layer into l-1 (M l-1 ) The feature scale is reduced and the number of channels is doubled, such as Figure 3 (b) is shown. And the last three layers all use residual connections after the multi-head attention layer to enhance the feature representation ability. The multi-scale features of each encoding layer of the dual-branch encoder are extracted to obtain F l and M l , l = {1, 2, 3, 4}.

[0052] Then the multi-scale feature F l and M l The input is fed into the designed recursive deformable decoder. The decoder has four recursive decoding layers, corresponding to the four feature encoding layers designed in the encoder. The decoding layer designed by this invention consists of a feature fusion module, a deformation estimation module, and a deformation aggregation module, which continuously learns and generates the deformation field θ1 under the scale feature in a recursive manner.

[0053] Specifically, in the fourth decoder layer, features F4 and M4 are fed into the feature fusion module, which consists of two {Multi-Head Attention Layer (MHSA)-Layer Norm Layer (LNorm)-LeakyRelu Activation Layer (LRelu)}, and the fused feature E4 is obtained, as shown in Figure 3 (c). The process is expressed as:

[0054] E4=f([F4,M4])

[0055] Where f((.) represents the feature fusion module, and [.] represents the cascade of the two along the channel dimension.

[0056] Subsequently, the fused features E4 are fed into the deformation estimation module. This module consists of a multi-head attention layer (MHSA) and a differential homomorphic layer (Diff), which aims to learn the representation of the fused features and generate the deformation field θ4, as shown in Figure 3 (d) The process can be expressed as:

[0057] θ4=g(E4)

[0058] In addition, we use a recursive loop to continuously optimize the predicted deformation field. The process is expressed as:

[0059]

[0060]

[0061]

[0062] In the first decoder layer, D4 is E4, i represents the number of cycles, i = 1, 2, ... k. k = 5 but not limited to 5. * represents the deformation operation, which is implemented by the STN network. fusion (.) represents the designed deformation aggregation module, such as Figure 3 (d), which is expressed as:

[0063]

[0064] Then we send the final fusion feature E4 and θ4 of this layer to the decoder of the next layer. For the following three layers, we use 1 (1ayer) to represent the number of layers. The process of recursively predicting the deformation field of each layer can be expressed as:

[0065]

[0066]

[0067]

[0068] Among them D l+1 , 1=3, 2, 1 represents the aggregated feature E obtained by the previous decoding layer l+1 fusion(.) is expressed as:

[0069]

[0070] This method establishes a pyramidal hierarchical structure to gradually represent and model the multi-scale information of preoperative and intraoperative plain scan images and gradually decodes them. It also uses a recursive optimization method to continuously perform fine-grained optimization on the deformation field. Finally, we apply the deformation field θ4 learned by the last decoding layer to the intraoperative image X m, and the NCC loss function is used to supervise the deformation field θ4 between the preoperative enhanced image and the intraoperative plain scan image. The NCC loss function is expressed as follows:

[0071]

[0072] Where pi represents a patch in the image, whose size is 9x9. as well as It represents the average value within this patch.

[0073] Step S3: Target tissue segmentation network training. For the registered intraoperative plain scan image x n , a segmentation network based on U-net is used to identify the target tissue in the image. Specifically, the encoder of U-net is combined with the encoder ε of the registration network f (·) Shared weights are used to simultaneously achieve the network's registration and segmentation functions, meeting the requirements of lightweight and efficient clinical scenarios. Finally, the true label is used to calculate the loss of the decoded predicted segmentation mask, and a binary cross-entropy loss function is used to implement supervised learning. In addition, considering that the content of lesion tissue in the entire image is small, in order to balance the quantitative relationship between positive samples and negative samples, an additional FocalLoss is added as a constraint to fully detect the lesion tissue. The process can be expressed as:

[0074] L seg =L BCE (y, p)+αL Focal (y, p)

[0075] Among them, y is the pixel value of the true label, p is the predicted value of the model, and α is the weight coefficient used to balance the contribution of the Focal loss function.

[0076] Step S4: Deployment of the registration network and preparation of the preoperative network. Deploy the trained network to prepare for real-time image registration during surgery.

[0077] Step S5: Intraoperative registration and lesion display. The intraoperative plain scan image and preoperative fixed image are input into the trained registration network, and the intraoperative plain scan image and preoperative enhanced image are registered, and the lesion area is displayed in color to assist the doctor in the operation.

[0078] (1) Preoperative data collection to train a recursive deformable non-rigid registration network;

[0079] (2) After the trained network is configured, the intraoperative network will realize the registration of the intraoperative plain scan image and the preoperative enhanced image to assist the doctor in the operation.

[0080] The above description is merely a preferred embodiment of the present invention and does not limit the present invention in any way. Any person skilled in the art who, without departing from the scope of the present invention, makes any equivalent substitution, modification, or other changes to the technical solution and technical content disclosed in the present invention shall be deemed to be within the scope of the present invention and still fall within the scope of protection of the present invention.

Claims

1. A target tissue highlighting method based on a recursive deformable non-rigid registration network, characterized in that: The steps include: S1. Collection and enhancement of training datasets; S2, recursive deformable non-rigid pyramid registration network training; S3, target tissue segmentation network training; S4, deployment of the registration network and preparation of the preoperative network; S5, real-time registration; The method of step S1 is as follows: a data augmentation strategy is used to obtain preoperative and intraoperative CT image pairs as training data for the method. First, preoperative enhanced CT images are collected. A traditional intensity-based image registration method is used to design n deformation fields φ. The deformation field used for data augmentation is arbitrarily selected from these n deformation fields. Thin plate spline interpolation is used to simulate small changes at a specific stage. Finally, the preoperative enhanced image and the intraoperative deformable image obtained by data augmentation are input into the registration network as a registration image pair. Manually mark whether there is target tissue in the image to be registered and obtain the segmentation mask of the corresponding target tissue; The method of step S2: During the training process, the preoperative image and images deformed during surgery Input into the designed dual-branch feature encoder respectively ,as well as Among them, features with different scale information are generated, which are recorded as as well as , k={1,2,3,4}, and then input it into the designed recursive deformable decoder to continuously learn and generate the deformation field under the scale feature , the deformation field of the second-level scale feature is obtained by recursive loop , deformation field Then, the learned deformation field is transformed into Application to preoperative images , generate postoperative images , and finally adopt the normalized cross-correlation loss For evaluation of preoperative images and postoperative images The similarities between The expression is as follows: ； Where pi represents a patch in the image, whose size is 9x9. as well as It represents the average pixel value in this patch; The method of step S3: for the registered intraoperative plain scan image ,Use a segmentation network based on U-net to identify target tissue conditions within the image; Combine the encoder of U-net with the encoder of the registration network The weights are shared, and the loss is calculated for the segmentation mask after decoding and prediction of the true label. The binary cross entropy loss function is used to implement supervised learning. In addition, a Focal Loss is added as a constraint term. The process is expressed as: ； Where y is the pixel value of the true label, p is the predicted value of the model, is the weight coefficient used to balance the contribution of the Focal loss function.

2. The target tissue highlighting method based on a recursive deformable non-rigid registration network according to claim 1, characterized in that: The encoder has dual-branch features and adopts a pyramid structure consisting of four encoding layers.

3. The target tissue highlighting method based on recursive deformable non-rigid registration network according to claim 2, characterized in that: The decoder consists of four recursive decoding layers, which correspond to the four feature encoding layers designed in the encoder. Each decoding module includes three core modules: feature fusion, deformation estimation, and deformation aggregation.

4. The target tissue highlighting method based on a recursive deformable non-rigid registration network according to claim 3, characterized in that: First, the features obtained by the fourth encoding layer as well as Input to the first decoding layer of the decoder, which consists of a feature fusion module, a deformation estimation module, and a deformation aggregation module, and continuously learns to generate the deformation field under the scale feature in a cyclic recursive manner , the deformation field And the features obtained by the third layer encoder as well as It is further input into the decoder of the second layer, and the deformation field of the scale feature of the second layer is obtained recursively. , and repeat this step until the deformation field learned by the fourth decoding layer is generated .

5. The target tissue highlighting method based on recursive deformable non-rigid registration network according to claim 4, characterized in that: The method of step S5 is as follows: inputting the intraoperative plain scan image and the preoperative fixed image into the trained registration network, registering the intraoperative plain scan image with the preoperative enhanced image, and displaying the lesion area in color.

Images