Virtual blood vessel enhancement-based CT-guided paracentesis navigation and early warning method, registration network and model

Abstract

The invention discloses a virtual blood vessel enhancement-based CT (Computed Tomography) guided intraparacentesis navigation and early warning method, a registration network and a model. The method comprises the following steps: acquiring a preoperative CT enhanced image and a corresponding intra-operative plain-scan image, and manually marking an intra-operative blood vessel; constructing and training a multi-granularity perceptual registration network to realize registration of a pre-operative enhanced image and an intra-operative image; a network architecture of nn-Unet is used to identify the distribution condition of blood vessels in the image, and feature encoders of the segmentation network and the registration network are shared; deploying the trained network to prepare for real-time image registration in an operation; outputting a model result to assist doctors in decision making. Under the condition of not using a contrast agent for enhancement, the blood vessel condition in the puncture path can be clearly displayed only according to the registration of the pre-operation enhanced image, safe distance reminding is given, and the method has the advantages of saving time, reducing the bleeding risk, and being simple and practical.

Description

Technical Field

[0001] The present invention relates to medical technology, and specifically relates to a navigation, warning method, registration network and model in CT-guided puncture based on virtual vessel enhancement. Background Art

[0002] In CT-guided puncture surgery, plain scan images are usually used as guiding images. Firstly, they cannot effectively and accurately display the blood vessels during puncture. At the same time, during the final stage of biopsy, the biopsy slot usually pops forward by 1.3 - 2.5 cm. It is more important to observe the situation of the blood vessels that can be touched during the forward extension of the biopsy slot. This process requires the operator to repeatedly flip through the images up and down carefully according to the blood vessel path and check the preoperative enhanced images to avoid touching the blood vessels as much as possible, which is time-consuming and laborious and increases the puncture time. This kind of operation relies on the doctor's own speculation and experience, which has great uncertainty. If the path selection is wrong and the pre-judgment is incorrect and not corrected and prompted in time, it is very likely to cause unnecessary bleeding. Bleeding is one of the most common complications in CT-guided puncture surgery, and in severe cases, it can lead to consequences such as massive bleeding, hemorrhagic shock and even death. In CT-guided puncture surgery, plain scan images are usually used as guiding images, and the blood vessels during puncture cannot be effectively observed. Relying on preoperative images and the operator's speculation, trying to avoid blood vessels in the puncture needle path and the forward extension length of the puncture slot is time-consuming and laborious and increases the puncture time. At the same time, misjudgment of the blood vessel trajectory and position is very likely to cause unnecessary bleeding.

[0003] If during the puncture process, the blood vessel images in the needle path and in front of the predicted forward extension of the biopsy slot can be accurately and real-time displayed and the operator is reminded of the safe distance, it will effectively avoid complications such as bleeding caused by puncturing the blood vessels. Summary of the Invention

[0004] Object of the Invention: The object of the present invention is to provide a navigation, warning method, registration network and model in CT-guided puncture based on virtual vessel enhancement. The multi-granularity perception registration network of the present invention can satisfy the registration of intraoperative plain scan images and preoperative enhanced images in real time and highlight the blood vessel distribution. The present invention can effectively avoid complications such as bleeding caused by puncturing the blood vessels.

[0005] Technical Solution: The navigation, warning method in CT-guided puncture based on virtual vessel enhancement according to the present invention includes the following steps:

[0006] (1) Collection and annotation of the training dataset: Collect preoperative CT enhanced images and their corresponding intraoperative plain scan images, manually mark the intraoperative blood vessels to generate a segmentation mask reflecting the blood vessel distribution, and use this mask to train the blood vessel segmentation network. The collected data is further divided into a training set and a validation set;

[0007] (2) Construction and training of the multi-granularity perception registration network to achieve the registration of preoperative enhanced images and intraoperative images;

[0008] (3) Training of the vascular segmentation network: Using the network architecture of nn-Unet to identify the distribution of blood vessels in the image, and sharing the feature encoders of the segmentation network and the registration network;

[0009] (4) Deploy the trained network to prepare for real-time image registration during surgery;

[0010] (5) Output the model results to assist doctors in decision-making: Input the intraoperative plain scan image and the preoperative fixed image into the trained registration network to achieve the registration of the intraoperative plain scan image and the preoperative enhanced image. At the same time, by highlighting the blood vessels in color, it provides assistance for doctors during surgery.

[0011] Furthermore, step (1): Collection and enhancement of the training dataset: Collect preoperative CT enhanced images and corresponding intraoperative plain scan images, and use a traditional intensity-based image registration method to generate n deformation fields to enhance the morphological diversity of the images. The deformation fields used for data enhancement are randomly selected from these generated deformation fields. To simulate the position offset caused by the free movement of the viscera and the ejection of the biopsy slot, thin plate spline (TPS) interpolation is used to reflect the stagewise small changes. The preoperative enhanced image and the intraoperative deformed image obtained through data enhancement are used as a registration image pair and input into the registration network designed by the present invention. In addition, the blood vessels in the intraoperative images are manually annotated to obtain the segmentation mask of the blood vessel distribution in the intraoperative images. This mask is used to train the vascular segmentation network to achieve the highlighting of blood vessels in the intraoperative images. According to the collected data, it is divided into a training set and a validation set. Among them, the training set is used for the training of this registration network, and the validation set is used to select the registration network weights with superior performance.

[0012] Furthermore, step (2): Construct a multi-granularity perception registration network. This registration network aims to achieve the registration of preoperative enhanced images and intraoperative images. Considering the requirement of accurate identification of blood vessels during surgery, this registration network can achieve multi-granularity feature modeling and more accurately achieve the image registration result.

[0013] Furthermore, step (3): Training of the vascular segmentation network. Use the network architecture of nn-Unet to identify the distribution of blood vessels in the image. Specifically, to simultaneously meet the clinical requirements of registration and blood vessel highlighting, the feature encoders of the segmentation network and the registration network are shared, so as to meet the requirements of lightweight and high efficiency in the clinical scenario.

[0014] Furthermore, step (4): Deployment of the network and preparation of the preoperative network. Deploy the trained network to prepare for real-time image registration during surgery.

[0015] Further, step (5): Output the model result to assist the doctor in making decisions. Input the intraoperative plain scan image and the preoperative fixed image into the trained registration network to achieve the registration of the intraoperative plain scan image and the preoperative enhanced image. At the same time, the blood vessels are highlighted in color to provide assistance for the doctor's surgery.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] (1) The virtual blood vessel enhancement technology method of the present invention can clearly display the large blood vessels and their branches in and in front of the puncture path, and give prompts for key dangerous areas, which can effectively improve the puncture efficiency and significantly reduce the risk of puncture bleeding.

[0018] (2) Without the need to use contrast agent enhancement, the present invention overcomes the disadvantage that it is difficult to clearly observe blood vessels during plain scan in biopsy surgery. Only based on the registration of preoperative enhanced images, it can clearly display the blood vessel conditions in the puncture path in real time, and has the advantages of saving time, reducing the bleeding risk, and being simple and practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic flow chart of the method of the present invention;

[0020] Figure 2 It is a schematic diagram of the multi-granularity perception module architecture;

[0021] Figure 3 It is a schematic diagram of the WCE-Attention module. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be further described below.

[0023] The navigation and early warning method during CT-guided puncture based on virtual blood vessel enhancement in this embodiment includes the following steps:

[0024] Step S1: Collection and annotation of the training dataset. Collect preoperative CT enhanced images and their corresponding intraoperative plain scan images, and generate n = 9 deformation fields through a traditional intensity-based image registration method To enhance the diversity of image morphology. During data augmentation, the deformation field is randomly selected from these generated fields. To simulate the displacements caused by organ activities and biopsy slot ejection, thin plate spline (TPS) interpolation is used to reflect small position changes at specific stages. The pre-operative enhanced images and the intra-operative deformed images generated by data augmentation are paired and input into the designed registration network for processing. In addition, the intra-operative blood vessels are manually marked to generate a segmentation mask reflecting the blood vessel distribution, and this mask is used to train the blood vessel segmentation network to achieve highlighting of the intra-operative blood vessels. The collected data is further divided into a training set and a validation set. The training set is used to optimize the registration network, and the validation set is used to evaluate and select the optimal network weights.

[0025] Step S2: Training of the multi-granularity perception registration network.

[0026] (1) Network training process:

[0027] As Figure 2 shown, during the training process of the designed multi-granularity perception registration network, the pre-operative enhanced CT image I f and the plain scan image I m are input into the designed multi-scale encoder PVT encoder. This encoder adopts the PVT architecture based on Vision Transformer and consists of four layers of pyramid-structured encoding layers. For each encoding layer, features of different scales and feature dimensions are extracted in sequence from shallow to deep. For the enhanced image I f , the feature is extracted. For the plain scan image I m , the feature F f and F m have the same feature scale. Taking F f as an example, F f the first-layer encoding layer feature where H and W are the feature space sizes and 8 is the number of feature channels. Thereafter, the feature of each encoding layer is successively halved in spatial scale and doubled in feature dimension, so as to extract feature expressions of different knowledge granularities and dimensions.

[0028] Subsequently, the multi-scale features F f and F m are fed into the designed multi-granularity perception registration decoder (Coarse-to-fine Registration Decoder) to fully exploit information of different granularities. As Figure 2 shown, the designed decoder contains multiple window-based feature correlation enhancement modules (WCE-Attention) to perform four-level multi-granularity perception registration on the extracted features. For the first level, first and are fed into the designed WCE - Attention module to explore their spatial correlation. The obtained feature R 1 is then fed into a deformable registration head, and the deformation field is predicted The registration head includes a 3x3 convolutional layer and a ReLU activation function layer. Subsequently, in the second - level registration step, the deformation field is used to guide the prediction of the deformation field at this level, and two WCE - Attention modules are designed. Specifically, the deformation field is used to perform a deformation operation. This operation means After that, the deformed feature and are input into the first WCE - Attention module to capture the spatial correlation between the registered images. Subsequently, the relevant feature R at the first level 1 is used to guide the obtained feature - related feature S 2 , and it is fed into the second WCE - Attention module to capture the correlation of features at different levels. Then, the second - level feature R 2 is obtained and input into a deformable registration head to generate the corresponding intermediate deformation field. To further enhance the feature extraction ability, through interpolation operation, the deformation field is connected in residual with the intermediate deformation field of the second - level feature, and finally the optimized second - level feature deformation field is obtained. After that, the above steps are repeated to generate the deformation fields at the third and fourth levels respectively as Figure 2 shown. is used as the prediction result of the final deformation field of the model. Using to perform a spatial deformation operation on the original non - enhanced image x m , and the NCC loss function is used to supervise the learning of the deformation field between the pre - operative enhanced image and the intra - operative non - enhanced image. The NCC loss function is expressed as follows:

[0029]

[0030] where pi represents a patch in the image, with a size of 9x9, and represent the average values within this patch.

[0031] (2) Introduction to key modules:

[0032] WCE - Attention Module: This module is responsible for constructing enhanced image features and spatial correlations of non - contrast - enhanced image features. As Figure 3 shown, for the two different input feature layers F 1 and F 2 , first, they are input into a correlation layer. This correlation layer concatenates F 1 and F 2 along the channel dimension and uses a MLP (Multi - layer perceptron) to fuse the features of the two, obtaining F 3 . Then, F 1 , F 2 and F 3 are concatenated along the channel dimension and passed through a 3x3 convolutional layer to obtain the correlation feature F corr . It is sent to a LayerNorm for normalization. Then, it is respectively sent into window pooling modules with 3 different window sizes. Taking the window pooling module with a window size of 3x3 as an example. This module contains a max - pooling layer with a window size of 3x3. It splits the input features into multiple windows with a spatial size of 3x3 and aggregates the windows into corresponding feature vectors W 3×3 to capture feature information at different spatial positions. Similarly, for the window pooling modules with window sizes of 5x5 and 7x7, corresponding feature vectors W 5×5 and W 7×7 are also aggregated. Subsequently, for adjacent aggregated features W 3×3 and W 5×5 , they are input into a cross - attention layer to fuse the features of the two, obtaining the feature F 4 . Similarly, for adjacent aggregated features W 5×5 and W 7×7 , a cross - attention layer is also used to perform feature interaction and fusion on the two, obtaining the feature F 5 . Subsequently, a cross - attention layer is used again to fuse the interacted features F4 and F 5 to obtain the feature F 6 that fuses information of different granularities. To enhance the feature extraction ability of the network, residual connections are used to connect the correlation features F corr and F 6 . And it is sent into a MLP to obtain the final interactive feature F out .

[0033] Step S3: Training of the vascular segmentation network. The network architecture of nn-Unet is used to identify the distribution of blood vessels in the image. Specifically, the encoder parameters of nn-Unet are shared with the PVT encoder parameters of the above registration network to meet the requirements of lightweight and high efficiency in clinical scenarios. The segmentation network is supervised by manually annotated vascular masks, and the binary cross-entropy (BCE) loss function and the intersection over union (IoU) loss function are used for learning. The final loss function of this network is expressed as follows:

[0034] L = L BCE (P s , Y s ) + γL IoU (P s , Y s ) + λL NCC (x m , x f )

[0035] where P s is the segmentation result predicted by the network, and Y s is the ground truth label of the segmentation mask. γ and λ are loss coefficients used to balance the contribution weights of multiple losses.

[0036] After the network training is completed, the model is debugged through the validation set, and finally a model that takes into account both registration and segmentation performance is obtained.

[0037] Step S4: Deployment of the network and preparation for the preoperative network. The trained network is deployed to prepare for real-time image registration during surgery.

[0038] Step S5: Output the model results to assist doctors in decision-making. The intraoperative plain scan image and the preoperative fixed image are input into the trained registration network to achieve the registration of the intraoperative plain scan image and the preoperative enhanced image. At the same time, the blood vessels are highlighted in color to provide assistance for doctors during surgery.

[0039] Usage method: Collect data before surgery to train the invented multi-granularity perception registration network. After configuring the trained network, the intraoperative network will achieve the registration of the intraoperative plain scan image and the preoperative enhanced image to assist doctors in surgery.

[0040] The above is only the preferred embodiment of the present invention and does not impose any limitation on the present invention. Any person skilled in the art within the technical field of the present invention, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification and other changes to the technical solution and technical content disclosed by the present invention, all of which fall within the content of the technical solution of the present invention and are still within the protection scope of the present invention.

Claims

1. A navigation and early warning method for puncture under CT guidance based on virtual vascular enhancement, characterized in that: The steps include: (1) Collection and annotation of training data sets: preoperative enhanced CT images and their corresponding intraoperative plain scan images were collected, intraoperative blood vessels were manually marked, segmentation masks reflecting blood vessel distribution were generated, and the vascular segmentation network was trained using this mask. The collected data were further divided into training and validation sets. (2) Construction and training of a multi-granularity perceptual registration network to achieve registration of preoperative enhanced images with intraoperative images; (3) Training of the vascular segmentation network: Use the nn-Unet network architecture to identify the distribution of blood vessels in the image, and share the feature encoders of the segmentation network and the registration network; (4) Deploy the trained network to prepare for real-time image registration during surgery; (5) Output model results to assist doctors in making decisions: Input the intraoperative plain scan images and preoperative fixed images into the trained registration network to achieve registration between the intraoperative plain scan images and the preoperative enhanced images. At the same time, the blood vessels are highlighted in color to assist doctors in surgery.

2. The method for navigation and early warning during puncture under CT guidance based on virtual vascular enhancement according to claim 1, characterized in that: The construction and training process of step (2): the preoperative enhanced CT image I f And the plain scan image I m Input into the designed multi-scale encoder PVT encoder to extract feature expressions of different knowledge granularity and dimensions. Then, the multi-scale feature F f With F m The deformation fields of the first, second, third and fourth levels are generated respectively, and the deformation field of the fourth level is sent to the Coarse-to-fine Registration Decoder to generate the deformation fields of the first, second, third and fourth levels respectively. As the prediction result of the final deformation field of the model, For the original plain scan image x m Perform spatial deformation operations and use the NCC loss function to calculate the deformation field between the preoperative enhanced image and the intraoperative plain scan image. Perform supervised learning.

3. The method for navigation and early warning during puncture under CT guidance based on virtual vascular enhancement according to claim 2, characterized in that: The multi-scale encoder PVT encoder adopts the PVT architecture based on Vision Transformer, which consists of four layers of pyramid structure coding layers. For each coding layer, features of different scales and feature dimensions are extracted from shallow to deep. f , extract the features For the plain scan image I m , extract features to get F f With F m With the same characteristic scale, F f For example, F f The first encoding layer features Among them, H and W are the feature space sizes, and 8 is the number of feature channels. After that, the feature spatial scale of each coding layer is halved and the feature dimension is doubled, so as to extract feature expressions of different knowledge granularity and dimensions.

4. The method for navigation and early warning during puncture under CT guidance based on virtual vascular enhancement according to claim 3, characterized in that: The third and fourth level deformation fields Acquisition method: The multi-granularity perceptual registration decoder Coarse-to-fine Registration Decoder contains multiple window-based feature correlation enhancement modules WCE-Attention, which performs four levels of multi-granularity perceptual registration on the extracted features. For the first level, first as well as The feature R1 is then sent to the designed WCE-Attention module, and then sent to a deformable registration head to predict the deformation field Then, in the second level registration step, the deformation field We guide the prediction of this hierarchical deformation field and design two WCE-Attention modules. In particular, we use the deformation field right Perform a deformation operation, which means Then, the deformed features as well as The first level of feature correlation R1 is used to guide the obtained feature correlation feature S2, which is then sent to the second WCE-Attention module to capture the correlation of features at different levels. Then, the second level feature R2 is obtained and input into a deformable registration head to generate the corresponding intermediate deformation field. To further enhance the feature extraction capability, the deformation field is interpolated. Perform residual connection with the intermediate deformation field of the second-level features to finally obtain the optimized deformation field of the second-level features After that, repeat the above steps to generate the third and fourth level deformation fields respectively.

5. The method for navigation and early warning during puncture under CT guidance based on virtual vascular enhancement according to claim 4, characterized in that: The NCC loss function is expressed as follows: Where pi represents a patch in the image, whose size is 9x9. as well as It represents the average value within this patch.

6. The method for navigation and early warning during puncture under CT guidance based on virtual vascular enhancement according to claim 4, characterized in that: The registration head contains a 3x3 convolutional layer and a ReLU activation function layer.

7. The method for navigation and early warning during puncture under CT guidance based on virtual vascular enhancement according to claim 6, characterized in that: The WCE-Attention module is responsible for constructing the spatial correlation of enhanced image features and plain scan image features. For the two different feature layers F1 and F2 input, they are first input into a correlation layer, which concatenates F1 and F2 along the channel direction and uses an MLP to fuse the two features to obtain F3. Then, F1, F2 and F3 are concatenated along the channel direction and passed through a 3x3 convolution layer to obtain the correlation feature F corr , and send it to a layer of LayerNorm for normalization. Then, it is sent to three window pooling modules with different window sizes. Taking the window pooling module with a window size of 3x3 as an example, this module contains a max-pooling layer with a window size of 3x3, which splits the input features into multiple windows with a spatial size of 3x3 and aggregates the windows into the corresponding feature vector W 3×3 To capture the feature information of different spatial positions, similarly, for the 5x5 and 7x7 window pooling modules, the corresponding feature vectors W are also aggregated 5×5 and W 7×7 , then, for the adjacent aggregated features W 3×3 and W 5×5 , input it into a cross-attention layer, fuse the two features, and get feature F4. Similarly, for the adjacent aggregate feature W 5×5 and W 7×7 , and the cross attention layer is also used to fuse the two features interactively to obtain feature F5. Subsequently, a cross attention layer is used again to fuse the interactive features F4 and F5 to obtain feature F6 that fuses information of different granularities. In order to enhance the feature extraction ability of the network, residual connection is used to fuse the correlation feature F corr And F6 are connected and sent to a layer of MLP to obtain the final interaction feature F out .

8. The method for navigation and early warning during puncture under CT guidance based on virtual vascular enhancement according to claim 7, characterized in that: The training method of the blood vessel segmentation network in step (3) is as follows: the encoder parameters of nn-Unet are shared with the PVT encoder parameters of the above-mentioned registration network, so as to meet the requirements of lightweight and high efficiency in clinical scenarios, and the segmentation network is supervised by using the manually annotated blood vessel mask, and the binary cross entropy BCE loss function and the intersection over union (IoU) loss function are used for learning. The final loss function of this network is expressed as follows: L=L BCE (P s ,Y s )+γL IoU (P s ,Y s )+λL Ncc (x m ,x f ) Among them, P s is the segmentation result predicted by the network, Y s is the true label of the segmentation mask, γ and λ are loss coefficients, which are used to balance the contribution weights of multiple losses.

9. A registration network for use in the method of claims 1-8.

10. A WCE-Attention module used in the method of claims 1-8.

Images