A navigation and early warning method based on virtual blood vessel enhancement in CT-guided puncture, and a registration network and model

Abstract

The application discloses a kind of based on virtual blood vessel enhancement's CT guided puncture in navigation, early warning method and registration network and model.The method includes: collecting preoperative CT enhancement image and its corresponding intraoperative plain scan image, and manually marking intraoperative blood vessel;Multi-granularity perception registration network construction and training are realized Registration of preoperative enhancement image and intraoperative image;The network architecture of nn-Unet is used to identify the distribution of blood vessels in the image, and the feature encoders of the segmentation network and the registration network are shared;The trained network is deployed to prepare for real-time image registration during surgery;Output model results to assist doctors in decision-making.The application can clearly display the blood vessel situation in the puncture path and give a safe distance reminder without using contrast agent enhancement, has the advantages of saving time, reducing the risk of bleeding, simple and practical.

Description

TECHNICAL FIELD

[0001] The present application relates to the medical technology field, and particularly relates to a navigation and early warning method based on virtual blood vessel enhancement in CT-guided puncture, and a registration network and model. BACKGROUND

[0002] In the CT-guided puncture surgery, a plain scan image is usually used as a guide image. Firstly, the blood vessels in the puncture path cannot be effectively and accurately displayed. In addition, the biopsy slot usually pops out 1.3-2.5 cm forward in the last stage of biopsy. The observation of the blood vessels during the forward extension of the biopsy slot is more important. This process requires the operator to repeatedly flip through the images up and down and check the preoperative enhanced images according to the blood vessel path, so as to avoid the blood vessels as much as possible, which is time-consuming and laborious, and increases the puncture time. This operation depends on the doctor's own speculation and experience, and has great uncertainty. If the path selection is wrong and the prediction is wrong, and no timely correction and prompt is given, unnecessary bleeding may be caused. Bleeding is one of the most common complications in CT-guided puncture surgery. In severe cases, it can lead to massive bleeding, hemorrhagic shock and even death. In the CT-guided puncture surgery, a plain scan image is usually used as a guide image, and the blood vessel situation in the puncture path cannot be effectively observed. The operator depends on the preoperative image and speculation to avoid blood vessels in the puncture needle path and the forward extension length of the biopsy slot, which is time-consuming and laborious, and increases the puncture time. At the same time, the judgment error of the blood vessel shape and position may cause unnecessary bleeding.

[0003] If the blood vessel image in the needle path and in front of the biopsy slot during the puncture process can be accurately and timely displayed, and the operator is reminded of the safe distance, the bleeding and other complications caused by puncturing the blood vessels can be effectively avoided. SUMMARY

[0004] The present application aims to provide a navigation and early warning method based on virtual blood vessel enhancement in CT-guided puncture, and a registration network and model. The multi-granularity registration network of the present application can satisfy the registration of the intraoperative plain scan image and the preoperative enhanced image in real time, and highlight the blood vessel distribution. The present application can effectively avoid the bleeding and other complications caused by puncturing the blood vessels.

[0005] TECHNICAL SCHEME: The navigation and early warning method based on virtual blood vessel enhancement in CT-guided puncture provided by the present application comprises the following steps:

[0006] (1) Collection and labeling of training data set: collect preoperative CT enhanced images and their corresponding intraoperative plain scan images, artificially mark the intraoperative blood vessels, generate a segmentation mask reflecting the blood vessel distribution, and train a blood vessel segmentation network with the mask. The collected data is further divided into a training set and a verification set;

[0007] (2) Construction and training of multi-granularity perception registration network to realize registration of preoperative enhanced images and intraoperative images;

[0008] (3) Training of blood vessel segmentation network: using the network architecture of nn-Unet to identify the distribution of blood vessels in the image, and sharing the feature encoders of the segmentation network and the registration network;

[0009] (4) Deploying the trained network to prepare for real-time image registration during surgery;

[0010] (5) Output model results to assist doctors in decision-making: inputting the intraoperative plain scan image and the preoperative fixed image into the trained registration network to realize the registration of the intraoperative plain scan image and the preoperative enhanced image, and at the same time, highlighting the blood vessels through color to provide assistance for the doctor's surgery.

[0011] Further, step (1): collection and enhancement of training data set: collecting preoperative CT enhanced images and corresponding intraoperative plain scan images, and generating n deformation fields using traditional image registration method based on intensity To enhance the morphological diversity of the image. The deformation field used for data enhancement is randomly selected from these generated deformation fields. To simulate the position offset caused by the free activity of the internal organs and the ejection of the biopsy slot, thin plate spline (TPS) interpolation is used to reflect the small changes in stages. The preoperative enhanced image and the intraoperative deformation image obtained through data enhancement are input into the registration network designed in the application as a registration image pair. In addition, the intraoperative blood vessels are manually annotated to obtain a segmentation mask of the blood vessel distribution in the intraoperative image. The mask is used to train the blood vessel segmentation network to realize the highlighting of the intraoperative blood vessels. According to the collected data, it is divided into a training set and a validation set. The training set is used for training the registration network, and the validation set is used to select the superior registration network weight.

[0012] Further, step (2): construction of multi-granularity perception registration network. The registration network aims to realize the registration of preoperative enhanced images and intraoperative images. Considering the requirement of accurate identification of blood vessels during surgery, the registration network can realize multi-granularity feature modeling and more accurately realize the image registration result.

[0013] Further, step (3): training of blood vessel segmentation network. The network architecture of nn-Unet is used to identify the distribution of blood vessels in the image. Specifically, to meet the clinical requirements of registration and blood vessel highlighting, the feature encoders of the segmentation network and the registration network are shared, thereby meeting the requirements of lightweight and efficiency in the clinical scenario.

[0014] Further, step (4): deployment of the network and preparation of the preoperative network. The trained network is deployed to prepare for real-time image registration during surgery.

[0015] Furthermore, step (5): outputting the model results to assist the doctor in making decisions. The intraoperative plain scan image and the preoperative fixed image are input into the trained registration network to achieve registration between the intraoperative plain scan image and the preoperative enhanced image. At the same time, the blood vessels are highlighted in color to assist the doctor in the operation.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] (1) The virtual blood vessel enhancement technology of the present invention can clearly display the large blood vessels and their branches in and ahead of the puncture path, and provide prompts for key dangerous areas, which can effectively improve the puncture efficiency and significantly reduce the risk of puncture bleeding.

[0018] (2) The present invention overcomes the shortcoming that it is difficult to clearly observe blood vessels during biopsy using plain scans without the need for contrast agent enhancement. It can clearly display the blood vessel conditions in the puncture path in real time based on the registration of preoperative enhanced images only, thus saving time, reducing the risk of bleeding, and being simple and practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the process of the present invention;

[0020] Figure 2 This is a schematic diagram of the multi-granularity perception module architecture;

[0021] Figure 3 Schematic diagram of the WCE-Attention module. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be further described below.

[0023] The method for CT-guided puncture navigation and early warning based on virtual blood vessel enhancement in this embodiment includes the following steps:

[0024] Step S1: Collection and annotation of training data set. Collect preoperative CT enhanced images and their corresponding intraoperative plain scan images, and generate n=9 deformation fields through traditional intensity-based image registration method. Used to enhance the diversity of image morphology. During the data augmentation process, the deformation field is randomly selected from these generated fields. In order to simulate the displacement caused by organ activity and biopsy slot ejection, thin plate spline (TPS) interpolation is used to reflect small position changes in specific stages. The preoperative enhanced image and the intraoperative deformed image generated by data augmentation are paired and input into the designed registration network for processing. In addition, the intraoperative blood vessels are manually labeled to generate a segmentation mask reflecting the blood vessel distribution, and this mask is used to train the vascular segmentation network to achieve intraoperative blood vessel highlighting. The collected data is further divided into a training set and a validation set. The training set is used to optimize the registration network, and the validation set is used to evaluate and select the network weights with the best performance.

[0025] Step S2: Training of multi-granularity-aware registration network.

[0026] (1) Network training process:

[0027] like Figure 2 As shown in the figure, during the training process of the designed multi-granularity perception registration network, the preoperative enhanced CT image I f And plain scan image I m Input into the designed multi-scale encoder PVT encoder. The encoder adopts the PVT architecture based on Vision Transformer, which consists of four layers of pyramid structure coding layers. For each coding layer, features of different scales and feature dimensions are extracted from shallow to deep. f , extract the features For the plain scan image I m , extract features to get F f With F m Have the same characteristic scale. f For example, F f The first encoding layer features Where H and W are the feature space sizes, and 8 is the number of feature channels. After that, the feature spatial scale of each coding layer is halved and the feature dimension is doubled, thereby extracting feature expressions of different knowledge granularity and dimensions.

[0028] Then, the multi-scale feature F f With F m The images are sent to the designed multi-granularity perceptual registration decoder (Coarse-to-fine Registration Decoder) to fully exploit the information of different granularities. Figure 2 As shown in the figure, the designed decoder contains multiple window-based feature correlation enhancement modules (WCE-Attention) to perform four-level multi-granularity perceptual registration on the extracted features. For the first level, first and The obtained features R1 are then fed into a deformable registration head to predict the deformation field The registration head contains a 3x3 convolutional layer and a ReLU activation function layer. Then, in the second level registration step, the deformation field guides the prediction of this level deformation field and two WCE-Attention modules are designed. Specifically, the deformation field is applied to . This operation represents Then, the deformed features and are input into the first WCE-Attention module to capture the spatial correlation between the registered images. Then, the obtained feature correlation features S2 are guided by the first level correlation features R1 and fed into the second WCE-Attention module to capture the correlation of different level features. Next, the second level features R2 are obtained and input into a deformable registration head to generate the corresponding intermediate deformation field. To further enhance the feature extraction capability, the deformation field is residually connected with the intermediate deformation field of the second level features, and the optimized second level deformation field is finally obtained. After that, the above steps are repeated to generate the third and fourth level deformation fields as shown in Figure 2 . The is taken as the prediction result of the final deformation field of the model. The original plain scan image x m is spatially deformed using , and the deformation field between the preoperative enhanced image and the intraoperative plain scan image is supervised to learn using the NCC loss function. The NCC loss function is expressed as follows:

[0029]

[0030] where pi represents a patch within the image, with a size of 9x9, and represents the average value within the patch.

[0031] (2) Key module introduction:

[0032] WCE-Attention module: This module is responsible for building the spatial correlation of enhanced image features and plain scan image features. As shown inFigure 3 As shown in the figure, for the two different input feature layers F1 and F2, they are first input into a correlation layer. The correlation layer concatenates F1 and F2 along the channel direction and uses an MLP (Multi-layer perceptron) to fuse the two features to obtain F3. Then, F1, F2 and F3 are concatenated along the channel direction and passed through a 3x3 convolution layer to obtain the correlation feature F corr . Send it to a layer of LayerNorm for normalization. Then, send it to three window pooling modules with different window sizes. Take the window pooling module with a window size of 3x3 as an example. This module contains a max-pooling layer with a window size of 3x3. It splits the input features into multiple windows of spatial size 3x3 and aggregates the windows into corresponding feature vectors W 3×3 To capture the feature information of different spatial positions. Similarly, for the 5x5 and 7x7 window pooling modules, the corresponding feature vectors W are also aggregated. 5×5 and W 7×7 . Then, for the adjacent aggregated features W 3×3 and W 5×5 , input it into a cross-attention layer, and fuse the two features to obtain feature F4. Similarly, for the adjacent aggregated feature W 5×5 and W 7×7 , and the cross attention layer is also used to interactively fuse the two features to obtain feature F5. Subsequently, a cross attention layer is used again to fuse the interactive features F4 and F5 to obtain feature F6 that fuses information of different granularities. In order to enhance the feature extraction ability of the network, residual connection is used to fuse the correlation feature F corr And F6 are connected. And sent to a layer of MLP to obtain the final interaction feature F out .

[0033] Step S3: Training of the vascular segmentation network. The nn-Unet network architecture is used to identify the distribution of blood vessels in the image. Specifically, the encoder parameters of nn-Unet are shared with the PVT encoder parameters of the above-mentioned registration network to meet the requirements of lightweight and high efficiency in clinical scenarios. The segmentation network is supervised by manually annotated vascular masks, and the binary cross entropy (BCE) loss function and the intersection over union (IoU) loss function are used for learning. The final loss function of this network is expressed as follows:

[0034] L=L BCE (P s , Ys )+γL IoU (P s ,Y s )+λL NCC (x m ,x f )

[0035] wherein, P s is the segmentation result predicted by the network, Y s is the real label of the segmentation mask. γ and λ are loss coefficients, used to balance the contribution weight of multiple losses.

[0036] After the network is trained, the model is debugged through the validation set, and finally a model that takes into account the registration and segmentation performance is obtained.

[0037] Step S4: deployment of the network and implementation of the preoperative network preparation. The trained network is deployed to prepare for real-time image registration during surgery.

[0038] Step S5: output the model results to assist the doctor in decision-making. The intraoperative plain scan image and the preoperative fixed image are input into the trained registration network to realize the registration of the intraoperative plain scan image and the preoperative enhanced image. At the same time, the blood vessels are displayed in color and highlighted to assist the doctor in surgery.

[0039] Method for use: collect data before surgery to train the invented multi-granularity perception registration network. After the trained network is configured, the intraoperative network will realize the registration of the intraoperative plain scan image and the preoperative enhanced image to assist the doctor in surgery.

[0040] The above is only the preferred embodiment of the present application, and does not limit the present application in any way. Any person skilled in the art, without departing from the scope of the technical solutions of the present application, can make any form of equivalent replacement or modification of the technical solutions and technical contents disclosed in the present application, and still belong to the protection scope of the present application.

Claims

1. A method for navigation and early warning in CT-guided puncture based on virtual blood vessel enhancement, characterized in that, The steps include: (1) Collect preoperative CT enhanced images and their corresponding intraoperative CT plain scan images, manually mark the intraoperative blood vessels, generate a segmentation mask reflecting the vascular distribution, and use this mask to train the vascular segmentation network; (2) Constructing a multi-granularity perception registration network, the network comprising: The Vision Transformer-based PVT encoder is used to extract multi-scale features of preoperative enhanced images and intraoperative plain scan images; A multi-granularity perceptual registration decoder, the decoder comprising multiple WCE-Attention modules; (3) training a vessel segmentation network using the nn-Unet architecture and sharing the encoder parameters of the nn-Unet with the PVT encoder parameters; (4) Deploy the trained network in an intraoperative real-time registration system; (5) Inputting the intraoperative plain scan image and the preoperative enhanced image into the network to achieve registration and display the blood vessels in a color-highlighted manner to assist the doctor in making decisions. The WCE-Attention module is responsible for constructing the spatial correlation of enhanced image features and plain scan image features. as well as , first input it into a correlation layer, which will as well as Cascade along the channel direction and use an MLP to fuse the two features to obtain , and then, Cascade along the channel direction and pass through a 3x3 convolution layer to obtain the correlation feature , send it to a layer of LayerNorm for normalization, and then send it to three window pooling modules with different window sizes. Taking the window pooling module with a window size of 3x3 as an example, this module contains a max-pooling layer with a window size of 3x3, which splits the input features into multiple windows of spatial size 3x3 and aggregates the windows into corresponding feature vectors To capture the feature information of different spatial positions, similarly, for the 5x5 and 7x7 window pooling modules, the corresponding feature vectors are also aggregated as well as , then, for adjacent aggregate features as well as , input it into a cross-attention layer, fuse the two features, and get the feature , similarly, for adjacent aggregate features as well as , and also use the cross attention layer to interactively fuse the two features to obtain the feature , and then, a cross attention layer is used again to integrate the interacted features as well as Fusion is performed to obtain features that combine information of different granularities In order to enhance the feature extraction capability of the network, residual connection is used to convert the correlation features as well as Connect and send it to a layer of MLP to obtain the final interaction features .

2. The method of claim 1, wherein the method is based on virtual vessel augmentation for CT-guided needle insertion. The construction and training process of step (2): the preoperative enhanced CT image and plain scan images Input into the designed multi-scale encoder PVT encoder to extract feature expressions of different knowledge granularity and dimensions, and then convert the multi-scale features into and The deformation fields of the first, second, third and fourth levels are generated respectively, and the deformation field of the fourth level is converted into As the prediction result of the final deformation field of the model, we use The original plain scan image Perform spatial deformation operations and use the NCC loss function to analyze the deformation field between the preoperative enhanced image and the intraoperative plain scan image. Perform supervised learning. 3.The virtual blood vessel enhancement based CT guided puncture navigation and early warning method according to claim 2, characterized in that, The multi-scale encoder PVT encoder adopts the PVT architecture based on Vision Transformer, which consists of four layers of pyramid structure coding layers. For each coding layer, features of different scales and feature dimensions are extracted from shallow to deep, and the preoperative enhanced CT image is , extract the features ; For plain scan images , extract features to get , and have the same characteristic scale, For example, The first encoding layer features , where H and W are the feature space sizes, and 8 is the number of feature channels. After that, the feature spatial scale of each coding layer is halved and the feature dimension is doubled, thereby extracting feature expressions of different knowledge granularity and dimensions.

4. The method of claim 3, wherein the method is based on virtual vessel augmentation for CT-guided needle insertion. third and fourth level deformation fields , The method for obtaining: the multi-granularity perception registration decoder Coarse-to-fine Registration Decoder contains multiple window-based feature correlation enhancement modules WCE-Attention, which performs four levels of multi-granularity perception registration on the extracted features. For the first level, the first and second levels of features and are input into the designed WCE-Attention module, and the obtained feature is then input into a deformable registration head to predict the deformation field . Subsequently, in the second level registration step, the deformation field is used to guide the prediction of the deformation field of this level, and two WCE-Attention modules are designed to use the deformation field to perform deformation operations on , which represents * , and then the deformed features and are input into the first WCE-Attention module to capture the spatial correlation between the registration images. Subsequently, the first level of correlation features is used to guide the obtained feature correlation features , which are then input into the second WCE-Attention module to capture the correlation between different levels of features. Then, the second level of features is obtained and input into a deformable registration head to generate the corresponding intermediate deformation field. To further enhance the feature extraction capability, the deformation field is connected with the intermediate deformation field of the second level of features through an interpolation operation, and the optimized second level of feature deformation field is finally obtained. Thereafter, the above steps are repeated to generate the third and fourth levels of deformation fields , .

5. The method of claim 4, wherein the method is based on virtual vessel augmentation for CT-guided needle insertion. The registration head consists of a 3x3 convolutional layer and a ReLU activation function layer.

6. A registration network for use in the method of claims 1-5.

7. A WCE-Attention module used in the method of claims 1-5.

Images