Catheter end attitude estimation method, system and medium

Through the two-step multi-source information fusion method of deep learning network, combined with the catheter end shape prior and electromagnetic sensor direction information, the shape ambiguity and direction ambiguity of catheter end posture estimation in single XCA image is solved, achieving high efficiency , accurate catheter end posture estimates, reducing the doctor's operation burden and radiation risk.

CN113947633BActive Publication Date: 2024-11-19顾力栩
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111052354.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-08
Publication Date
2024-11-19
Estimated Expiration
2041-09-08

AI Technical Summary

Technical Problem

In the prior art, the accuracy of catheter end posture estimation in single XCA images is low, mainly due to the ambiguity of shape and ambiguity of direction, and there is a lack of effective solutions.

Method used

Using a two-step multi-source information fusion method based on deep learning, the shape priors of the catheter ends are obtained before intubation, and the direction information of the electromagnetic sensor is combined during intubation, and posture estimation is used to fusion shape priors. The two-dimensional projection image information is combined with the electromagnetic sensor direction signal, and the attitude estimation result in the point sequence format is output.

Benefits of technology

The accuracy and robustness of catheter end posture estimation is improved, the computational burden is reduced, the radiation problem and model-based instability are avoided in traditional multi-plane imaging, and efficient and accurate automatic attitude estimation is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113947633B_ABST
    Figure CN113947633B_ABST
Patent Text Reader

Abstract

The present invention provides a method, system and medium for estimating the posture of the end of a catheter, including: obtaining a priori shape of the end of the catheter in the form of three-dimensional scanning or manual mapping before intubation, cutting out a two-dimensional projection image of the end of the catheter in the XCA image during intubation, and recording the direction information on both sides of the curved section of the end of the catheter respectively; fusing the priori shape of the end of the catheter with the two-dimensional projection image information at the current moment in the pre-fusion stage, integrating the electromagnetic sensor direction information at the current moment into the middle layer of the deep learning network in the post-fusion stage, and fusing it with the middle layer features; inputting the pre-fusion result into the encoder of the deep learning network, inputting the post-fusion result into the decoder of the deep learning network, and the deep learning network finally outputs the posture estimation in the point sequence format. The present invention is the first to use a deep neural network to estimate the posture of the end of the catheter, and has the advantages of strong robustness, high computational efficiency, and accurate estimation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the medical field, and in particular, to a catheter tip posture estimation method, system and medium, and more particularly, to a catheter tip posture estimation method based on a two-step multi-source information fusion deep learning network. Background Art

[0002] In PCI (percutaneous coronary intervention) surgery, intraoperative XCA angiography images are usually used as references to observe the patient's vascular system and medical tools in the body. It has the advantages of real-time rapid imaging and clear anatomical structure. However, since it is a single-plane two-dimensional projection image, the three-dimensional depth information of the space is lost, so the doctor needs to rely on experience and operating feel to reconstruct the three-dimensional posture of the catheter end from the two-dimensional image information. The automatic catheter end posture estimation technology can be integrated into the image navigation technology to provide doctors with a more intuitive, complete, and real-time catheter end posture estimation reference, which will effectively reduce the doctor's operating burden and reduce misjudgment, greatly improving the efficiency and accuracy of the operation. Therefore, the development of this technology has important clinical significance.

[0003] After literature search, it was found that a series of automatic estimation technologies for the catheter tip posture have been published, which are mainly divided into modeling-based, multi-sensor-based and image information-based methods.

[0004] Greigarn et al. used pseudo-rigid-body theory in the paper "Experimental validation of the pseudo-rigid-body model of the MRI-actuated catheter" to assume that the catheter is a multi-body system composed of rigid rods connected by multiple joints, and approached the physical model of the catheter tip by optimizing the joint parameters.

[0005] In the paper "3D Catheter Shape Reconstruction Using Electromagnetic and Image Sensors", Tran et al. extracted compact two-dimensional image information from XCA and combined it with the discrete posture information of electromagnetic sensors, thus making up for the disadvantage that electromagnetic sensors are susceptible to external environmental fluctuations. At the same time, a probabilistic model was proposed to model the catheter as a three-dimensional B-spline tube, and the catheter contour in the two-dimensional projection was matched with the three-dimensional catheter shape using the traditional camera pinhole model. Finally, the maximum a posteriori algorithm was used to solve and reconstruct the three-dimensional posture of the catheter. The estimation method based on image information avoids the use of a large number of micro sensors and relies on model assumptions.

[0006] In the paper "Reconstruction method for curvilinear structures from two views", Hoffmann et al. established a triangulation model through the known camera parameter matrix to calculate the matching relationship between corresponding points in two image planes. At the same time, they abandoned the time-consuming operation of manually identifying points and proposed a method to automatically select corresponding points along the curved structure, and finally completed the reconstruction of the three-dimensional structure of the catheter. This method has a very high accuracy rate, but the use of multi-plane imaging technology increases the radiation exposure to patients and doctors during the operation.

[0007] In the paper "Real-Time Pose Estimation of Devices from X-ray Images: Application to X-ray / Echo Registration for Cardiac Interventions", Charles et al. directly restored the three-dimensional posture details from a single two-dimensional image information through the registration idea and proposed two novel 2D / 3D registration algorithms: direct hybrid correlation method and block region gradient correlation method.

[0008] After a literature review, it was found that the accuracy of the catheter tip posture estimation directly realized by a single XCA is often low. The two main reasons that affect the accuracy are shape ambiguity and direction ambiguity. Shape ambiguity occurs when catheters with various tip shapes produce similar 2D projection images; direction ambiguity occurs when a single 2D projection image corresponds to multiple possible 3D postures. At present, there is still no good way to improve the accuracy of 3D posture estimation by solving the above ambiguities. Summary of the invention

[0009] In view of the defects in the prior art, an object of the present invention is to provide a catheter tip posture estimation method, system and medium.

[0010] A catheter tip posture estimation method provided by the present invention comprises the following steps:

[0011] Multi-source information acquisition steps: Before intubation, obtain the catheter end shape prior in the form of three-dimensional scanning or manual mapping. During the intubation process, crop the two-dimensional projection image of the catheter end in the XCA image, and record the direction information on both sides of the curved section of the catheter end respectively;

[0012] Two-step information fusion process: In the pre-fusion stage, the catheter tip shape prior and the two-dimensional projection map information at the current moment are fused; in the post-fusion stage, the electromagnetic sensor direction information at the current moment is integrated into the middle layer of the deep learning network and fused with the middle layer features;

[0013] Pose point sequence estimation step: input the pre-fusion result into the encoder of the deep learning network, input the post-fusion result into the decoder of the deep learning network, and the deep learning network finally outputs the pose estimation in the point sequence format.

[0014] Preferably, the multi-source information acquisition step includes:

[0015] The catheter end shape prior acquisition step: before intubation, a catheter end shape prior expressed in the form of a point sequence is obtained by three-dimensional scanning or manual mapping, and the shape is m×3, where m represents the length of the point sequence. The shape prior describes the basic shape and initial posture of the catheter end. Two electromagnetic sensors are respectively installed on both sides of the bend of the catheter end to obtain the direction information of the measurement point during the intubation process;

[0016] Training steps: Randomly select coronary angiography images containing the catheter end, annotate the catheter end detection frame pixel by pixel, and train the convolutional neural network YOLO v3 for detecting the catheter end based on the center position and size of the original image and the corresponding detection frame;

[0017] Image processing steps: During the intubation process, YOLO v3 is used to detect the position of the catheter end in XCA in real time, and the two-dimensional projection image containing the catheter end is cropped according to the detection frame information of YOLO v3. The two-dimensional projection image is processed into a binary image through noise reduction and threshold segmentation. After the above processing, the size of the two-dimensional projection image is 1×H×W, where H is the image height and W is the image width;

[0018] Information collection step: collect the direction information sent back by the two electromagnetic sensors and process it into unit direction vectors d1 and d2.

[0019] Preferably, the two-step information fusion step includes:

[0020] Pre-fusion step: The catheter shape prior is expanded to a size of m×1×3, and is copied in height and width to make it consistent with the spatial size of the matching two-dimensional projection image. After the copy is completed, the size is m×H×W. At this time, the catheter shape prior is spliced ​​with the two-dimensional projection image in the channel direction to complete the pre-fusion. The result size after pre-fusion is (m+1)×H×W;

[0021] Sampling process: The result after pre-fusion is input into the encoder of the deep learning network. After three downsampling processes of convolutional layers of different sizes, the size is reduced to one eighth of the original size.

[0022] Post-fusion step: Calculate the unit direction vectors d1 and d2 with the image plane normal vector n using the formula to obtain the direction signal quantities S1 and S2:

[0023]

[0024]

[0025] Expand the S1 and S2 dimensions and copy them, and copy them in height and width to make them consistent with the size of the matching two-dimensional projection space after three times downsampling. The sizes after the copy are At this time, the directional signal is then spliced ​​with the output features of the deep learning encoder in the channel direction and then fused.

[0026] Preferably, the posture point sequence estimation step comprises:

[0027] Pre-fusion processing step: input the pre-fusion result that fuses the shape prior and the two-dimensional projection image into the encoder of the deep learning network. The encoder of the deep learning network is based on a series of two-dimensional convolution kernels with decreasing number of channels, and performs three down-sampling processing on the pre-fusion result. After the initial input is processed by the encoder, the feature size is reduced to one eighth of the original;

[0028] Post-fusion processing step: the post-fusion result that fuses the output features of the deep learning encoder and the electromagnetic direction signal is input into the decoder of the deep learning network. The decoder of the deep learning network is based on a series of two-dimensional convolution kernels that are inverse to its encoder, and performs three up-sampling processing on the post-fusion result. The size of the decoder output result is restored to be consistent with the original pre-fusion result;

[0029] Output step: The result of the deep learning decoder is input into a fully connected layer containing m×3 neurons, and after being processed by the Sigmoid function, the posture estimation result expressed as a point sequence is output.

[0030] A catheter tip posture estimation system provided by the present invention includes the following modules:

[0031] Multi-source information acquisition module: Before intubation, the catheter end shape prior is obtained in the form of 3D scanning or manual mapping. During intubation, the 2D projection image of the catheter end is cropped from the XCA image, and the direction information on both sides of the curved section of the catheter end is recorded respectively.

[0032] Two-step information fusion module: In the pre-fusion stage, the catheter tip shape prior and the current two-dimensional projection map information are fused; in the post-fusion stage, the electromagnetic sensor direction information at the current moment is integrated into the middle layer of the deep learning network and fused with the middle layer features;

[0033] Pose point sequence estimation module: The pre-fusion result is input into the encoder of the deep learning network, and the post-fusion result is input into the decoder of the deep learning network. The deep learning network finally outputs the pose estimation in the point sequence format.

[0034] Preferably, the multi-source information acquisition module includes:

[0035] Catheter end shape prior acquisition module: Before intubation, a catheter end shape prior expressed in the form of a point sequence is obtained by three-dimensional scanning or manual mapping. The shape is m×3, where m represents the length of the point sequence. The shape prior describes the basic shape and initial posture of the catheter end. Two electromagnetic sensors are installed on both sides of the bend of the catheter end to obtain the direction information of the measurement point during the intubation process.

[0036] Training module: Randomly select coronary angiography images containing the catheter end, annotate the catheter end detection frame pixel by pixel, and train the convolutional neural network YOLO v3 for detecting the catheter end based on the center position and size of the original image and the corresponding detection frame;

[0037] Image processing module: During the intubation process, YOLO v3 is used to detect the position of the catheter end in XCA in real time, and the two-dimensional projection image containing the catheter end is cropped according to the detection frame information of YOLO v3. The two-dimensional projection image is processed into a binary image through noise reduction and threshold segmentation. After the above processing, the size of the two-dimensional projection image is 1×H×W, where H is the image height and W is the image width;

[0038] Information collection module: collects the direction information sent back by the two electromagnetic sensors and processes it into unit direction vectors d1 and d2.

[0039] Preferably, the two-step information fusion module includes:

[0040] Pre-fusion module: The catheter shape prior is expanded to a size of m×1×3, and is copied in height and width to make it consistent with the spatial size of the matching two-dimensional projection image. After the copy is completed, the size is m×H×W. At this time, the catheter shape prior is spliced ​​with the two-dimensional projection image in the channel direction to complete the pre-fusion. The result size after pre-fusion is (m+1)×H×W.

[0041] Sampling processing module: The result after pre-fusion is input into the encoder of the deep learning network. After three downsampling processes of convolutional layers of different sizes, the size is reduced to one eighth of the original size.

[0042] Post-fusion module: The unit direction vectors d1 and d2 are calculated with the image plane normal vector n to obtain the direction signal quantities S1 and S2:

[0043]

[0044]

[0045] Expand the S1 and S2 dimensions and copy them, and copy them in height and width to make them consistent with the size of the matching two-dimensional projection space after three times downsampling. The sizes after the copy are At this time, the directional signal is then spliced ​​with the output features of the deep learning encoder in the channel direction and then fused.

[0046] Preferably, the posture point sequence estimation module comprises:

[0047] Pre-fusion processing module: input the pre-fusion result that fuses the shape prior and the two-dimensional projection image into the encoder of the deep learning network. The encoder of the deep learning network is based on a series of two-dimensional convolution kernels with decreasing number of channels, and performs three down-sampling processing on the pre-fusion result. After the initial input is processed by the encoder, the feature size is reduced to one eighth of the original;

[0048] Post-fusion processing module: The post-fusion result that fuses the output features of the deep learning encoder and the electromagnetic direction signal is input into the decoder of the deep learning network. The decoder of the deep learning network is based on a series of two-dimensional convolution kernels that are inverse to its encoder. It performs three upsampling processing on the post-fusion result, and the size of the decoder output result is restored to be consistent with the original pre-fusion result.

[0049] Output module: The result of the deep learning decoder is input into a fully connected layer containing m×3 neurons, and after being processed by the Sigmoid function, the posture estimation result expressed as a point sequence is output.

[0050] According to a computer-readable storage medium storing a computer program provided by the present invention, the steps of the above method are implemented when the computer program is executed by a processor.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. The present invention is the first to use deep neural networks to estimate the posture of the catheter tip, abandoning the multi-plane projection idea in traditional estimation methods, and proposes a new idea of ​​two-step multi-source information fusion, which has the advantages of strong robustness, high computational efficiency, and accurate estimation.

[0053] 2. The present invention utilizes a catheter tip posture estimation method based on deep learning, which avoids the shortcomings of large radiation dose of traditional multi-plane imaging methods and instability of model-based methods. In addition to a single two-dimensional projection image, it additionally refers to shape priors and electromagnetic direction signals to solve shape ambiguity and direction ambiguity, respectively.

[0054] 3. The present invention utilizes a data-driven deep learning network to perform nonlinear fitting on the estimation model, thereby achieving accurate and efficient automatic posture estimation of the catheter tip while greatly ensuring the external size and functionality of the catheter tip, thereby reducing the burden on doctors and misjudgment during surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0056] Figure 1 This is a flow chart of the catheter tip posture estimation method based on a two-step multi-source information fusion deep learning network.

[0057] Figure 2 This is the network structure diagram for catheter tip posture estimation.

[0058] Figure 3 This is the effect diagram of the catheter tip posture estimation. DETAILED DESCRIPTION

[0059] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0060] like Figures 1 to 3 As shown, according to a catheter tip posture estimation method, system and medium provided by the present invention, the method comprises the following steps:

[0061] Step 1: Multi-source information acquisition: Before intubation, the catheter end shape prior is obtained in the form of 3D scanning or manual mapping. During the intubation process, the detection frame information provided by YOLO v3 (You Only Look Once Version 3) is used to crop the 2D projection image of the catheter end in XCA (X-ray coronary angiography), and two electromagnetic sensors are used to record the direction information on both sides of the curved segment of the catheter end.

[0062] Step 2, two-step information fusion: In the pre-fusion stage, the catheter tip shape prior and the two-dimensional projection map information at the current moment are fused; in the post-fusion stage, the electromagnetic sensor direction information at the current moment is integrated into the middle layer of the deep learning network and fused with the middle layer features;

[0063] Step 3: Estimation of posture point sequence: Input the pre-fusion result into the encoder of the deep learning network, and input the post-fusion result into the decoder of the deep learning network. The deep learning network finally outputs the posture estimation in the point sequence format.

[0064] Furthermore, the step 1 comprises the following steps:

[0065] Step 11: Before intubation, a shape prior of the catheter end expressed in the form of a point sequence is obtained by three-dimensional scanning or manual mapping. The shape is m×3. The shape prior describes the basic shape and initial posture of the catheter end. In addition, two electromagnetic sensors are installed on both sides of the bend of the catheter end to obtain the direction information of the measurement point during the intubation process.

[0066] Step 12: randomly select some coronary angiography images containing the catheter end, annotate the catheter end detection frame pixel by pixel, and train the convolutional neural network YOLO v3 for detecting the catheter end based on the center position and size of the original image and the corresponding detection frame;

[0067] Step 13: During the intubation process, YOLO v3 is used to detect the position of the catheter end in XCA in real time, and the two-dimensional projection image containing the catheter end is cropped according to the detection frame information of YOLO v3. The two-dimensional projection image is processed into a binary image through noise reduction and threshold segmentation. After the above processing, the size of the two-dimensional projection image is 1×H×W;

[0068] Step 14: While step 13 is being performed, the direction information returned by the two electromagnetic sensors is synchronously collected and processed into unit direction vectors d1 and d2.

[0069] Furthermore, the step 2 comprises the following steps:

[0070] Step 21, the size of the catheter shape prior is expanded to m×1×3, and the height and width are copied to make it consistent with the spatial size of the matching two-dimensional projection image. After the copy is completed, the size is m×H×W. At this time, the catheter shape prior is spliced ​​with the two-dimensional projection image in the channel direction to complete the front fusion. The result size after the front fusion is (m+1)×H×W;

[0071] Step 22: The result after the pre-fusion is input into the encoder of the deep learning network. After three downsampling processes of convolutional layers of different sizes, the size is reduced to one eighth of the original size.

[0072] Step 23, calculate the unit direction vectors d1 and d2 and the image plane normal vector n by using a formula to obtain the direction signal quantities S1 and S2:

[0073]

[0074]

[0075] Expand the S1 and S2 dimensions and copy them, and copy them in height and width to make them consistent with the size of the matching two-dimensional projection space after three times downsampling. The sizes after the copy are At this time, the directional signal is then spliced ​​with the output features of the deep learning encoder in the channel direction and then fused.

[0076] Furthermore, the step 3 comprises the following steps:

[0077] Step 31, input the pre-fusion result that fuses the shape prior and the two-dimensional projection image into the encoder of the deep learning network. The encoder of the deep learning network is based on a series of two-dimensional convolution kernels with decreasing number of channels, and performs three down-sampling processing on the pre-fusion result. After the initial input is processed by the encoder, the feature size is reduced to one eighth of the original;

[0078] Step 32: input the post-fusion result that fuses the output features of the deep learning encoder and the electromagnetic direction signal into the decoder of the deep learning network. The decoder of the deep learning network is based on a series of two-dimensional convolution kernels that are inverse to its encoder, and performs three up-sampling processing on the post-fusion result. The size of the decoder output result is restored to be consistent with the original pre-fusion result.

[0079] Step 33: The result of the deep learning decoder is input into a fully connected layer containing m×3 neurons, and after being processed by the Sigmoid function, the posture estimation result expressed as a point sequence is output.

[0080] The present invention also provides a catheter end posture estimation system, comprising the following modules:

[0081] Multi-source information acquisition module: Before intubation, the catheter end shape prior is obtained in the form of 3D scanning or manual mapping. During intubation, the 2D projection image of the catheter end is cropped from the XCA image, and the direction information on both sides of the curved section of the catheter end is recorded respectively.

[0082] Two-step information fusion module: In the pre-fusion stage, the catheter tip shape prior and the current two-dimensional projection map information are fused; in the post-fusion stage, the electromagnetic sensor direction information at the current moment is integrated into the middle layer of the deep learning network and fused with the middle layer features;

[0083] Pose point sequence estimation module: The pre-fusion result is input into the encoder of the deep learning network, and the post-fusion result is input into the decoder of the deep learning network. The deep learning network finally outputs the pose estimation in the point sequence format.

[0084] The multi-source information acquisition module includes:

[0085] Catheter end shape prior acquisition module: Before intubation, a catheter end shape prior expressed in the form of a point sequence is obtained by three-dimensional scanning or manual mapping. The shape is m×3, where m represents the length of the point sequence. The shape prior describes the basic shape and initial posture of the catheter end. Two electromagnetic sensors are installed on both sides of the bend of the catheter end to obtain the direction information of the measurement point during the intubation process.

[0086] Training module: Randomly select coronary angiography images containing the catheter end, annotate the catheter end detection frame pixel by pixel, and train the convolutional neural network YOLO v3 for detecting the catheter end based on the center position and size of the original image and the corresponding detection frame;

[0087] Image processing module: During the intubation process, YOLO v3 is used to detect the position of the catheter end in XCA in real time, and the two-dimensional projection image containing the catheter end is cropped according to the detection frame information of YOLO v3. The two-dimensional projection image is processed into a binary image through noise reduction and threshold segmentation. After the above processing, the size of the two-dimensional projection image is 1×H×W, where H is the image height and W is the image width;

[0088] Information collection module: collects the direction information sent back by the two electromagnetic sensors and processes it into unit direction vectors d1 and d2.

[0089] The two-step information fusion module includes:

[0090] Pre-fusion module: The catheter shape prior is expanded to a size of m×1×3, and is copied in height and width to make it consistent with the spatial size of the matching two-dimensional projection image. After the copy is completed, the size is m×H×W. At this time, the catheter shape prior is spliced ​​with the two-dimensional projection image in the channel direction to complete the pre-fusion. The result size after pre-fusion is (m+1)×H×W.

[0091] Sampling processing module: The result after pre-fusion is input into the encoder of the deep learning network. After three downsampling processes of convolutional layers of different sizes, the size is reduced to one eighth of the original size.

[0092] Post-fusion module: The unit direction vectors d1 and d2 are calculated with the image plane normal vector n to obtain the direction signal quantities S1 and S2:

[0093]

[0094]

[0095] Expand the S1 and S2 dimensions and copy them, and copy them in height and width to make them consistent with the size of the matching two-dimensional projection space after three times downsampling. The sizes after the copy are At this time, the directional signal is then spliced ​​with the output features of the deep learning encoder in the channel direction and then fused.

[0096] The posture point sequence estimation module comprises:

[0097] Pre-fusion processing module: input the pre-fusion result that fuses the shape prior and the two-dimensional projection image into the encoder of the deep learning network. The encoder of the deep learning network is based on a series of two-dimensional convolution kernels with decreasing number of channels, and performs three down-sampling processing on the pre-fusion result. After the initial input is processed by the encoder, the feature size is reduced to one eighth of the original;

[0098] Post-fusion processing module: The post-fusion result that fuses the output features of the deep learning encoder and the electromagnetic direction signal is input into the decoder of the deep learning network. The decoder of the deep learning network is based on a series of two-dimensional convolution kernels that are inverse to its encoder. It performs three upsampling processing on the post-fusion result, and the size of the decoder output result is restored to be consistent with the original pre-fusion result.

[0099] Output module: The result of the deep learning decoder is input into a fully connected layer containing m×3 neurons, and after being processed by the Sigmoid function, the posture estimation result expressed as a point sequence is output.

[0100] This embodiment is implemented on a computer with a 64-bit Linux system, 64GB of RAM, Intel(R) Xeon E5-2630 v3 CPU, and Nvidia Titan X pascal GPU. The entire method is based on the Pytorch deep learning framework and implemented in the python programming language. The collected clinical medical data is loaded into the network model to complete the registration work. The effect diagram shows four examples. The black dots represent the actual catheter end posture, and the star points represent the catheter end posture predicted by this method. The present invention can quickly and effectively realize the catheter posture, and the average error is controlled at 0.24mm.

[0101] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A catheter tip posture estimation method, characterized in that: The steps include: Multi-source information acquisition steps: Before intubation, obtain the catheter end shape prior in the form of three-dimensional scanning or manual mapping. During the intubation process, crop the two-dimensional projection image of the catheter end in the XCA image, and record the direction information on both sides of the curved section of the catheter end respectively; Two-step information fusion process: In the pre-fusion stage, the catheter tip shape prior and the two-dimensional projection map information at the current moment are fused; in the post-fusion stage, the electromagnetic sensor direction information at the current moment is integrated into the middle layer of the deep learning network and fused with the middle layer features; Attitude point sequence estimation step: input the pre-fusion result into the encoder of the deep learning network, input the post-fusion result into the decoder of the deep learning network, and the deep learning network finally outputs the attitude estimation in the point sequence format; The multi-source information acquisition step comprises: The catheter end shape prior acquisition step: before intubation, a catheter end shape prior expressed in the form of a point sequence is obtained by three-dimensional scanning or manual mapping, and the shape is m×3, where m represents the length of the point sequence. The shape prior describes the basic shape and initial posture of the catheter end. Two electromagnetic sensors are respectively installed on both sides of the bend of the catheter end to obtain the direction information of the measurement point during the intubation process; Training steps: Randomly select coronary angiography images containing the catheter end, annotate the catheter end detection frame pixel by pixel, and train the convolutional neural network YOLO v3 for detecting the catheter end based on the center position and size of the original image and the corresponding detection frame; Image processing steps: During the intubation process, YOLO v3 is used to detect the position of the catheter end in XCA in real time, and the two-dimensional projection image containing the catheter end is cropped according to the detection frame information of YOLO v3. The two-dimensional projection image is processed into a binary image through noise reduction and threshold segmentation. After the above processing, the size of the two-dimensional projection image is 1×H×W, where H is the image height and W is the image width; Information collection step: collect the direction information returned by the two electromagnetic sensors and process it into unit direction vectors d1 and d2; The two-step information fusion step includes: Pre-fusion step: The catheter shape prior is expanded to a size of m×1×3, and is copied in height and width to make it consistent with the spatial size of the matching two-dimensional projection image. After the copy is completed, the size is m×H×W. At this time, the catheter shape prior is spliced ​​with the two-dimensional projection image in the channel direction to complete the pre-fusion. The result size after pre-fusion is (m+1)×H×W; Sampling process: The result after pre-fusion is input into the encoder of the deep learning network. After three downsampling processes of convolutional layers of different sizes, the size is reduced to one eighth of the original size. Post-fusion step: Calculate the unit direction vectors d1 and d2 with the image plane normal vector n using the formula to obtain the direction signal quantities S1 and S2: Expand the S1 and S2 dimensions and copy them, and copy them in height and width to make them consistent with the size of the matching two-dimensional projection space after three times downsampling. The sizes after the copy are At this time, the directional signal is then spliced ​​with the output features of the deep learning encoder in the channel direction and then fused.

2. The catheter tip posture estimation method according to claim 1, characterized in that: The posture point sequence estimation step comprises: Pre-fusion processing step: input the pre-fusion result that fuses the shape prior and the two-dimensional projection image into the encoder of the deep learning network. The encoder of the deep learning network is based on a series of two-dimensional convolution kernels with decreasing number of channels, and performs three down-sampling processing on the pre-fusion result. After the initial input is processed by the encoder, the feature size is reduced to one eighth of the original; Post-fusion processing step: the post-fusion result that fuses the output features of the deep learning encoder and the electromagnetic direction signal is input into the decoder of the deep learning network. The decoder of the deep learning network is based on a series of two-dimensional convolution kernels that are inverse to its encoder, and performs three up-sampling processing on the post-fusion result. The size of the decoder output result is restored to be consistent with the original pre-fusion result; Output step: The result of the deep learning decoder is input into a fully connected layer containing m×3 neurons, and after being processed by the Sigmoid function, the posture estimation result expressed as a point sequence is output.

3. A catheter tip posture estimation system, characterized in that: Includes the following modules: Multi-source information acquisition module: Before intubation, the catheter end shape prior is obtained in the form of 3D scanning or manual mapping. During intubation, the 2D projection image of the catheter end is cropped from the XCA image, and the direction information on both sides of the curved section of the catheter end is recorded respectively. Two-step information fusion module: In the pre-fusion stage, the catheter tip shape prior and the current two-dimensional projection map information are fused; in the post-fusion stage, the electromagnetic sensor direction information at the current moment is integrated into the middle layer of the deep learning network and fused with the middle layer features; Pose point sequence estimation module: input the pre-fusion result into the encoder of the deep learning network, input the post-fusion result into the decoder of the deep learning network, and the deep learning network finally outputs the pose estimation in the point sequence format; The multi-source information acquisition module includes: Catheter end shape prior acquisition module: Before intubation, a catheter end shape prior expressed in the form of a point sequence is obtained by three-dimensional scanning or manual mapping. The shape is m×3, where m represents the length of the point sequence. The shape prior describes the basic shape and initial posture of the catheter end. Two electromagnetic sensors are installed on both sides of the bend of the catheter end to obtain the direction information of the measurement point during the intubation process. Training module: Randomly select coronary angiography images containing the catheter end, annotate the catheter end detection frame pixel by pixel, and train the convolutional neural network YOLO v3 for detecting the catheter end based on the center position and size of the original image and the corresponding detection frame; Image processing module: During the intubation process, YOLO v3 is used to detect the position of the catheter end in XCA in real time, and the two-dimensional projection image containing the catheter end is cropped according to the detection frame information of YOLO v3. The two-dimensional projection image is processed into a binary image through noise reduction and threshold segmentation. After the above processing, the size of the two-dimensional projection image is 1×H×W, where H is the image height and W is the image width; Information collection module: collects the direction information sent back by the two electromagnetic sensors and processes it into unit direction vectors d1 and d2; The two-step information fusion module includes: Pre-fusion module: The catheter shape prior is expanded to a size of m×1×3, and is copied in height and width to make it consistent with the spatial size of the matching two-dimensional projection image. After the copy is completed, the size is m×H×W. At this time, the catheter shape prior is spliced ​​with the two-dimensional projection image in the channel direction to complete the pre-fusion. The result size after pre-fusion is (m+1)×H×W. Sampling processing module: The result after pre-fusion is input into the encoder of the deep learning network. After three downsampling processes of convolutional layers of different sizes, the size is reduced to one eighth of the original size. Post-fusion module: The unit direction vectors d1 and d2 are calculated with the image plane normal vector n to obtain the direction signal quantities S1 and S2: Expand the S1 and S2 dimensions and copy them, and copy them in height and width to make them consistent with the size of the matching two-dimensional projection space after three times downsampling. The sizes after the copy are At this time, the directional signal is then spliced ​​with the output features of the deep learning encoder in the channel direction and then fused.

4. The catheter tip posture estimation system according to claim 3, characterized in that: The posture point sequence estimation module comprises: Pre-fusion processing module: input the pre-fusion result that fuses the shape prior and the two-dimensional projection image into the encoder of the deep learning network. The encoder of the deep learning network is based on a series of two-dimensional convolution kernels with decreasing number of channels, and performs three down-sampling processing on the pre-fusion result. After the initial input is processed by the encoder, the feature size is reduced to one eighth of the original; Post-fusion processing module: The post-fusion result that fuses the output features of the deep learning encoder and the electromagnetic direction signal is input into the decoder of the deep learning network. The decoder of the deep learning network is based on a series of two-dimensional convolution kernels that are inverse to its encoder. It performs three upsampling processing on the post-fusion result, and the size of the decoder output result is restored to be consistent with the original pre-fusion result. Output module: The result of the deep learning decoder is input into a fully connected layer containing m×3 neurons, and after being processed by the Sigmoid function, the posture estimation result expressed as a point sequence is output.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.