Risk management for robotic catheter navigation systems

Abstract

A system and method for navigating a catheter within a patient using a robotic navigation system with risk management are provided. The system receives input medical images from the patient. A trained segmentation network is used to determine a trajectory based on the input medical images to navigate the catheter from its current location within the patient to a target location. One or more actions of the robotic navigation system for navigating the catheter from its current location toward the target location, along with confidence levels associated with said actions, are determined by a trained AI (artificial intelligence) agent based on the generated trajectory and the current view of the catheter. In response to a confidence level meeting a threshold, one or more actions are evaluated based on the view of the catheter during navigation according to the one or more actions. Based on the evaluated one or more actions, the robotic navigation system navigates the catheter from its current location toward the target location.

Description

Technical Field

[0001] This invention generally relates to robotic catheter navigation, and more particularly to risk management of robotic catheter navigation systems. Background Technology

[0002] Robotic catheter navigation systems have been developed to assist surgeons in performing minimally invasive procedures. These systems reduce the difficulty of surgeon training and decrease their radiation exposure. However, despite these advantages, robotic catheter navigation systems have not yet been widely adopted due to the risks and uncertainties associated with them. Summary of the Invention

[0003] According to one or more embodiments, a system and method for risk management in a robotic catheter navigation system are provided. An input medical image from a patient is received. A trained segmentation network is used to determine a trajectory for navigating the catheter from its current location within the patient's body to a target location based on the input medical image. One or more actions of the robotic navigation system for navigating the catheter from its current location toward the target location, and a confidence level associated with said one or more actions, are determined by a trained AI (artificial intelligence) agent based on the generated trajectory and the current view of the catheter. In response to a confidence level meeting a threshold, one or more actions are evaluated based on the view of the catheter during navigation according to the one or more actions. Based on the evaluated one or more actions, the robotic navigation system navigates the catheter from its current location toward the target location.

[0004] In one embodiment, a trajectory is generated by creating a color-coded dynamic route map of blood vessels in an input medical image, including color coding to indicate uncertainty. The uncertainty is quantified by a trained segmentation network. To train the trained segmentation network, a training image set is received. An initial segmentation network is trained based on the training image set annotated by a single user. Blood vessels are segmented from a subset of the training image set using the trained initial segmentation network. A variability distribution of annotations from multiple users for the training image subset is determined based on the segmented blood vessels. Annotations from some of the multiple users for the training image set are weighted based on the variability distribution. A final segmentation network is trained based on this training image set, the weighted annotations, and the uncertainty associated with each weighted annotation. The trained final segmentation network is output.

[0005] In one embodiment, in response to a confidence level not meeting a threshold, the AI ​​agent moves to a previous position on the trajectory. The trajectory is optimized to navigate the catheter from the previous position to the target position. The AI ​​agent is placed at the position closest to the current position on the optimal trajectory. Using the view of the catheter at the position closest to the current position on the optimal trajectory as the current view, the AI ​​agent restarts its navigation of the catheter. The trajectory is optimized based on the possible actions of the robot navigation system and vascular segmentation from the input medical image.

[0006] In one embodiment, one or more actions are evaluated by determining whether the catheter view during navigation according to one or more actions is outside the domain of the training data, on which a trained AI agent is trained. In another embodiment, one or more actions are evaluated by assessing the bending stress of the guidewire used for catheter navigation and the number of punctures.

[0007] In one embodiment, when a user performs a set of actions in the navigation conduit, the configuration of the robot navigation system is stored. The set of actions is then replayed based on the stored configuration of the robot navigation system.

[0008] In one embodiment, user input is received to select the location to which the catheter should be navigated. Kinematics for navigating the catheter to the selected location are calculated. The catheter is then navigated to the selected location based on the calculated kinematics.

[0009] In one embodiment, a safety margin is calculated for the path in the input medical image. Based on the current position of the catheter relative to the safety margin, tactile feedback is provided to the user navigating the catheter.

[0010] These and other advantages of the present invention will be apparent to those skilled in the art from the following detailed description and accompanying drawings. Attached Figure Description

[0011] Figure 1 A workflow for automated catheter navigation with risk management according to one or more embodiments is illustrated;

[0012] Figure 2 An automated catheter navigation method with risk management is illustrated according to one or more embodiments;

[0013] Figure 3 An exemplary network architecture of a segmentation network according to one or more embodiments is shown;

[0014] Figure 4 The workflow for low-confidence mobility assessment according to one or more embodiments is illustrated;

[0015] Figure 5 The workflow for catheter status assessment according to one or more embodiments is illustrated;

[0016] Figure 6 A method for training a segmentation network according to one or more embodiments to segment blood vessels and quantify the level of uncertainty associated with segmentation is illustrated.

[0017] Figure 7A An image rendered in a "first-person" perspective according to one or more embodiments is shown;

[0018] Figure 7B Images rendered using Mercator technology according to one or more embodiments are shown;

[0019] Figure 8 An exemplary artificial neural network that can be used to implement one or more embodiments is shown;

[0020] Figure 9 This illustrates a convolutional neural network that can be used to implement one or more embodiments; and

[0021] Figure 10 A high-level block diagram of a computer that can be used to implement one or more embodiments is shown. Detailed Implementation

[0022] This invention generally relates to methods and systems for risk management of robotic conduit navigation systems. Embodiments of the invention are described herein to provide a visual understanding of such methods and systems. Digital images typically consist of digital representations of one or more objects (or shapes). The digital representations of objects herein are generally described based on the object's identification and manipulation. Such manipulation is a virtual manipulation performed in the memory or other circuitry / hardware of a computer system. Therefore, it should be understood that embodiments of the invention can be performed within a computer system using data stored within the computer system.

[0023] Conventional robotic catheter navigation systems typically utilize pre-trained machine learning models. Catheter navigation using conventional robotic catheter navigation systems is associated with several risk factors. For example, one risk factor is out-of-domain input data, where the input data to the machine learning model is outside the domain of the training data used to train the model. Another risk factor is model limitation, where the machine learning model is limited due to uncertainties in the annotations of the training data or due to the limited amount of training data. Additional risk factors include catheter / guidewire-related risks, where pressure or puncture at the vessel boundaries by the catheter / guidewire can lead to bleeding or hematoma.

[0024] The embodiments described herein provide for managing the risks associated with robotic catheter navigation systems. In one embodiment, a color-coded dynamic route map is provided to expose risks and uncertainties to the user during catheter navigation using a color-coded route map. In another embodiment, low-confidence movement risks, out-of-domain (OOD) risks, and catheter state risks are quantified, and remedial solutions are provided. Advantageously, the embodiments described herein help prevent high-risk movements when navigating catheters.

[0025] Figure 1 A workflow 100 for automated catheter navigation with risk management according to one or more embodiments is illustrated. Workflow 100 includes two phases: a trajectory generation phase 102 for generating a trajectory to navigate the catheter from its current location to a target location; and a proxy navigation phase 104 for navigating the catheter from its current location toward the target location based on the generated trajectory. (The remaining text appears to be unrelated and likely refers to further details about the workflow.) Figure 2-6 Continue to refer to Figure 1 .

[0026] Figure 2 A method 200 for automated catheter navigation with risk management according to one or more embodiments is illustrated. The steps of method 200 can be performed by one or more suitable computing devices (such as, for example...). Figure 10 The computer (1002) is used to implement this.

[0027] exist Figure 2 At step 202, the input medical image of the patient is received. The input medical image may depict the patient's blood vessels (e.g., coronary arteries) in which the catheter will be navigated. The input medical image may have any suitable modality, such as, for example, MRI (magnetic resonance imaging), CT (computed tomography), US (ultrasound), X-ray, and / or any other medical imaging modality or combination of medical imaging modalities. The input medical image may be a 2D (two-dimensional) image or a 3D (three-dimensional) volume, and may comprise a single image or multiple images (e.g., forming a 2.5D image). When the image is acquired, the input medical image may be received directly from an image acquisition device (such as, for example, a CT scanner), or may be received by loading previously acquired images from the storage device or memory of a computer system (e.g., PACS (picture archiving and communication system)) or by receiving images that have been transmitted from a remote computer system.

[0028] exist Figure 2 At step 204, a trajectory is generated based on the input medical image to navigate the catheter from its current location within the patient to a target location. In one example, such as... Figure 1 As shown, the trajectory can be generated during the trajectory generation phase 102 of workflow 100.

[0029] During trajectory generation phase 102, a CDRM (Color Dynamic Roadmap) 112 is first generated from the input medical image. The CDRM 112 is a roadmap of the patient's blood vessels, color-coded to indicate levels of uncertainty. This uncertainty may stem from data annotation uncertainty and model prediction uncertainty. The CDRM 112 is generated by segmenting blood vessels from the input medical image using a trained machine learning-based segmentation network. The segmentation network receives the input medical image as input and generates a corresponding probability map as output, representing pixel-by-pixel segmentation of blood vessels encoding levels of uncertainty. Figure 3 An exemplary machine learning-based segmentation network is shown below. The following is about... Figure 6 The training of the segmentation network to segment blood vessels from input medical images is further described, and the level of uncertainty associated with the segmentation is quantified.

[0030] Figure 3 An exemplary network architecture of a segmentation network 300 according to one or more embodiments is shown. The segmentation network 300 in... Figure 3 The segmentation network 300 is implemented as a dense UNet. The segmentation network 300 receives an angiography image 302 as input and generates a probability heatmap 304 as output. The probability heatmap 304 has the same size as the angiography image 302. Each pixel of the probability heatmap 304 has a value between 0 and 1, representing the level of uncertainty associated with the vessel segmentation, such that 0 indicates no vessel is shown on the corresponding pixel, while 1 indicates 100% certainty (i.e., no uncertainty) that a vessel is shown on the corresponding pixel. Therefore, the heatmap 304 may represent a vessel segmentation from the angiography image 302, where the uncertainty level is encoded. While the segmentation network 300 in Figure 3 The implementation is shown as denseUNet, but it should be understood that the segmentation network 300 can be implemented based on any suitable machine learning-based network, such as UNet, fully convolutional networks, etc.

[0031] Return to Figure 1The trajectory generation phase 102 determines the trajectory from the CDRM 112 for navigating the catheter from its current position to a target position. To generate the trajectory, endpoint identification 114 is implemented in the CDRM 112 by identifying the current and target positions. For example, user input identifying the current and target positions in the CDRM 112 may be received. The vascular path between the current and target positions is extracted from the CDRM 112 as the trajectory. If the trajectory is determined to be ambiguous, additional anchor points (e.g., from the user) are requested at decision box 116. The additional anchor points include one or more points that will be located between the current and target positions in the trajectory. The trajectory is extracted from the CDRM 112 based on the additional anchor points along with the current and target positions. If the trajectory is determined to be unambiguous, at decision box 116, the trajectory is output to the AI ​​(artificial intelligence) agent in the agent navigation phase 104 to navigate the catheter from its current position to the target position.

[0032] In one embodiment, the trajectory is blurred when the path along the blood vessel is discontinuous between the current position and the target position. This could be due to segmentation errors from the CDRM 112 or due to incorrect points set by the user. In this scenario, user input is received from the user manual depicting the discontinuous sections, or the input is corrected to define the path from the current position to the target position. In another embodiment, the trajectory is blurred when multiple paths connect the current position to the target position. Since blood vessels are 3D structures in the real world, and the segmented CDRM 112 is its 2D projection, self-crossing of blood vessels may occur. Figure 1 (As shown in decision box 116). From an image perspective, there are multiple paths from the current position to the target position. However, given the tree structure of a 3D blood vessel, in the real world, there is usually only one path. In this scenario, user input is received from the user, and the correct path is manually highlighted while the incorrect path is removed.

[0033] exist Figure 2 At step 206, the trained AI agent, based on the generated trajectory and the current view of the duct, determines 1) one or more actions of the robot navigation system used to navigate the duct from its current position toward a target position, and 2) the confidence associated with the one or more actions. In one example, such as... Figure 1 As shown, AI agent 120 is trained to observe the current state s i 118, to predict the Q-value 122 and uncertainty C 124 associated with each possible action of the robot navigation system used for the navigation conduit. Current state s iImage 118 is an image of the current view of the catheter. The Q value 122 represents the expected reward for taking the associated action. In one example, possible actions of a robotic navigation system for navigating a catheter include rotation on retraction (ROR, automatically rotating the guidewire during retraction), oscillation (automatically oscillating the guidewire during advancement), rotation (rotation of the guidewire), punctuation (back-and-forth movement as the catheter advances), and constant speed. The possible actions of the robotic navigation system are constrained by the generated trajectory. The AI ​​agent 120 cannot perform an action that removes it from the generated trajectory. The AI ​​agent 120 selects the action with the highest Q value 122 from the constrained set of actions. The AI ​​agent 120 can be implemented using known techniques.

[0034] exist Figure 2 At step 208, in response to the confidence level meeting the threshold, one or more actions are evaluated based on the duct view during navigation based on one or more actions.

[0035] like Figure 1 As shown, the confidence level is evaluated by the low-confidence movement rating component 106. The low-confidence movement rating component 106 addresses the uncertainty caused by the high uncertainty of the determined action. (See reference...) Figure 4 To describe the low-confidence moving assessment component 106, Figure 4 A workflow 400 for low-confidence mobility assessment according to one or more embodiments is shown.

[0036] exist Figure 4 At step 402, the AI ​​agent navigates and gets stuck at a location with low confidence. Uncertainty C 124 is compared to a pre-determined threshold τ to determine if the AI ​​agent 120 is located at a low-confidence location. If uncertainty C 124 meets the threshold τ (e.g., C > τ, meaning not low confidence), workflow 100 proceeds to pseudo-movement 126, where the duct view is based on navigation according to one or more actions (i.e., based on the next state s). i+1 The AI ​​agent evaluates one or more actions. If the uncertainty C124 does not meet the threshold τ (e.g., C < τ, meaning low confidence), the AI ​​agent moves back to the previous state at the previous position in the trajectory and determines an optimized trajectory from the previous position to the target position. In one example, such as Figure 4 As shown, at step 404, the AI ​​agent rolls back to the previous position t0, and at step 406, the optimal trajectory is solved.

[0037] The optimal trajectory is determined using a trajectory optimization algorithm. Given an initial trajectory... Each step t i It is represented by two values ​​indicating the x and y coordinates of the catheter tip in the image (x i yi The objective function for determining the optimal trajectory is given as follows:

[0038]

[0039] Where s(T) represents the total curvature potential of trajectory T on the blood vessel. Trajectory optimization can be described as a non-convex problem as follows:

[0040]

[0041] Subject to the following constraints:

[0042]

[0043] Where the function g w Let V represent the action space of all possible actions of the robot navigation system, and let V represent the segmented blood vessels from the input medical image.

[0044] In the first constraint, Due to the material properties of the guidewire, the trajectory is constrained to the possible actions of the robot navigation system. The guidewire cannot move arbitrarily in space; instead, given the current position t0, the choice of the next position t1 can only have a finite number of possibilities. Mathematically, this constraint is equivalent to the following:

[0045]

[0046] in And w1, w2, w3, and w4 are weights.

[0047] Second constraint In this process, each step of the trajectory is constrained within the segmented vessel V given by CDRM 112. This constraint ensures that the trajectory must be within the vessel.

[0048] This optimization problem is nonconvex (i.e., there is more than one optimal solution), and due to the constraints... The initial exponential search space becomes difficult to solve as it grows. Therefore, this constraint increases with... l 1 Relaxed and relaxed, that is Then, the optimization problem becomes (using Lagrange operators):

[0049]

[0050] Subject to the following constraints:

[0051] .

[0052] Solving the above optimization problem yields an optimal trajectory T. If the optimal trajectory T is solvable, then at checkpoint 128, the trajectory is updated with the optimal trajectory, and workflow 100 continues to pseudo-movement 126. The AI ​​agent locates itself at the position on the optimal trajectory that is closest to its current position but closer to the target position, and the AI ​​agent continues navigation using its own network. For example, in Figure 4 At step 408, the AI ​​agent is placed at location t. k At that point, navigation is restarted. Therefore, method 200 uses position t. k The duct view at that point is returned as the current view to step 206. Otherwise, if the optimal trajectory T is unsolvable, a flag 130 requesting user input is generated at determination box 128. User input may include, for example, manual navigation of the duct or robot repositioning.

[0053] Since trajectory optimization is independent of image features, determining the optimal trajectory is not performed immediately. As seen in the optimization problem, only the vascular tree location feature V is included in the optimization, and no image features or vascular appearance features are used. On the other hand, the AI ​​agent utilizes visual features and can adaptively make action decisions. For example, some steps t generated by trajectory optimization... i Operating the robot too close to the container's boundary could pose a high risk. An AI agent might find a better trajectory that keeps operational risk low.

[0054] At pseudo-movement 126, when navigating based on one or more actions (i.e., based on the next state), one or more actions are evaluated based on the duct view. At OOD detection component 108, the next state s i+1 132 was evaluated to determine the next state. i+1 132 Whether the next state s is outside the domain of the training data used to train the AI ​​agent 120 i+1 132 represents an image of the duct view during navigation based on one or more actions. In other words, OOD detection determines whether the AI ​​agent 120 has seen this image before during training. The OOD detection component 108 prevents the AI ​​agent 120 from making random movements.

[0055] In one embodiment, in order to provide information about the next state s i+1The OOD estimation of 132 involves extracting patches from the training data (e.g., randomly) used to train the AI ​​agent 120. A binary label is applied to each corresponding patch using the following rules: First, if no blood vessels are depicted in the corresponding patch, it is labeled as 0. Second, if blood vessels are depicted in the corresponding patch, the AI ​​agent 120 is applied to that patch. If the AI ​​agent 120 is able to perform correct navigation (i.e., the movement is legal), the corresponding patch is labeled as 1; otherwise, it is labeled as 0. A binary classifier is then trained to classify the image as OOD using the labeled patches.

[0056] If the next state s i+1 If 132 is identified as OOD and therefore a risk, then at decision box 134, a flag 136 is generated to request user input. User input may include, for example, manual navigation of a conduit until a state that is not OOD is reached (i.e., within the distribution of the training data used to train the AI ​​agent 120). If the next state s i+1 If 132 is determined not to be OOD, then at decision box 134, workflow 100 continues to the conduit status assessment component 110.

[0057] At catheter status assessment component 110, the overall status of the catheter is assessed, including bending stress and the total number of guidewire punctures on the vessel wall since navigation began. High pressure or frequent punctures can cause severe bleeding or hematoma. Such damage to the patient may not be directly visualized from fluoroscopic images. (Refer to...) Figure 5 Description of catheter status assessment component 110, Figure 5 A workflow 500 for catheter condition assessment according to one or more embodiments is shown.

[0058] Condition assessment 138 assesses catheter bending stress by first segmenting the vessel and guidewire to extract the guidewire's relative position to the vessel. For example, as... Figure 5 As shown, in step 504, blood vessel and guidewire segmentation is performed from fluorescence imaging 502. The elastic potential of the guidewire is then estimated. Given a segmented guidewire (i.e., a line), the guidewire is discretized into segments. The elastic potential of the guidewire is estimated based on the angles between the segments as follows:

[0059]

[0060] Where E is the Young's modulus of the guidewire, I is the moment of inertia, and It is the angle between two consecutive segments. For example, as... Figure 5 As shown, discretization and force estimation are performed at step 506. The bending stress between the two segments is proportional to the angle between the two segments, i.e. Given a trajectory of a duct. By Defined as and The elastic potential of the trajectory s(T) can be calculated according to Equation 2 based on the angle between the joints. For example, at step 508, the bending potential estimate is determined based on the angle between the joints.

[0061] The total number of punctures made by the guidewire into the vessel wall can be determined using any suitable method. In one embodiment, the total number of punctures can be determined by the user manually examining the vessel wall (e.g., based on X-ray imaging), by automatically counting how many times a robot forcefully pushes against the vessel boundary (the definition of "force" is determined by the user), or by adding a small pressure sensor to the tip of the guidewire and counting the number of times the pressure exceeds a user-defined threshold.

[0062] If the bending stress of the guidewire or the number of punctures does not meet a predetermined threshold, status assessment 138 fails. If status assessment 138 fails, workflow 100 returns to decision box 128 to determine if the optimal trajectory is solvable. If not, a flag 130 is generated. For example, flag 130 could be an alarm for excessive stress on the user. If status assessment 138 passes, the catheter is navigated according to one or more of the determined actions.

[0063] exist Figure 2 At step 210, based on one or more actions based on the evaluation, the duct is navigated from its current position toward a target position using a robot navigation system. Method 200 can return to step 206 and repeat steps 206-210 for any number of iterations using the duct view as navigated based on one or more actions as the current view to iteratively navigate the duct to the target position.

[0064] Figure 6 A method 600 is illustrated for training a segmentation network according to one or more embodiments to segment blood vessels and quantify the level of uncertainty associated with the segmentation. The steps of method 600 can be performed by one or more suitable computing devices (such as, for example...). Figure 10 The process is performed using a computer (1002). The steps of method 600 are implemented to train the segmentation network in a previous offline or training phase. Once trained, the trained segmentation network is applied in an online or testing phase to segment blood vessels from an input medical image and quantify the level of uncertainty associated with the segmentation. In one example, method 600 can be implemented to train the network in a... Figure 2 Step 204 is used to generate the segmentation network for CDRM.

[0065] exist Figure 6 At step 602, the training image set is received. The training image set is labeled as x. iWhere i = 1, ..., n. The training images depict the patient's blood vessels. The training images can have any suitable modality, such as, for example, MRI, CT, US, X-ray and / or any other medical imaging modality or combination of medical imaging modalities. The training images can be 2D images or 3D volumes, and can comprise a single image or multiple images (e.g., forming a 2.5D image). The training images can be received directly from the image acquisition device (such as, for example, a CT scanner) when the image is acquired, or can be received by loading previously acquired images from the storage device or memory of a computer system (e.g., PACS) or by receiving images that have been transmitted from a remote computer system.

[0066] exist Figure 6 At step 604, an initial segmentation network M0 is trained based on a training image set annotated by a single user. The initial segmentation network M0 can be trained according to... Figure 3 The network architecture is implemented using 300.

[0067] exist Figure 6 At step 606, the trained initial segmentation network M0 is used to segment blood vessels from a subset of the training images.

[0068] exist Figure 6 At step 608, a variability distribution of annotations from multiple users is determined on a subset of training images based on the segmented blood vessels. The subset of training images can be (e.g., randomly) a sampled subset of the training image set. The variability distribution defines how many users have annotated each pixel (e.g., as background). In other words, the variability distribution is a binary distribution for each pixel, counting the number of users who annotated the pixel as either a blood vessel or background. For pixels far from or very clear about blood vessels, the distribution should be highly consistent, but for less clear pixels, such as those on minor blood vessel branches or blood vessel boundaries, variability occurs. The variability distribution can be determined by evaluating the accuracy of a trained initial segmentation network, which is done by comparing the segmented blood vessels determined using the trained initial segmentation network with annotations from multiple users. K-means clustering is then performed on the accuracy.

[0069] exist Figure 6 At step 610, annotations on the training image set from some users among multiple users are weighted based on a variable distribution. Some users can be selected (e.g., arbitrarily) from multiple users. The weighted annotations are denoted as... , where w i y represents the weight associated with a user i, while y represents the weight associated with a user i. i This represents a binary mask annotated by a user i. Weight w i This refers to aggregating comments from all users while boosting comments from more experienced commenters. Weight wi This can be user-defined, allowing larger weights to be assigned to more experienced users. All weights w i All are non-negative.

[0070] exist Figure 6 At step 612, the final segmentation network M is trained based on the training image set, weighted annotations, and the uncertainty associated with each weighted annotation. f The uncertainty associated with each weighted annotation is calculated as follows:

[0071]

[0072] Where x c This represents the centerline of the annotated vessel segmentation. According to Equation 3, the uncertainty of the annotated vessel segmentation is defined as the increase from the vessel centerline to the vessel boundary.

[0073] exist Figure 6 At step 614, the trained final segmentation network is output. For example, the trained final segmentation network can be output by storing it in the memory or storage device of a computer system, or by transmitting it to a remote computer system. This can be done during the online or inference phase (e.g., in...). Figure 2 (At step 204) the trained final segmentation network is applied to generate CDRM.

[0074] Sometimes, users can manually navigate the duct via a robot navigation system. Various embodiments are provided below to facilitate user navigation of the duct.

[0075] In one embodiment, trajectory-based navigation is provided. In trajectory-based navigation, a robot trajectory comprising a set of actions performed by the user at an initial time during the program can be replayed at any future time during the program. The robot trajectory is a combination of simple (e.g., translation, rotation, etc.) actions and / or complex (e.g., rapid rotation and push, rotation and retraction, etc.) actions performed by the user during the program. To record the robot trajectory, the configuration of the robot navigation system is stored (e.g., in memory or storage device) when the user performs the set of actions. The saved configuration of the robot navigation system can then be retrieved at a future point in time to replay the robot trajectory. Trajectory-bounded control can be used to replay the robot trajectory, such that the actions of the trajectory are reproduced in a forward or backward direction with acceleration or deceleration, etc. Trajectory-based navigation enables highly precise manipulation to be repeated with a simple and explicit one-dimensional user interface, which can be achieved using motion sensing, pedals, voice control, etc. Advantageously, for example, the robot trajectory can be stored at the start of the program for a particular conduit and replayed for other conduits in an over-the-wire setup. Furthermore, the stored trajectories can be used to enable non-specialist operators to maneuver the robot to handle potentially complex situations, such as in multi-operator setups in remote configurations. In cases where the robot manipulates an imaging catheter, the trajectory can be replayed to visualize the same location multiple times with high precision in procedures such as therapy delivery or complication monitoring.

[0076] In one embodiment, "follow me" navigation is provided. In this embodiment, the user does not directly manipulate the catheter, but instead selects a point on the user interface that is relatively close to the catheter's current position. The user then gradually pulls the catheter to the target position. For each selection, the path from the current position of the catheter tip to the selected point is calculated over the total path planned at the start of the program. The inverse kinematics from the tip position to the robot's joint space are calculated, and the calculated torque is applied to the robot to navigate the catheter tip to the selected position. This process can be repeated until the user stops at the selected point or until the target position is reached. Advantageously, "follow me" navigation provides an intuitive way to control the catheter because the user can focus on where the catheter should be navigated, rather than how to manipulate it. Therefore, the learning curve is significantly lower. Robot control can also be implemented on mobile devices, allowing for a more efficient arrangement of the operating room. Advanced safety mechanisms can also be implemented, for example, through watchdog rules, to prevent the catheter from puncturing blood vessels, moving in the wrong direction, etc.

[0077] In one embodiment, vision-based tactile feedback is provided based on real-time imaging, prior information from preoperative images (e.g., CT / MR images), computational modeling of blood vessels and vessel / catheter interactions, and robot state sensors. This vision-based tactile feedback allows the user to feel feedback from manipulating a joystick when the catheter navigates outside a safety margin (e.g., toward the vessel wall). To achieve vision-based tactile feedback, path planning in a vascular tree is calculated from preoperative images or multiple vascular views. If preoperative images are used, 3D / 2D deformable registration is performed to fuse the path plan to the patient's anatomy. If multiple vascular views are used, the vessel is segmented from angiography, point matching is performed between segments in the multiple views, and a 3D vascular lumen and centerline are reconstructed from the multiple views. The safety margin is then calculated from the path planning and the vascular lumen. The safety margin may take into account, for example, uncertainties in lumen segmentation and catheter tracking, the presence of vascular health / plaque, the known stiffness of the catheter, etc. During robotic manipulation, the catheter tip is automatically tracked. Visual and / or tactile feedback can be provided when the catheter tip is within the safety margin. Tactile feedback can be a force on a joystick navigating the catheter, proportional to the distance of the catheter tip within a safety margin. Alternatively, feedback can be a force on the joystick related to an equivalent force exerted on the catheter by the vessel wall, estimated using imaging and real-time computational modeling of vessel-catheter interactions. Preoperative images are re-registered to real-time images to maintain accuracy whenever needed (e.g., triggered by an accuracy watchdog module). With this system, the user receives tactile feedback that can potentially be more complete than what the user routinely feels while manipulating the catheter, as the tactile feedback will be directly correlated with the position of the catheter tip. Furthermore, feedback can be transmitted in various ways: visual, resistance in the joystick or other user interface, vibrations from mobile or wearable devices, etc.

[0078] In one embodiment, visualization of the blood vessel is provided. This embodiment provides visualization of catheter movement during the procedure via either "first-person" rendering or Mercator rendering. "First-person" rendering is a visualization of the catheter view. Mercator rendering is a flattened visualization of the blood vessel from a 3D preoperative image. To generate the visualization ("first-person" or Mercator), the preoperative image is co-registered with a real-time angiographic image. The following operations are performed in real-time (i.e., at the same frame rate as the fluorescence fluoroscopy image): 1) combining robotic information to perform real-time tracking of the catheter tip in the fluorescence fluoroscopy image; 2) mapping the catheter position to the preoperative image; 3) generating a "first-person" rendered view of the blood vessel; 4) projecting the catheter tip position onto the Mercator map of the blood vessel and visualizing the catheter on the map; and 5) combining the "first-person" rendering and Mercator rendering... Figure 2In this process, tissue types are color-coded based on, for example, image segmentation to facilitate navigation. Preoperative images are re-registered to real-time angiographic images to maintain accuracy when needed. According to one or more embodiments, Figure 7A Image 700, rendered in a "first-person" perspective, is shown. Figure 7B Image 710, rendered by Mercator, is shown. The visualizations provided in this paper allow for more precise analysis of the tissue environment surrounding the catheter, while enabling a more intuitive navigation method by removing the mental projection necessary to arrive at 3D vascular anatomy from projected angiographic / fluorescence images. Visualization helps maintain focus on the blood vessels and their environment, potentially improving efficiency, safety, and reducing the learning curve.

[0079] The embodiments described herein relate to the claimed system and the claimed method. Features, advantages, or alternative embodiments described herein can be assigned to other claimed objects, and vice versa. In other words, the system claims can be modified using features described or claimed in the context of the method. In this case, the functional characteristics of the method are embodied by the target unit providing the system.

[0080] Furthermore, some embodiments described herein relate to methods and systems for utilizing trained machine learning-based networks (or models), and to methods and systems for training machine learning-based networks. Features, advantages, or alternative embodiments described herein can be assigned to other claimed objects, and vice versa. In other words, the claims for methods and systems for training machine learning-based networks can be modified with features described or claimed in the context of methods and systems for utilizing trained machine learning-based networks, and vice versa.

[0081] Specifically, the trained machine learning-based networks used in the embodiments described herein can be adapted by methods and systems for training machine learning-based networks. Furthermore, the input data of the trained machine learning-based network can include advantageous features and embodiments of the training input data, and vice versa. Similarly, the output data of the trained machine learning network can include advantageous features and embodiments of the output training data, and vice versa.

[0082] Generally speaking, trained machine learning-based networks mimic human cognitive functions associated with other human thought processes. In particular, through training on training data, trained machine learning-based networks can adapt to new environments and detect and infer patterns.

[0083] Generally, the parameters of machine learning-based networks can be adapted through training. Specifically, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Furthermore, representation learning (an alternative term is "feature learning") can be used. In particular, the parameters of a trained machine learning-based network can be iteratively adapted through several training steps.

[0084] Specifically, the trained machine learning-based network can include neural networks, support vector machines, decision trees, and / or Bayesian networks, and / or the trained machine learning-based network can be based on k-means clustering, Q-learning, genetic algorithms, and / or association rules. Specifically, the neural network can be a deep neural network, a convolutional neural network, or a convolutional deep neural network. Furthermore, the neural network can be an adversarial network, a deep adversarial network, and / or a generative adversarial network.

[0085] Figure 8 An embodiment of an artificial neural network 800 according to one or more embodiments is shown. Alternative terms for "artificial neural network" are "neural network," "artificial neural network," or simply "neural network." The artificial neural network 800 can be used to implement the machine learning networks described herein, such as those used for generating... Figure 1 The CDRM 112, AI agent 120, and segmentation network used to determine the classification network for OOD data, the segmentation network utilized in step 204, the trained AI agent in step 206, and in Figure 2 The classification network used in step 208 Figure 3 The segmentation network shown and according to Figure 6 The method is to train a machine learning network of 600.

[0086] The artificial neural network 800 includes nodes 802-822 and edges 832, 834, ..., 836, where each edge 832, 834, ..., 836 is a directed connection from a first node 802-822 to a second node 802-822. Generally, the first node 802-822 and the second node 802-822 are different nodes 802-822, but it is also possible that the first node 802-822 and the second node 802-822 are the same. For example, in... Figure 8 In the diagram, edge 832 is a directed connection from node 802 to node 806, while edge 834 is a directed connection from node 804 to node 806. Edges 832, 834, ..., 836 from the first node 802-822 to the second node 802-822 are also labeled as "input edges" of the second node 802-822 and "output edges" of the first node 802-822.

[0087] In this embodiment, nodes 802-822 of the artificial neural network 800 can be arranged in layers 824-830, wherein the layers can include an inherent order introduced by edges 832, 834, ..., 836 between nodes 802-822. Specifically, edges 832, 834, ..., 836 may exist only between adjacent node layers. Figure 8 In the illustrated embodiment, input layer 824 consists only of nodes 802 and 804 with no input edges, output layer 830 consists only of node 822 with no output edges, and hidden layers 826 and 828 are located between input layer 824 and output layer 830. Generally, the number of hidden layers 826 and 828 can be arbitrarily chosen. The number of nodes 802 and 804 in input layer 824 is typically related to the number of input values ​​of neural network 800, and the number of nodes 822 in output layer 830 is typically related to the number of output values ​​of neural network 800.

[0088] Specifically, (real) numbers can be assigned as values ​​to each node 802-822 of the neural network 800. Here, x （n） i This represents the value of the i-th node 802-822 in the n-th layer 824-830. The values ​​of nodes 802-822 in the input layer 824 are equivalent to the input values ​​of the neural network 800, and the value of node 822 in the output layer 830 is equivalent to the output value of the neural network 800. Furthermore, each edge 832, 834, ..., 836 may include a weight as a real number, specifically a real number within the interval [-1, 1] or the interval [0, 1]. Here, w... （m,n） i，j This indicates the weight of the edge between the i-th node 802-822 in layer m (824-830) and the j-th node 802-822 in layer n (824-830). Additionally, the abbreviation w... （n） i，j Defined as weight w （n,n+1） i，j .

[0089] Specifically, to calculate the output value of neural network 800, the input values ​​are propagated through the neural network. Specifically, the values ​​of nodes 802-822 in the (n+1)th layer 824-830 can be calculated based on the values ​​of nodes 802-822 in the nth layer 824-830 using the following formula.

[0090] .

[0091] In this paper, the function f is the transfer function (another term is "activation function"). Known transfer functions are step functions, sigmoid functions (e.g., logic functions, generalized logic functions, hyperbolic tangent functions, arctangent functions, error functions, smooth step functions), or correction functions. Transfer functions are primarily used for standardization purposes.

[0092] Specifically, the values ​​are propagated layer by layer through the neural network, where the value of the input layer 824 is given by the input of the neural network 800, the value of the first hidden layer 826 can be calculated based on the value of the input layer 824 of the neural network, the value of the second hidden layer 828 can be calculated based on the value of the first hidden layer 826, and so on.

[0093] To set the value w of the edge （m,n） i，j The neural network 800 must be trained using training data. Specifically, the training data includes training input data and training output data (denoted as t). i For the training step, the neural network 800 is applied to the training input data to generate computational output data. Specifically, the training data and the computational output data include multiple values, the number of which is equal to the number of nodes in the output layer.

[0094] Specifically, the comparison between the calculated output data and the training data is used to recursively adapt the weights within the neural network 800 (backpropagation algorithm). Specifically, the weights change according to the following formula:

[0095]

[0096] Where γ is the learning rate, and the number It can be recursively calculated as

[0097]

[0098] based on If the (n+1)th layer is not an output layer, and

[0099]

[0100] If the (n+1)th layer is the output layer 830, where f' is the first derivative of the activation function, and y （n+1） j It is the comparison training value of the j-th node of the output layer 830.

[0101] Figure 9 A convolutional neural network 900 according to one or more embodiments is illustrated. The convolutional neural network 900 can be used to implement the machine learning networks described herein, such as those for generating... Figure 1The CDRM 112, AI agent 120, and segmentation network used for classifying OOD data, the segmentation network utilized in step 204, and the trained AI agent in step 206 are all part of this framework. Figure 2 The classification network used in step 208 Figure 3 The segmentation network shown and according to Figure 6 The method is to train a machine learning network of 600.

[0102] exist Figure 9 In the illustrated embodiment, the convolutional neural network 900 includes an input layer 902, a convolutional layer 904, a pooling layer 906, a fully connected layer 908, and an output layer 910. Alternatively, the convolutional neural network 900 may include several convolutional layers 904, several pooling layers 906, and several fully connected layers 908, as well as other types of layers. The order of the layers can be arbitrarily chosen; typically, the fully connected layer 908 is used as the last layer before the output layer 910.

[0103] Specifically, within the convolutional neural network 900, nodes 912-920 of layer 902-910 can be viewed as arranged as a d-dimensional matrix or a d-dimensional image. In particular, in the two-dimensional case, the values ​​of nodes 912-920 indexed by i and j in the nth layer 902-910 can be denoted as x. （n） [i，j] However, the arrangement of nodes 912-920 in layer 902-910 has no impact on the computations performed within the convolutional neural network 900, because these are given only by the structure and weights of the edges.

[0104] Specifically, the convolutional layer 904 is characterized by the structure and weights of the input edges that form the convolution operation based on a certain number of kernels. In particular, the structure and weights of the input edges are chosen such that the value x of node 914 of the convolutional layer 904... （n） k The value x is calculated as the value of node 912 based on the previous layer 902. （n-1） convolution Where convolution* is defined in the two-dimensional case as

[0105] .

[0106] Here, the k-th core K kThe kernel is a d-dimensional matrix (two-dimensional in this embodiment), which is typically small compared to the number of nodes 912-918 (e.g., a 3×3 or 5×5 matrix). Specifically, this means that the weights of the input edges are not independent but are chosen such that they produce the convolution equation. Specifically, for the kernel, which is a 3×3 matrix, there are only 9 independent weights (each entry in the kernel matrix corresponds to one independent weight), regardless of the number of nodes 912-920 in the corresponding layers 902-910. Specifically, for convolutional layer 904, the number of nodes 914 in the convolutional layer is equal to the number of nodes 912 in the previous layer 902 multiplied by the number of kernels.

[0107] If the nodes 912 of the previous layer 902 are arranged as a d-dimensional matrix, using multiple kernels can be interpreted as adding another dimension (denoted as the "depth" dimension), so that the nodes 914 of the convolutional layer 904 are arranged as a (d+1)-dimensional matrix. If the nodes 912 of the previous layer 902 are already arranged as a (d+1)-dimensional matrix including the depth dimension, then using multiple kernels can be interpreted as extending along the depth dimension, so that the nodes 914 of the convolutional layer 904 are also arranged as a (d+1)-dimensional matrix, where the size of the (d+1)-dimensional matrix with respect to the depth dimension is a multiple of the number of larger kernels in the previous layer 902.

[0108] The advantage of using convolutional layer 904 is that it can take advantage of the spatial local correlation of the input data by implementing local connection patterns between nodes of adjacent layers, in particular by having each node connect only to a small region of the node in the previous layer.

[0109] exist Figure 9 In the illustrated embodiment, the input layer 902 comprises 36 nodes 912 arranged in a two-dimensional 6×6 matrix. The convolutional layer 904 comprises 72 nodes 914 arranged in two two-dimensional 6×6 matrices, each of which is the result of the convolution of the input layer values ​​with the kernel. Equivalently, the nodes 914 of the convolutional layer 904 can be interpreted as arranged in a three-dimensional 6×6×2 matrix, where the last dimension is the depth dimension.

[0110] The characteristics of pooling layer 906 can be found in the structure and weights of the input edges and the activation function of its nodes 916, forming a pooling operation based on a nonlinear pooling function f. For example, in the two-dimensional case, the value x of node 916 in pooling layer 906... （n） It can be based on the value x of node 914 of the previous layer 904. （n-1） Calculated as

[0111]

[0112] In other words, by using pooling layer 906, the number of nodes 914 and 916 can be reduced by replacing the number of adjacent nodes 914 in the previous layer 904 with a single node 916. The individual node 916 is calculated as a function of the number of adjacent nodes in the pooling layer. Specifically, the pooling function f can be a maximum function, an average function, or an L2 norm function. Specifically, for pooling layer 906, the weights of the input edges are fixed and are not modified through training.

[0113] The advantage of using pooling layer 906 is that it reduces the number of nodes 914 and 916 and the number of parameters. This leads to a reduction in the computational cost of the network and controls overfitting.

[0114] exist Figure 9 In the illustrated embodiment, pooling layer 906 is max pooling, replacing four adjacent nodes with only one node, where the value is the maximum of the four adjacent nodes. Max pooling is applied to each d-dimensional matrix of the previous layer; in this embodiment, max pooling is applied to each of the two 2D matrices, reducing the number of nodes from 72 to 18.

[0115] The characteristic of the fully connected layer 908 is that there are most, in particular all, edges between the node 916 of the previous layer 906 and the node 918 of the fully connected layer 908, and the weight of each edge can be adjusted individually.

[0116] In this embodiment, the nodes 916 of the preceding layer 906 of the fully connected layer 908 are displayed as a two-dimensional matrix, and additionally as unrelated nodes (indicated as a row of nodes, where the number of nodes is reduced for better presentation). In this embodiment, the number of nodes 918 in the fully connected layer 908 is equal to the number of nodes 916 in the preceding layer 906. Alternatively, the number of nodes 916 and 918 can be different.

[0117] Furthermore, in this embodiment, the value of node 920 in output layer 910 is determined by applying the Softmax function to the value of node 918 in the previous layer 908. By applying the Softmax function, the sum of the values ​​of all nodes 920 in output layer 910 is 1, and all values ​​of all nodes 920 in output layer 910 are real numbers between 0 and 1.

[0118] The convolutional neural network 900 may also include ReLU (Modified Linear Unit) layers or activation layers with non-linear transfer functions. Specifically, the number and structure of nodes in the ReLU layer are identical to those in the previous layer. In particular, the value of each node in the ReLU layer is calculated by applying a correction function to the value of the corresponding node in the previous layer.

[0119] The inputs and outputs of different convolutional neural network blocks can be connected using summation (residual / dense neural networks), element-wise multiplication (note), or other differentiable operators. Therefore, if the entire pipeline is differentiable, the convolutional neural network architecture can be nested rather than sequential.

[0120] Specifically, a convolutional neural network 900 can be trained based on the backpropagation algorithm. To prevent overfitting, regularization methods can be used, such as dropping nodes 912-920, random pooling, the use of artificial data, and weight decay based on L1 or L2 norm or maximum norm constraints. Different loss functions can be combined to train the same neural network to reflect the joint training objective. A subset of neural network parameters can be excluded from optimization to retain weights pre-trained on another dataset.

[0121] The systems, apparatus, and methods described herein can be implemented using digital circuitry or using one or more computers employing well-known computer processors, storage units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include or be coupled to one or more mass storage devices, such as one or more disks, internal hard disks and removable disks, magneto-optical disks, optical disks, etc.

[0122] The systems, apparatus, and methods described herein can be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computer is located remotely from the server computer and interacts via a network. The client-server relationship can be defined and controlled by computer programs running on the respective client and server computers.

[0123] The systems, apparatus, and methods described herein can be implemented in a network-based cloud computing system. In such a system, a server or other processor connected to the network communicates with one or more client computers via the network. For example, a client computer may communicate with the server via a web browser application residing on and operating on the client computer. The client computer may store data on the server and access the data via the network. The client computer may transmit data requests or online service requests to the server via the network. The server may perform the requested service and provide data to one or more client computers(s). The server may also transmit data suitable for enabling the client computer to perform specified functions, such as performing calculations, displaying specified data on a screen, etc. For example, the server may transmit requests suitable for enabling the client computer to perform one or more steps or functions of the methods and workflows described herein, including... Figure 1-2And 6 or more steps or functions. Some steps or functions of the methods and workflows described herein include Figure 1-2 One or more steps or functions of 6 can be performed by a server or another processor in a web-based cloud computing system. Some steps or functions of the methods and workflows described herein include... Figure 1-2 One or more steps of 6 can be performed by a client computer in a web-based cloud computing system. The steps or functions of the methods and workflows described herein include... Figure 1-2 One or more steps of 6 can be performed by servers and / or client computers in any combination in a web-based cloud computing system.

[0124] The systems, apparatuses, and methods described herein can be implemented using computer program products tangibly embodied in an information carrier, for example, executed by a programmable processor in a non-transitory machine-readable storage device; and the methods and workflow steps described herein include Figure 1-2 One or more steps or functions of 6 can be implemented using one or more computer programs executable by such a processor. A computer program is a set of computer program instructions that can be used directly or indirectly in a computer to perform an activity or produce a result. Computer programs can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0125] exist Figure 10 A high-level block diagram of an example computer 1002, which can be used to implement the systems, apparatus, and methods described herein, is depicted. Computer 1002 includes a processor 1004 operatively coupled to a data storage device 1012 and a memory 1010. Processor 1004 controls the overall operation of computer 1002 by executing computer program instructions that define such operations. The computer program instructions may be stored in data storage device 1012 or other computer-readable medium and loaded into memory 1010 when execution of the computer program instructions is desired. Therefore, Figure 1-2 The methods and workflow steps or functions of 6 can be defined by computer program instructions stored in memory 1010 and / or data storage device 1012, and controlled by processor 1004 that executes the computer program instructions. For example, the computer program instructions can be implemented as computer-executable code programmed by a person skilled in the art to perform... Figure 1-2 The methods and workflow steps or functions of 6. Therefore, by executing computer program instructions, processor 1004 performs... Figure 1-2The computer 1002 may also include one or more network interfaces 1006 for communicating with other devices via a network. The computer 1002 may also include one or more input / output devices 1008 (e.g., monitor, keyboard, mouse, speakers, buttons, etc.) that enable a user to interact with the computer 1002.

[0126] Processor 1004 may include both general-purpose and special-purpose microprocessors, and may be the sole processor of computer 1002 or one of multiple processors. For example, processor 1004 may include one or more central processing units (CPUs). Processor 1004, data storage device 1012 and / or memory 1010 may include one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs), or may be supplemented or incorporated therein by one or more ASICs and / or one or more FPGAs.

[0127] Each of the data storage device 1012 and the memory 1010 includes a tangible, non-transitory, computer-readable storage medium. Each of the data storage device 1012 and the memory 1010 may include high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), dual data rate synchronous dynamic random access memory (DDR RAM), or other random access solid-state storage devices, and may include non-volatile memory, such as one or more disk storage devices, such as internal hard disks and removable disks, magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), digital universal disc read-only memory (DVD-ROM), or other non-volatile solid-state storage devices.

[0128] Input / output device 1008 may include peripheral devices such as printers, scanners, displays, etc. For example, input / output device 1008 may include display devices such as cathode ray tube (CRT) or liquid crystal display (LCD) monitors for displaying information to a user, keyboards, and pointing devices such as mice or trackballs, through which the user can provide input to computer 1002.

[0129] Image acquisition device 1014 can be connected to computer 1002 to input image data (e.g., medical images) into computer 1002. It is possible to implement image acquisition device 1014 and computer 1002 as a single device. Image acquisition device 1014 and computer 1002 may also communicate wirelessly via a network. In a possible embodiment, computer 1002 may be remotely located relative to image acquisition device 1014.

[0130] Any or all of the systems and devices discussed herein can be implemented using one or more computers (such as computer 1002).

[0131] Those skilled in the art will recognize that the implementation of an actual computer or computer system may have other structures and may include other components, and Figure 10 It is a high-level representation of some components of such a computer for illustrative purposes.

[0132] The foregoing detailed description should be understood as illustrative and exemplary in each respect, and not restrictive, and the scope of the invention disclosed herein is not determined by the detailed description, but by the claims as interpreted in their full breadth as permitted under patent law. It will be understood that the embodiments shown and described herein are merely illustrative of the principles of the invention, and various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention. Various other combinations of features can be implemented by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. An apparatus comprising: A component used to receive medical images input by the patient; A component used to generate a trajectory for navigating a catheter from its current location within the patient to a target location based on an input medical image, using a trained segmentation network; A component used to determine, through a trained AI (artificial intelligence) agent and based on the generated trajectory and the current view of the duct, 1) one or more actions of a robotic navigation system for navigating the duct from its current position toward a target position, and 2) the confidence level associated with the one or more actions; A component for evaluating one or more actions based on a duct view during navigation, in response to a confidence threshold being met; and A component used to navigate a conduit from its current position toward a target position using a robot navigation system based on one or more actions evaluated.

2. The apparatus of claim 1, wherein the component for generating a trajectory for navigating a catheter from its current location within the patient to a target location based on an input medical image using a trained segmentation network comprises: A component for generating a color dynamic route map of blood vessels in an input medical image, the color dynamic route map including color coding to indicate uncertainty.

3. The apparatus of claim 2, wherein the uncertainty is quantified by a trained segmentation network, the trained segmentation network being trained by: A component used to receive the training image set; Components used to train an initial segmentation network based on a set of training images annotated by a single user; A component used to segment blood vessels from a subset of the training image set using an initially trained segmentation network; A component for determining the variability distribution of annotations from multiple users for a subset of training images based on segmented blood vessels; A component for weighting annotations of training image sets from some users among multiple users based on a variable distribution; Components used to train the final segmentation network based on the training image set, weighted annotations, and the uncertainty associated with each weighted annotation; and The component used to output the final segmentation network after training.

4. The apparatus according to claim 1, further comprising: In response to confidence levels not meeting the threshold: Components used to move the AI ​​agent to a previous position in the trajectory; Components used to optimize the trajectory of the catheter from its previous position to its target position; The component used to place the AI ​​agent in the position closest to its current location on the optimal trajectory; and A component used by an AI agent to restart duct navigation by using the duct view at the location closest to the current position in the optimal trajectory as the current view.

5. The apparatus of claim 4, wherein the components for optimizing the trajectory of navigating the catheter from a previous position to a target position include: A component for optimizing trajectories based on possible actions of a robot navigation system and vascular segmentation from input medical images.

6. The apparatus of claim 1, wherein the component for evaluating the one or more actions based on a duct view during navigation of the one or more actions comprises: A component used to determine whether the duct view during navigation based on the one or more actions is outside the domain of the training data used to train the trained AI agent.

7. The apparatus of claim 1, wherein the component for evaluating the one or more actions based on a duct view during navigation of the one or more actions comprises: A component used to assess the bending stress and number of punctures of the guidewire used in navigation catheters.

8. The apparatus according to claim 1, further comprising: A component used to store the configuration of the robot navigation system when the user performs a set of actions on the navigation conduit; and A component for storing and replaying action sets based on robot navigation systems.

9. A non-transitory computer-readable medium storing computer program instructions, which, when executed by a processor, cause the processor to perform operations, comprising: Receive medical images input from the patient; A trained segmentation network is used to generate a trajectory based on the input medical image to navigate the catheter from its current location within the patient to a target location. The trained AI (artificial intelligence) agent, based on the generated trajectory and the current view of the duct, determines 1) one or more actions of the robotic navigation system used to navigate the duct from its current position toward a target position, and 2) the confidence associated with one or more actions; In response to a confidence threshold being met, the one or more actions are evaluated based on the duct view during navigation according to the one or more actions; and Based on the evaluation-based actions, the robot navigation system navigates the conduit from its current position toward a target position.

10. The non-transitory computer-readable medium of claim 9, wherein generating a trajectory for navigating a catheter from its current location within the patient to a target location based on an input medical image using a trained segmentation network comprises: Generate a color-coded, dynamic route map of blood vessels in an input medical image, including color coding to indicate uncertainty.

11. The non-transitory computer-readable medium of claim 9, wherein the operation further comprises: In response to confidence levels not meeting the threshold: Move the AI ​​agent to its previous position in the trajectory; Optimize the trajectory used to navigate the catheter from its previous position to the target position; Place the AI ​​agent at the position closest to the current position on the optimal trajectory; and The AI ​​agent restarts duct navigation by using the duct view at the location closest to the current position in the optimal trajectory as the current view.

12. The non-transitory computer-readable medium of claim 9, wherein evaluating the one or more actions based on a duct view during navigation of the one or more actions comprises: Determine whether the duct view during navigation based on the one or more actions is outside the domain of the training data used to train the trained AI agent.

13. The non-transitory computer-readable medium of claim 9, wherein evaluating the one or more actions based on a duct view during navigation of the one or more actions comprises: Assess the bending stress of the guidewire used for the navigation catheter and the number of punctures.

14. The non-transitory computer-readable medium of claim 9, wherein the operation further comprises: Receive user input selecting the location to be navigated by the catheter; Calculate the kinematics used to navigate the duct to the selected location; and The duct is navigated to the selected location based on the calculated kinematics.

15. The non-transitory computer-readable medium of claim 9, wherein the operation further comprises: Calculate the safety margin of paths in the input medical image; and Tactile feedback is provided to the user of the navigation catheter based on the catheter's current position relative to the safety margin.

Images