Positioning Controller for Interventional Devices

Abstract

A positioning controller (50) including an imaging prediction model (80) and an inverse contrast prediction model (70). In operation, the controller (50) applies the imaging prediction model (80) to imaging data generated by an imaging device (40) to present a predicted navigation pose of the imaging device (40), and applies the contrast prediction model (70) to error positioning data derived in terms of a difference between a target pose of the imaging device (40) and the predicted navigation pose of the imaging device (40) to present a predicted corrective positioning movement of the imaging device (40) (or a part of the interventional device associated with the imaging device) to the target pose. According to the prediction result, the controller (50) further generates a continuous positioning command for controlling the corrective positioning of the imaging device (40) (or the part of the interventional device) to the target pose by the interventional device (30) based on the predicted corrective positioning movement of the interventional device (30).

Description

Technical Field

[0001] The present disclosure generally relates to position control of a portion (e.g., an end effector of an interventional device) of an interventional device used in an interventional procedure (e.g., minimally invasive surgery, video-assisted thoracic surgery, vascular procedure, endoluminal procedure, deformity correction procedure). The present disclosure can particularly relate to incorporating a predictive model in the position control of such a portion of an interventional device used in an interventional procedure. Background Art

[0002] Continuous (or non - continuous) control - positioning the device portion (e.g., end effector) within a certain workspace - is one of the most commonly attempted forms of control in conventional rigid - link robots. By utilizing the discrete rigid - link structure of the robot, precise positioning of the portion (e.g., end effector) of the interventional device can be achieved as needed in structured applications (e.g., manufacturing). However, due to the deformable and fragile nature of soft - tissue human organs and patient safety, the use of rigid - link robots in a clinical environment is less desirable.

[0003] More specifically, biologically inspired robots can produce movements similar to those of snakes, elephants, and octopuses, which can be very effective in manipulating soft anatomical objects. Nevertheless, due to the complexity of the continuum (or quasi - continuum) structure for the desired degrees of freedom being difficult to mathematically model or difficult to provide sufficient actuator inputs to achieve consistent control, effective control (and particularly effective continuous control) of the robotic structure in a clinical environment has proven to be extremely difficult to achieve.

[0004] For example, a method for continuous position control of an end effector supported by a continuum robot involves modeling and controlling the configuration of the continuum robot, where a static model (formulated as a set of non - linear differential equations) attempts to account for the lance deformation of the continuum robot due to bending, torsion, and elongation. However, the accuracy of the mathematical modeling is susceptible to changes in the environmental conditions (e.g., temperature and humidity) of the robot, which can affect the mechanical properties of the robot components, and is also susceptible to any manufacturing errors or various workloads.

[0005] As another example, another method for positioning control of a robot-supported end effector is to manipulate the robot by projecting a set of allowable motions and a set of allowable forces into the joint space corresponding to the operator's control. For example, after inserting the robot into the nasal cavity, the controller adjusts the position of each segment of the robot to increase the difference between the measured generalized force and the expected generalized force on each end plate. However, increasing the degrees of freedom of the robot to make it more maneuverable has the adverse effect of complicating the robot kinematics. This can be particularly problematic in the case of continuously controlling a continuum robot.

[0006] In addition, even if effective positioning control of the robot-supported end effector is achieved, incorrect calibration or use of the robot system or general wear of mechanical components will have a negative impact on the prediction accuracy of the kinematics of the robot structure. Similarly, this can be particularly problematic in the case of continuously controlling a continuum robot (or continuum robot structure).

[0007] EP 2942029 A1 describes a surgical robot that includes: an image information acquisition unit configured to acquire image information of the intra-abdominal environment while the surgical robot performs a surgical operation; and / or a controller configured to identify the positions of the endoscope and the tool based on the acquired image information and kinematic information of the linkages included in the endoscope and the tool mounted on the surgical robot.

[0008] US2006 / 258938 A1 describes performing tool tracking during minimally invasive robotic surgery. The tool state is determined using triangulation techniques or Bayesian filters from any one or both of non-endoscope-derived tool state information and endoscope-derived tool state information, or based on any one or both of non-vision-derived tool state information and vision-derived tool state information.

[0009] US2017 / 334066 A1 describes predicting the (one or more) motions (if any) that will occur to (one or more) objects in response to a particular movement of a robot in the robot's environment.

[0010] US2018 / 078318 A1 describes a method performed by a computing system that includes receiving shape information for an elongate flexible portion of a medical device. The medical device includes a reference portion movably coupled to a fixture having a known pose in a surgical reference frame. The fixture includes a constraint structure having a known constraint structure position in the surgical reference frame. The elongate flexible portion is coupled to the reference portion and is sized to pass through the constraint structure. The method further includes: receiving reference portion position information in the surgical reference frame; determining an estimated constraint structure position in the surgical reference frame based on the reference portion position information and the shape information; determining a correction factor by comparing the estimated constraint structure position to the known constraint structure position; and modifying the shape information based on the correction factor. SUMMARY OF THE INVENTION

[0011] Known techniques designed to provide position control for parts of an intervention device (e.g., an end effector) provide limited benefits. Accordingly, there is still a need for improved techniques for providing effective position control for these parts of an intervention device. To that end, the present disclosure teaches feedforward position control, feedback position control, and data collection. The control is preferably performed continuously.

[0012] To that end, the present invention provides, as a first embodiment, a position controller for an intervention device that includes an imaging device.

[0013] As a second and third embodiment, the present invention provides an (optionally non-transitory) machine-readable storage medium encoded with instructions and a positioning method executable by a position controller for an intervention device. The examples listed below are for assisting in understanding the present invention.

[0014] Feedforward (preferably continuous) positioning control The present disclosure also teaches a predictive model method for feedforward (preferably continuous) position control of manual or automated navigation positioning of a device part (e.g., an end effector) supported by an intervention device based on a predictive model configured (optionally trained on) the kinematics of the intervention device.

[0015] Another example of the feedforward (preferably continuous) positioning control for a device part (e.g., an end effector) of the present disclosure is a (continuous) positioning controller that includes a forward prediction model and / or a control (optionally inverse) prediction model. The forward prediction model is configured (optionally trained on these forward kinematics of the intervening device) using the (optionally forward) kinematics of the intervening device that predicts the navigation pose of the end effector. The control (optionally inverse) prediction model is configured (optionally trained on the inverse kinematics of the intervening device that predicts the positioning motion of the intervening device) using the kinematics of the intervening device that predicts the positioning motion of the intervening device.

[0016] For the description and purposes of the present disclosure, the term "navigation pose" broadly encompasses the pose of the said part of the intervening device (e.g., the end effector of the intervening device) when the intervening device is navigated to a spatial position via the intervening device during an intervention procedure, and the term "positioning motion" broadly encompasses any movement of the intervening device used to navigate the device part to a spatial position during an intervention procedure.

[0017] In operation, the continuous positioning controller applies the forward prediction model to the commanded positioning motion of the intervening device to present the predicted navigation pose of the end effector and generates continuous positioning data regarding the information of positioning the device part to a target pose by the intervening device based on the predicted navigation pose of the said device part.

[0018] Alternatively, antecedently or simultaneously, the (preferably continuous) positioning controller applies the inverse prediction model to the target pose of the said part of the intervening device (e.g., the end effector) to present the predicted positioning motion of the intervening device and generates a (continuous) positioning command for controlling the positioning of the device part to the target pose by the intervening device based on the predicted positioning motion of the intervening device.

[0019] Feedback (preferably continuous) positioning control . The present disclosure also teaches a prediction model method for (preferably continuous) positioning control of the feedback of the manual navigation positioning or (semi-)automatic navigation positioning of an imaging device associated with or attached to a part of an intervening device (e.g., the end effector of the intervening device) based on the following imaging prediction model: The imaging prediction model is configured using the kinematics of the intervening device (optionally related to image data) to receive imaging data from the imaging device as feedback for the manual navigation positioning or automatic navigation positioning of the imaging device (or the part of the intervening device linked to the imaging device - e.g., the end effector) to a target pose.

[0020] One example of the feedback (preferably continuous) control positioning for an imaging device (or a part of an interventional device linked to the imaging device, e.g., an end effector) of the present disclosure is a continuous positioning controller that includes an imaging prediction model trained on the correlation between relative imaging performed by the end effector and the forward kinematics of the interventional device predicting the navigation pose of the end effector. The (continuous) positioning controller may further include a control prediction model configured using the kinematics of the interventional device predicting the corrective positioning movement of the interventional device. Optionally, the control prediction model is trained or has been trained on the inverse kinematics of the interventional device to output the prediction of the corrective positioning movement.

[0021] For the description and purposes of the present disclosure, the term "relative imaging" broadly encompasses the generation of an image of an interventional procedure by an end effector in a given pose relative to a reference image of the interventional procedure.

[0022] In operation, after a device part (e.g., an end effector) is navigated to a target pose, the (preferably) continuous positioning controller applies the imaging prediction model to the imaging data generated by the end effector to present the predicted navigation pose of the device part (e.g., the end effector), applies the control (or particularly inverse) prediction model to error positioning data derived in terms of the difference between the target pose of the end effector and the predicted navigation pose of the end effector to present the predicted corrective positioning movement of the interventional device, and generates a continuous positioning command for controlling the corrective positioning of the imaging device (40) or the part of the interventional device associated with the imaging device (e.g., the end effector) to the target pose based on the predicted corrective positioning movement of the interventional device.

[0023] Training data collection In addition, to facilitate the (optionally continuous) positioning control of the manual navigation positioning or the automatic navigation positioning of a part of the interventional device (e.g., an end effector) via a prediction model invariant to environmental differences (e.g., anatomical differences between patients, e.g., patient body size, heart position, etc.), the present disclosure also teaches (optionally trained) data collection techniques, which are premised on navigating the positioning of the part of the interventional device via a predefined pattern of data points and recording the spatial positioning of the interventional device and the pose of the part of the interventional device at each data acquisition point, so as to collect (train) data for the prediction model to infer the forward kinematics or the inverse kinematics of the interventional device.

[0024] One example of collecting (optionally training) data for a prediction model in the present disclosure is a (optionally training) data collection system for an interventional device, the interventional device including the portion of the interventional device (e.g., the end effector) and a sensor adapted to provide position and / or orientation and / or shape information, at least a portion of the sensor (332) being attached (optionally in a fixed shape) to the interventional device portion. Such sensors can include markers visible from imaging systems (e.g., X-ray systems, MRI systems), electromagnetic tracking sensors, transducer sensors, and / or optical shape sensing provided by / within optical fibers.

[0025] A specific example of this example is a (optionally training) data collection system for an interventional device, the interventional device including a portion of the interventional device (e.g., the end effector) and an optical shape sensor, wherein segments of the optical shape sensor are attached (optionally in a fixed shape) to the end effector.

[0026] (Training) data collection systems can use a robotic controller, a data acquisition controller, a position determination module (or a shape sensing controller in the specific example mentioned above), and a data storage controller.

[0027] In operation, the data acquisition controller can command the robotic controller to control the (one or more) motion variables of the interventional device according to a predefined data point pattern, and the position determination module (or shape sensing controller) is configured to determine position information based on the position and / or orientation and / or shape information received from the sensor so as to output an estimated result of the pose of the portion of the interventional device portion at each data point of the predefined data point pattern and / or an estimated result of the positioning motion of the interventional device. Thus, for the purpose of the estimation, the determined position information is optionally retrieved or derived or extracted or received based on the kinematics or the behavior of the interventional device configuring the position determination module. In a specific example, the position determination module can derive or receive derived shape data according to the position and / or orientation and / or shape information to determine the estimated result. If the position determination module is a shape sensing controller (as in the specific example mentioned above), then the position determination module controls the shape sensing of the optical shape sensor, including the estimation of the pose of the end effector at each data point of the predefined data point pattern and the estimation of the positioning motion of the interventional device at each data point of the predefined data point pattern.

[0028] The act of "deriving shape data based on the position and / or orientation and / or shape information" can be implemented according to known techniques for deriving the shape from data provided by "sensors". For example, the positions tracked by sensors can well indicate the general shape of the segments of the intervention device carrying these sensors, and algorithms (more or less developed based on the distances between the sensors and the possible shapes of the intervention device along the segment) can be developed to derive or reconstruct the shape. This dynamic tracking of the positioning can also give an indication of the orientation of the deformation. The sensors can also provide strain information (e.g., Rayleigh or Bragg grating sensors), which can indicate the local positioning and orientation of the intervention device, based on which the shape can be derived and reconstructed (also known techniques).

[0029] The estimation result of the pose of the intervention device part (e.g., the end effector) is derived based on at least part of the sensors (optionally in a fixed shape) attached to the end effector.

[0030] The robot controller can always control the (one or more) motion variables of the intervention device according to a predefined data point pattern. The data storage controller can receive communication information of the estimated pose of the end effector for each data point from the shape sensing controller, can receive communication information of the estimated positioning motion of the intervention device for each data point from the positioning determination module (or shape sensing controller), and can receive communication information of at least one motion variable of the intervention device for each data point from the robot controller.

[0031] In response to this communication information, the data storage controller can store a time data series for the intervention device derived based on the estimated pose of the end effector at each data point, the estimated spatial positioning of the intervention device at each data point, and the (one or more) motion variables of the intervention device at each data point. This time data series can serve as training data for any type of machine learning model (especially the prediction model of the present disclosure).

[0032] In addition, the data acquisition controller can further command the robot controller to control the (one or more) motion variables of the intervention device according to an additional (one or more) predefined data point pattern, whereby the data storage controller generates an additional time data series for any type of machine learning model (especially the prediction model of the present disclosure).

[0033] Moreover, for the description of the present disclosure:

[0034] (1) Terms in the art including, but not limited to, "end effector", "kinematics", "position", "positioning", "orientation", "achieving an orientation", "motion", and "navigation" are construed to be known in the field of the present disclosure and are described exemplarily in the present disclosure;

[0035] (2) Examples of end effectors include, but are not limited to, intraoperative imaging devices, interventional tools / surgical instruments, and surgical sutures as known in the field of the present disclosure and contemplated hereinafter;

[0036] (3) The term "intraoperative imaging device" broadly encompasses all imaging devices for visualizing anatomical objects / regions as known in the field of the present disclosure and contemplated hereinafter. Examples of intraoperative imaging devices include, but are not limited to, transesophageal echocardiography transducers (e.g., X7-2t transducer, Philips), laparoscopic ultrasound transducers (e.g., L10-4lap transducer, Philips), optical cameras, and sensing devices (e.g., tissue spectral sensing sensors, ECG electrodes, probes for electrophysiological mapping);

[0037] (4) Examples of interventional tools / surgical instruments include, but are not limited to, scalpels, cauterizers, ablation devices, needles, forceps, Kirschner wires and associated drivers, endoscopes, awls, screwdrivers, osteotomes, chisels, stylets, curettes, clamps, pliers, periosteal elevators, and J-wires as known in the field of the present disclosure and contemplated hereinafter;

[0038] (5) The term "interventional device" broadly encompasses all devices for supporting the positioning of an end effector during application as known in the field of the present disclosure and contemplated hereinafter. Examples of interventional devices include, but are not limited to, continuum flexible robots, flexible endoscopes, and guidewires;

[0039] (6) Examples of flexible robots include, but are not limited to, hyper-redundant robots (e.g., multiple discrete links, serpentine links, or concentric tubes), continuum backbone robots (e.g., cable actuated), tether-driven robots, gel-like soft robots, and fluid-filled tube actuated robots as known in the field of the present disclosure and contemplated hereinafter;

[0040] (7) Examples of flexible endoscopes include, but are not limited to, endoscopes for transesophageal echocardiography (TEE) probes, endoscopes for intracardiac echocardiography (ICE) probes, laparoscopes, and bronchoscopes as known in the field of the present disclosure and contemplated hereinafter;

[0041] (8) The term "prediction model" broadly encompasses all types of models that predict navigation variables for the (optional) continuous positioning control of a portion of an intervention device (e.g., an end effector) as exemplary described in the present disclosure, as known in the art of the present disclosure and contemplated hereinafter. Optionally, these prediction models can be configured using kinematics to output such prediction results based on a positioning data set. Optionally, the prediction model is trained or has been trained in the kinematics of the intervention device. Examples of prediction models include, but are not limited to, artificial neural networks (e.g., feedforward convolutional neural networks, recurrent neural networks, long short-term memory networks, autoencoder networks, generative adversarial networks, and many other deep learning neural networks);

[0042] (9) The term "controller" broadly encompasses all structural configurations of a main circuit board or integrated circuit for controlling the application of various inventive principles of the present disclosure as understood in the art of the present disclosure and exemplary described in the present disclosure. The structural configuration of the controller can include, but is not limited to, (one or more) processors, (one or more) computer-usable / computer-readable storage media, an operating system, (one or more) application modules, (one or more) peripheral device controllers, (one or more) slots, and (one or more) ports, control instructions. The controller can be housed within a workstation or communicatively connected to a workstation;

[0043] (10) The term "application module" broadly encompasses applications contained within or accessible by the controller, the controller including electronic circuitry (e.g., electronic components and / or hardware) and / or executable programs for performing a particular application (e.g., executable software stored on (one or more) non-transitory (transitory) computer-readable media and / or firmware); and

[0044] (11) The terms "signal", "data", and "command" broadly encompass all forms of detectable physical quantities or pulses (e.g., voltage, current, or magnetic field strength) for transmitting information and / or instructions for supporting the application of various inventive principles of the present disclosure as understood in the art of the present disclosure and exemplary described in the present disclosure. The signal / data / command communication of the various components of the present disclosure can involve any communication method known in the art of the present disclosure, including but not limited to signal / data / command transmission / reception via any type of wired or wireless data link and reading of signals / data / commands uploaded to computer-usable / computer-readable storage media.

[0045] Upon reading the following detailed description of various embodiments of the present disclosure in conjunction with the accompanying drawings, the foregoing and other embodiments of the present disclosure, as well as various structures and advantages of the present disclosure, will become more apparent. The detailed description and the drawings are merely illustrative of the present disclosure and not limiting, and the scope of the present disclosure is defined by the claims and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] FIG. 1 illustrates an exemplary embodiment of an interventional device including an end effector as known in the art of the present disclosure.

[0047] Figure 2A Illustrated is an exemplary embodiment of a continuous position controller in accordance with various aspects of the present disclosure.

[0048] Figure 2B Illustrated is an exemplary embodiment of a continuous positioning state machine in accordance with various aspects of the present disclosure.

[0049] FIG. 3A illustrates an exemplary embodiment of a transesophageal echocardiography (TEE) probe as known in the art of the present disclosure.

[0050] FIG. 3B illustrates an exemplary embodiment of the handle of a transesophageal echocardiography (TEE) probe as known in the art of the present disclosure.

[0051] FIGS. 3C - 3F illustrate exemplary movements of a Figure 2A TEE probe as known in the art of the present disclosure.

[0052] FIG. 3G illustrates an exemplary embodiment of a robotically - manipulated transthoracic echocardiography (TTE) probe as known in the art of the present disclosure.

[0053] FIG. 3H illustrates an exemplary embodiment of an optically - shaped sensing continuum robot as known in the art of the present disclosure.

[0054] Figures 4A - 4E Illustrated is a first exemplary embodiment of the forward prediction model and the forward prediction method of the present disclosure.

[0055] Figures 5A - 5E Illustrated is a first exemplary embodiment of the inverse prediction model and the inverse prediction method of the present disclosure.

[0056] Figures 6A - 6E Illustrated is a second exemplary embodiment of the forward prediction model and the forward prediction method of the present disclosure.

[0057] Figures 7A - 7E Illustrated is a second exemplary embodiment of the inverse prediction model and the inverse prediction method of the present disclosure.

[0058] Figures 8A - 8B Illustrates a third exemplary embodiment of the forward prediction model and the forward prediction method of the present disclosure.

[0059] Figures 9A - 9F Illustrates a fourth exemplary embodiment of the forward prediction model and the forward prediction method of the present disclosure.

[0060] Figures 10A - 10B And 12A illustrate a first exemplary embodiment of the image prediction model and the image prediction method of the present disclosure.

[0061] Figures 11A - 11B And 12B illustrate a second exemplary embodiment of the forward prediction model and the forward prediction method of the present disclosure.

[0062] Figures 13A - 13D Illustrates the closed-loop attitude control of the present disclosure.

[0063] Figures 14A - 14D Illustrates the closed-loop vector velocity control of the present disclosure.

[0064] Figures 15 - 18 Illustrates the training data collection system and method of the present disclosure.

[0065] Figure 19 Illustrates an exemplary embodiment of the continuous positioning controller of the present disclosure. Detailed Description

[0066] The present disclosure is applicable to a wide variety of applications that require continuous position control of an end effector. Examples of such applications include, but are not limited to, minimally invasive procedures (e.g., endoscopic hepatectomy, necrosectomy, prostatectomy, etc.), video-assisted thoracic surgery (e.g., lobectomy, etc.), microvascular procedures (e.g., via catheter, sheath, deployment system, etc.), micro medical diagnostic procedures (e.g., intraluminal procedures via endoscope or bronchoscope), deformity correction procedures (e.g., via Kirschner wire, screwdriver, etc.), and non-medical applications.

[0067] The present disclosure improves the continuous position control of the end effector during such applications by providing a prediction of the attitude of the end effector and a positioning of the movement of an intervention device that can be used to control and / or verify the manual or automatic navigation positioning of the end effector.

[0068] To facilitate understanding of the present disclosure, the following description of FIG. 1 teaches exemplary embodiments of an interventional device including an end effector as known in the art of the present disclosure. Based on the description of FIG. 1, a person of ordinary skill in the art of the present disclosure will recognize how to apply the present disclosure to make and use additional embodiments of an interventional device including an end effector as known in the art of the present disclosure and contemplated hereinafter.

[0069] Referring to FIG. 1, in practice, the interventional device 30 supports manual navigation positioning or automatic navigation positioning of the end effector 40 during application, as represented by the arrows extending from the end effector 40. Examples of the interventional device 30 include, but are not limited to, continuum flexible robots, flexible endoscopes, and guidewires, and examples of the end effector 40 include, but are not limited to, intraoperative imaging devices, interventional tools / surgical instruments, and surgical sutures.

[0070] In operation, one or more navigation commands 31, one or more actuation signals 32, and / or one or more navigation forces 33 are transmitted to / imposed on the interventional device 30, whereby the interventional device 30 translates, rotates, and / or pivots in accordance with the one or more navigation commands 31, the one or more actuation signals 32, and / or the one or more navigation forces 33 to navigate the end effector 40 to a target pose (i.e., a position and orientation in the application space).

[0071] For example, FIGS. 3A and 3B illustrate an embodiment of a transesophageal echocardiography (TEE) probe 130 as an interventional device 30 having an imaging end effector 140 that can be inserted through a patient P's mouth and into the esophagus to capture an image of the patient P's heart, and a practitioner (not shown) or a robot controller 100 ( Figure 2B ) operates the handle 132 of the TEE probe 130 to reposition the TEE probe 130 within the patient P to navigate the imaging end effector 140 to a target pose.

[0072] More specifically, the TEE probe 130 includes a flexible elongate member 131, a handle 132, and an imaging end effector 140. The flexible elongate member 131 is structurally arranged and / or otherwise configured in terms of size and / or shape to be positioned within a body lumen of a patient (e.g., the esophagus). The imaging end effector 140 is mounted at the distal end of the member 131 and includes one or more ultrasound transducer elements, whereby the imaging end effector 140 is configured to transmit ultrasound energy towards an anatomy of the patient P (e.g., the heart). The ultrasound energy is reflected by the patient's vasculature and / or tissue structure, whereby the ultrasound transducer elements in the imaging end effector 140 receive the reflected ultrasound echo signals. In some embodiments, the TEE probe 130 may include internal or integrated processing components that are capable of locally processing the ultrasound echo signals to generate image signals representative of the anatomy of the patient P being imaged. In practice, the (one or more) ultrasound transducer elements may be arranged to provide a two-dimensional (2D) image or a three-dimensional (3D) image of the anatomy of the patient P. As described in more detail herein, the images acquired by the TEE probe 130 depend on the insertion depth, rotation, and / or tilt of the imaging end effector 140.

[0073] The handle 132 is coupled to the proximal end of the member 131. The handle 132 includes control elements for navigating the imaging end effector 140 to a target pose. As shown, the handle 132 includes knobs 133 and 134 and a switch 135. The knob 133 causes the member 131 and the imaging end effector 140 to bend along the anterior-posterior plane of the patient P (e.g., the heart). The knob 134 causes the member 131 and the imaging end effector 140 to bend along the left-right plane of the patient P. The switch 135 controls beamforming (e.g., adjusting the angle of the imaging plane) performed at the imaging end effector 140.

[0074] In a manual navigation embodiment, a practitioner manually toggles the knobs 133 and 134 and / or manually opens and / or closes the switch 135 as needed to navigate the imaging end effector 140 to a target pose. The practitioner may receive a display of the image generated by the imaging end effector 140, thereby applying a navigation force 33 (FIG. 1) to control the knobs 133 and 134 and / or the switch 135 on the handle 132.

[0075] In an automated navigation embodiment, a robotic system (not shown) can include electrical and / or mechanical components (e.g., motors, rollers, and gears) configured to turn knobs 133 and 134 and / or open and / or close switch 135, whereby robotic controller 100 can receive motion control commands 31 (FIG. 1) from a navigation controller (not shown) or an input device (not shown) to control knobs 133 and 134 and / or switch 135 on handle 132. Alternatively, robotic controller 100 can be configured to directly manipulate the TEE probe 130 via actuation signal 32 (FIG. 1) based on a guidance method implemented by robotic controller 100.

[0076] The TEE probe 130 can be manipulated in various degrees of freedom. FIGS. 3C - 3F illustrate various mechanisms for manipulating the TEE probe 130.

[0077] FIG. 3C is a schematic illustration showing that the TEE probe 130 can be manually advanced into a patient's esophagus as shown by arrow 131b or withdrawn from the patient's esophagus as shown by arrow 131c. The TEE probe 130 can be rotated manually or automatically to the left (e.g., counterclockwise) or to the right (e.g., clockwise) relative to the longitudinal axis 131a of the TEE probe 130 as shown by arrows 139a and 139b, respectively. The rotation of member 131 can be described by a parameter denoted as γ.

[0078] FIG. 3D is a schematic illustration showing that the TEE probe 130 can be electronically rotated from 0 degrees to 180 degrees (e.g., for beamforming) as shown by arrows 136a and 136b, e.g., by manual or robotic control of switch 135 on handle 132. The rotation of the imaging plane can be described by a parameter denoted as ω.

[0079] FIG. 3E is a schematic illustration showing that the TEE probe 130 can be bent along, e.g., an anterior - posterior plane relative to a patient's heart as shown by arrows 137a and 137b, e.g., by manually or robotically turning knob 134 on handle 132. The bending along the anterior - posterior plane can be described by a parameter denoted as α.

[0080] FIG. 3F is a schematic illustration showing that the TEE probe 130 can be bent along, e.g., a left - right plane relative to a patient's heart as shown by arrows 138a and 138b, e.g., by manually or robotically turning knob 133 on handle 132. The bending along the left - right plane can be described by a parameter denoted as β.

[0081] As another example of an exemplary embodiment of the access device 30 (FIG. 1), FIG. 3G shows a transthoracic echocardiography (TTE) probe 240 configured to capture an ultrasound image of the anatomy of a patient P from outside the body of the patient P and a robot 230 shown for manually or robotically manipulating the TTE probe 240 to reposition the TTE probe 240 to a target pose (i.e., the positioning and / or orientation of the TTE probe 240 relative to the patient P). More specifically, the robot 230 includes a plurality of links 231 coupled to a plurality of joints 232, the plurality of joints 232 being configured to hold the TTE probe 240 and manipulate the TTE probe 240 on the outer surface of the patient P (e.g., around the chest region for imaging the heart).

[0082] In a manual navigation embodiment, a practitioner manually applies a navigation force 33 (FIG. 1) to the link(s) 231, thereby translating, rotating, and / or pivoting the link(s) 231 of the robot 230 to navigate the imaging TTE probe 240 to a target pose. The practitioner may receive a display of the image generated by the TTE probe 240 for use as a basis for controlling the link(s) 231 of the robot 230.

[0083] In an automatic navigation embodiment, a robot system (not shown) includes electrical and / or mechanical components (e.g., motors, rollers, and gears) configured to control the link(s) 231 of the guiding robot 230, whereby the robot controller 101 receives motion control commands 32 (FIG. 1) in the form of Cartesian velocity parameters or joint velocity parameters from a navigation controller (not shown) or an input device (not shown) to control the link(s) 231 of the guiding robot 230. Alternatively, the robot controller 101 may be configured to directly manipulate the TTE probe 240 based on a guiding method implemented by the robot controller 101 via an actuation signal 32 (FIG. 1).

[0084] As a further example of an exemplary embodiment of the access device 30 (FIG. 1), FIG. 3H shows a shape-sensing guidewire 332 embedded or attached to a continuum robot 331 having an end effector 340. The shape-sensing guidewire 232 incorporates optical shape sensing (OSS) technology known in the field of the present disclosure. More specifically, the shape-sensing controller 103 uses light in the multi-core optical fiber 333 along the guidewire 332 for device positioning and navigation during the surgery. The principle involved utilizes distributed strain measurements in the optical fiber using characteristic Rayleigh backscattering or controlled grating patterns (e.g., fiber Bragg gratings). In practice, when the robot controller 102 or a practitioner (not shown) navigates the continuum robot 331 within the patient P to position the end effector 340 in a target pose, the shape-sensing controller 103 acquires the registered shape of the continuum robot 331 via the shape-sensing guidewire 332.

[0085] To further facilitate understanding of the present disclosure, the following descriptions of Figure 2A and Figure 2B teach exemplary embodiments of the continuous position controller of the present disclosure and the continuous positioning state machine of the present disclosure, respectively. According to Figure 2A and Figure 2B 's descriptions, those of ordinary skill in the art of the present disclosure will realize how to apply the present disclosure to make and use additional embodiments of the continuous position controller of the present disclosure and the continuous positioning state machine of the present disclosure.

[0086] Additionally, the TEE probe 130 (FIG. 3A), the robot 230 / TTE probe 240 (FIG. 3G), and the continuum robot 331 (FIG. 3H) are used herein as non-limiting examples of the access device 30 including the end effector 40 (FIG. 1) to support the description of various embodiments of the continuous position controller of the present disclosure. Nevertheless, those of ordinary skill in the art of the present disclosure will realize how to apply the present disclosure to a wide variety of additional embodiments of the access device 30 including the end effector 40.

[0087] Referring to FIG. 1, Figure 2A and Figure 2B , the access device 30 of the present disclosure including the end effector 40 and the continuous positioning controller 50 represents a continuous positioning state machine 90.

[0088] Specifically, as Figure 2BAs shown, the state S92 of the continuous positioning state machine 90 includes navigation of the intervention device 30 including the end effector 40 according to a particular application (e.g., a minimally invasive procedure, a video-assisted thoracic surgery, a microvascular procedure, a micro medical diagnosis procedure, or a deformity correction procedure). In practice, the application may involve manual navigation positioning or automatic navigation positioning of the end effector 40 to a target pose, where the application incorporates imaging guidance (e.g., image segmentation, image registration, path planning, etc.), intervention device tracking (e.g., electromagnetic, optical, or shape sensing), and / or any other navigation technique for positioning the end effector 40 to the target pose.

[0089] Execution of the state S92 of the continuous positioning state machine 90 causes generation of navigation data 34 and optional generation of assistance data 35. Generally, in practice, the navigation data 34 will be transmitted to / imposed on the intervention device 30 in the form of one or more navigation commands 31, one or more actuation signals 32, and / or one or more navigation forces 33, and the assistance data 35 will be in the form of one or more images of the intervention device 30 and / or the end effector 40, operating characteristics of the intervention device 30 (e.g., shape, strain, distortion, temperature, etc.), and operating characteristics of the end effector 40 (e.g., pose, force, etc.).

[0090] In response thereto, the state S94 of the continuous positioning state machine 90 includes continuous positioning control by the continuous position controller 50 that navigates the intervention tool 30 and the end effector 40 according to the state S92. To this end, the continuous position controller 50 uses the forward prediction model 60 of the present disclosure, the inverse prediction model 70 of the present disclosure, and / or the imaging prediction model 80 of the present disclosure.

[0091] In practice, as will be further explained in the present disclosure, the forward prediction model 60 can be any type of machine learning model or equivalent (e.g., a neural network) suitable for regressing the positioning movement of the intervention device 30 to the navigation pose of the end effector 40 for a particular type of intervention device 30 used in the particular type of application being implemented, where the forward prediction model 60 is trained on the forward kinematics of the intervention device 30 predicting the navigation pose of the end effector 40.

[0092] In operation, the forward prediction model 60 inputs navigation data 34 (and auxiliary data 35, if transmitted) associated with manual or automatic navigation of the intervention device 30, thereby predicting the navigation pose of the end effector 40 corresponding to the navigation of the intervention device 30 and outputting continuous positioning data 51 regarding information on positioning the end effector 40 to a target pose by the intervention device 30 based on the predicted navigation pose of the end effector 40. The continuous positioning data 51 can be used in state S92 as a control operation to determine accuracy and / or perform recalibration of the manual or automatic navigation of the intervention device 30 to position the end effector 40 to the target pose.

[0093] In practice, as will be further explained in this disclosure, the inverse prediction model 70 can be any type of machine learning model or equivalent (e.g., a neural network) suitable for regressing the target pose of the end effector 40 to the positioning movement of the intervention device 30 for a particular type of intervention device 30 used in a particular type of application being implemented, whereby the inverse prediction model 60 is trained in the inverse kinematics of the intervention device 30 that predicts the positioning movement of the intervention device 30.

[0094] In operation, the inverse prediction model 70 inputs navigation data 34 (and auxiliary data 35, if transmitted) associated with the target pose of the end effector 40, thereby predicting the positioning movement of the intervention device 30 for positioning the end effector 40 to the target pose, and the inverse prediction model 70 outputs continuous positioning commands 52 for positioning the end effector 40 to the target pose by the intervention device 30 based on the predicted positioning movement of the intervention device 30. The continuous positioning commands 52 can be used in state S92 as a control operation to perform manual or automatic navigation of the intervention device 30 to position the end effector 40 to the target pose.

[0095] In practice, as will be further explained in this disclosure, the imaging prediction model 80 can be any type of machine learning model or equivalent (e.g., a neural network or a scale-invariant feature transform network) suitable for regressing the relative imaging performed by the end effector 40 to the navigation pose of the end effector 40 for a particular type of end effector 40 used in a particular type of application being implemented, whereby the inverse prediction model 60 is trained in the correlation between the relative imaging performed by the end effector 40 and the forward kinematics of the intervention device 30 that predicts the navigation pose of the end effector 40.

[0096] In operation, the imaging prediction model 60 inputs the auxiliary data 35 in the form of an image generated by the end effector 40 in one or more poses, so as to predict the navigation pose of the end effector 40 as feedback data regarding the information that the intervention device 30 corrects and positions the end effector 40 to the target pose. The feedback data is used in the closed loop of state S94 to generate the difference between the target pose of the end effector 40 and the predicted navigation pose of the end effector 40. Thus, the inverse prediction model 70 can input this difference to predict the corrective positioning movement of the intervention device 30 to reposition the end effector 30 to the target pose.

[0097] In practice, embodiments of the continuous positioning controller 50 can use the forward prediction model 60, the inverse prediction model 70, and / or the imaging prediction model 80.

[0098] For example, an embodiment of the continuous positioning controller 50 can use only the forward prediction model 60 to facilitate the display of the accuracy of manual or automatic navigation of the intervention device 30 to position the end effector 40 to the target pose.

[0099] More particularly, a user interface is provided to display an image of the attempted navigation of the end effector 40 to the target pose and an image of the predicted navigation pose of the end effector 40 obtained by the forward prediction model 60. The predicted confidence ratio is shown to the user. To evaluate the prediction uncertainty, as known in the field of the present disclosure, multiple feed-forward iterations of the forward prediction model 60 are performed with dropout randomly enabled.

[0100] As another example, an embodiment of the continuous positioning controller 50 can use only the inverse prediction model 70 to command the manual or automatic navigation of the intervention device 30 to position the end effector 40 to the target pose.

[0101] As another example, an embodiment of the continuous positioning controller 50 can use only the imaging prediction model 80 to provide feedback data giving the information of the following operation, that is, the intervention device 30 corrects and positions the end effector 40 to the target pose.

[0102] As another example, an embodiment of the continuous positioning controller 50 can use the forward prediction model 60 and the inverse prediction model 70 to command the manual or automatic navigation of the intervention device 30 to position the end effector 40 to the target pose and display the accuracy of the manual or automatic navigation of the intervention device 30 to position the end effector 40 to the target pose.

[0103] As another example, embodiments of the continuous positioning controller 50 may use an inverse prediction model 70 and an imaging prediction model 80 to command manual or automatic navigation of the intervention device 30 to position the end effector 40 at a target pose and provide feedback data giving information on the following operation: correcting the positioning of the end effector 40 to the target pose by the intervention device 30.

[0104] To further facilitate understanding of the present disclosure, the following description of FIGS. 4-14 teaches exemplary embodiments of the forward prediction model, inverse prediction model, and imaging prediction model of the present disclosure. Based on the description of FIGS. 4-14, those of ordinary skill in the art of the present disclosure will recognize how to apply the present disclosure to make and use additional embodiments of the forward prediction model, inverse prediction model, and imaging prediction model of the present disclosure.

[0105] Figures 4A - 4E Illustrated is the training and application of the forward prediction model 60a of the present disclosure, which is trained on the forward kinematics of the intervention device 30 (FIG. 1) predicting the navigation pose of the end effector 40 (FIG. 1), thereby facilitating the application of the forward prediction model 60a to the commanded positioning movement of the intervention device 30 during the intervention process to present the predicted navigation pose of the end effector 40, whereby the continuous positioning controller 50 (FIG. 1) can generate continuous positioning data 51 regarding the information of positioning the end effector 40 to the target pose by the intervention device 30 based on the predicted navigation pose of the end effector 40.

[0106] More particularly, referring to Figure 4A , the training phase of the forward prediction model 60a involves a training controller (not shown) configured to interpret the ground truth training dataset D as exemplarily taught in the subsequent description of FIGS. 16-18 of the present disclosure. The dataset D consists of n sequences W of i data points represented by the 2-tuple d i =(T i ,Q i ). The 2-tuple consists of the end effector pose (T∈

[0107] SE(3))62a and a sequence (Q∈(q t ,q t+1 …q t+j ))61a of j successive joint variables collected at sequential time points starting from t. The entry q t represents all joint variables controlled by a robot controller (not shown).

[0108] In practice, the training dataset D is a collection of expert data that reasonably covers different navigations of the intervention device 30. To this end, the diverse dataset training dataset D for learning should include manufacturing differences between various types of robots, performance characteristics, wear of hardware components, and other system-related and system-unrelated factors (e.g., the temperature or humidity of the environment in which the robot operates).

[0109] Reference Figure 4B , the application phase of the forward prediction model 60a involves the continuous positioning controller 50 using the feedforward prediction model 60a to execute a deep learning type algorithm to regress the commanded position motion of the intervention device 30 to the navigation pose of the end effector 40. The forward prediction model 60a is configured to infer the pose of the end effector 40 ( ) 62b given a sequence Q of j successive joint variables 61b of the intervention device 30 (e.g., the parameters α, β as shown in FIG. 3B).

[0110] In one embodiment as shown in Figure 4E , the forward prediction model 60a uses a neural network library 160a that includes an input layer, a hidden layer, and an output layer, which are derived from a combination of one or more fully connected layers (FCLs) 163a, one or more convolutional layers (CNLs) 164a, one or more recurrent layers (RCLs) 165a, and one or more long short-term memory (LSTM) layers 166a.

[0111] In practice, the combination of these layers is configured to implement the regression of the joint variable Q to the pose .

[0112] In one embodiment for implementing the regression of the joint variable Q to the pose , the neural network library 160a includes a set of N fully connected layers 163a.

[0113] In a second embodiment for implementing the regression of the joint variable Q to the pose , the neural network library 160a includes a set of N convolutional layers 164a, followed by a set of M fully connected layers 163a or a set of W recurrent layers 165a or a set of W long short-term memory layers 166a.

[0114] In a third embodiment for implementing the regression of the joint variable Q to the pose , the neural network library 160a includes a set of N convolutional layers 164a, followed by a combination of a set of M fully connected layers 163a and a set of W recurrent layers 165a or a set of W long short-term memory layers 166a.

[0115] In practice, the fully connected layer 163a can include K neurons, where N, M, W, and K can be any positive integers, and the values can vary according to the embodiments. For example, N can be approximately 8, M can be approximately 2, W can be approximately 2, and K can be approximately 1000. Moreover, the convolutional layer 164a can perform a non-linear transformation, which can be a composite function of operations (such as batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 164a can also include a non-linear function (such as including the rectified non-linear ReLU operation) configured to extract the corrected feature map.

[0116] In addition, in practice, one of the layers 163a or 164a serves as the input layer for the sequence 161a of input joint variables Q. Thus, the size of the sequence of joint variables Q can be ≥1, and one of the layers 163a, 165a, and 166a can serve as the output layer for the pose 162a of the end effector 40 in the Cartesian space (e.g., the translation and rotation of the end effector 40 in the Cartesian space). The output pose of the end effector 40 in the Cartesian space can be represented as a vector parameterization and / or non-vector parameterization of the position and orientation of a rigid body. More particularly, the parameterization can be in the form of Euler angles, quaternions, matrices, exponential maps, and / or angle axes representing rotation and / or translation (e.g., including the direction and magnitude of translation).

[0117] Moreover, in practice, the output layer can be the non-linear fully connected layer 163a, which gradually shrinks the high-dimensional output of the last convolutional layer 164a of the neural network library 160a to produce a set of output variables.

[0118] During training, by comparing the output inferred by the forward prediction model (given the input sequence Q) i with the ground truth end effector pose T

[0119] from a batch of training data set D 62a.

[0120] Still referring to Figure 4E, depending on the complexity of the task, the neural architecture of the exemplary embodiment of the forward prediction model 60a has a unique number of layers, which is specified in the output layer 169a for predicting the pose of the end effector 40 given a sequence Q of j successive joint variables 61a. The loss function for training such a neural architecture can be defined as the sum of the Euclidean distances between the translational component and the rotational component, for example, in.

[0121] Reference Figure 4C and Figure 4D , stage S92a of the exemplary intervention process 90a uses the TEE probe 130 (FIG. 3A) and the robot controller 100 (FIG. 3B), and includes the robot controller 100 receiving the position of the TEE probe 130 as a joint variable and sending a movement signal to the handle 132 (FIG. 3A) of the TEE probe 130. By using the electric knob, the handle 132 tightens / relaxes the tether of the TEE probe 130, which will cause the end effector 140 (FIGS. 3C-3F) to move to the target pose. In practice, the position of the TEE probe 130 as a joint variable can be indicated by the user or an external tracking device or guidance system.

[0122] Stage S94a of process 90a involves the robot controller 100, the forward prediction model 50a, and the display controller 104. The robot controller 100 stores the sequence (Q) of successive joint variables 61a and transmits it to the forward prediction model 60a that predicts the navigation pose of the end effector 140. The continuous positioning controller 50a generates a prediction confidence ratio derived from the uncertainty of multiple feed-forward iterations of the forward prediction model 60a, and such multiple feed-forward iterations of the forward prediction model 60a are performed with randomly enabled dropout as is known in the art of the present disclosure. The forward prediction model 50a transmits the continuous positioning data 51a including the predicted navigation pose of the end effector 140 and the confidence ratio to the display controller 104, and the display controller 104 in turn controls the display of the image 105a of the navigation pose of the end effector 140, the image 106a of the navigation pose of the end effector 140, and the confidence ratio for the purpose of guiding the end effector 140 to the target pose.

[0123] Figures 5A - 5EIllustrated is the training and application of the inverse prediction model 70a of the present disclosure, which is trained on the inverse kinematics of the intervention device 30 (FIG. 1) that predicts the positioning movement of the intervention device 30, thereby facilitating the application of the inverse prediction model 70a to the target pose (T) 71a of the end effector 40 during the intervention process to present the joint variable movement (q) 72 of the intervention device 30. t , whereby the continuous positioning controller 50 (FIG. 1) can generate a continuous positioning command for repositioning the end effector 40 to the target pose by the intervention device 30 based on the predicted positioning movement of the intervention device 40.

[0124] More particularly, referring to Figure 5A , the training phase of the inverse prediction model 70a involves a training controller (not shown) configured to interpret the ground truth training data set D as exemplarily taught in the description of Figures 17 - 18 . The data set D consists of n sequences W containing i data points represented by the 2-tuple d i = (T i , Q i ). The 2-tuple consists of the end effector pose (T ∈ SE(3)) 71a and a sequence (Q ∈ (q t , q t+1 …q t+j )) 72a of j successive joint variables collected at sequential time points starting from t to t + j. The entry q t represents all joint variables controlled by a robot controller (not shown). The variable j can also be equal to 1, which means that the prediction model will be trained to infer a single set of joint variables.

[0125] In practice, the training data set D is a collection of expert data that reasonably covers different navigations of the intervention device 30. To this end, the diverse data set D for learning should include mechanical differences between various types of robots, wear of hardware components, and other system-related factors.

[0126] Referring to Figure 5B , the application phase of the inverse prediction model 70a involves the continuous positioning controller 50 using the inverse prediction model 70a to perform a deep learning type algorithm for motion regression. The inverse prediction model 70a is configured to infer a sequence of j successive joint variables 72 t (e.g., the parameters α, β as shown in FIG. 3B) to reach the pose (T ∈ SE(3)) 71t of the end effector 40.

[0127] In as Figure 5EIn one illustrated embodiment, the inverse prediction model 70a uses a neural network library 170a that includes an input layer, a hidden layer, and an output layer, which are derived from a combination of one or more fully connected layers (FCLs) 173a, one or more convolutional layers (CNLs) 174a, one or more recurrent layers (RCLs) 175a, and one or more long short-term memory (LSTM) layers 176a.

[0128] In practice, the combination of these layers is configured to perform a regression of pose T to joint variables .

[0129] In one embodiment for performing a regression of pose T to joint variables , the neural network library 170a includes a set of N fully connected layers 173a.

[0130] In a second embodiment for performing a regression of pose T to joint variables , the neural network library 170a includes a set of N convolutional layers 174a, followed by a set of M fully connected layers 173a or a set of W recurrent layers 175a or a set of W long short-term memory layers 176a.

[0131] In a third embodiment for performing a regression of pose T to joint variables , the neural network library 170a includes a set of N convolutional layers 174a, followed by a combination of a set of M fully connected layers 173a and a set of W recurrent layers 175a or a set of W long short-term memory layers 176a.

[0132] In practice, the fully connected layer 173a can include K neurons, where N, M, W, and K can be any positive integers, and the values can vary according to the embodiment. For example, N can be approximately 8, M can be approximately 2, W can be approximately 2, and K can be approximately 1000. Moreover, the convolutional layer 174a can perform a non-linear transformation, which can be a composite function of operations (such as batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 174a can also include a non-linear function (such as including a rectified non-linear ReLU operation) configured to extract a modified feature map.

[0133] Additionally, in practice, one of layers 173a or 174a serves as an input layer for the pose 171a of the input end effector 40 in Cartesian space (e.g., translation and rotation of the end effector 40 in Cartesian space), and one of layers 173a, 175a, and 176a can serve as an output layer for the sequence 172a of output joint variables Q, whereby the size of the sequence of joint variables Q can be ≥1. The input pose of the end effector 40 in Cartesian space can be represented as a vector parameterization and / or non-vector parameterization of the position and orientation of a rigid body. More particularly, the parameterization can be in the form of Euler angles, quaternions, matrices, exponential maps, and / or angle axes representing rotation and / or translation (e.g., including the direction and magnitude of translation).

[0134] Moreover, in practice, the output layer can be a non-linear fully connected layer 173a that progressively shrinks the high-dimensional output of the last convolutional layer 174a of the neural network library 170a to produce a set of output variables.

[0135] During training, by comparing the output from the inverse prediction model (with the given ground truth end effector pose T as input) i with the ground truth sequence Q from a batch of training dataset D

[0136] (which can be systematically selected or randomly selected from a data memory (not shown)), the training weights of the inverse prediction model 70a are continuously updated. More particularly, the coefficients of the filter can be initialized with predefined values or arbitrary values. The coefficients of the filter are applied to this batch of training dataset D via forward propagation and are adjusted via backpropagation to minimize any output error. to achieve the provided pose (T∈SE(3)) 71b of the end effector 40.

[0137] Still referring to Figure 5E , depending on the complexity of the task, the neural architecture of the exemplary embodiment of the inverse prediction model 70a as shown has a unique number of layers, which is specified in the output layer 179a defined to predict a sequence of j successive joint variables 72b (e.g., parameters α, β as shown in FIG. 3B) to achieve the pose (T∈SE(3)) 71b of the end effector 40. The mean squared error (MSE) can be used as the loss function.

[0138] Referring to Figure 5C and Figure 5D, stage S92b of the exemplary intervention process 90b uses the TEE probe 130 (Figure 3A) and the navigation controller 103, and includes the navigation controller 103 determining the target pose of the end effector 140 and sending the target pose 71a to the inverse prediction model 70a. In practice, the navigation controller 130 may implement any known guidance algorithm known in the field of the present disclosure.

[0139] Stage S94A of process 90b involves the inverse prediction model 70a and the robot controller 100. The inverse prediction model 70a infers a sequence of j successive joint variables 72a (e.g., parameters α, β as shown in Figure 3B) to reach the pose (T ∈ SE(3)) 71a of the end effector 40, and the continuous positioning controller 50b communicates with the command controller 104 to control the intervention device 30 via the end effector 40 to position the end effector 50 to the target pose based on the predicted positioning movement of the intervention device 40.

[0140] Figures 6A - 6E Illustrated is the training and application of the forward prediction model 60b of the present disclosure, which is trained on the forward kinematics of the intervention device 30 (Figure 1) predicting the navigation pose of the end effector 40 (Figure 1), thereby facilitating the application of the forward prediction model 60a to the command n vector of the joint velocities of the intervention device 30 during the intervention process to present the predicted linear velocity and / or angular velocity of the end effector 40, whereby the continuous positioning controller 50 (Figure 1) can generate continuous positioning data 51 regarding the information of repositioning the end effector 40 to the target pose by the intervention device 30 based on the predicted linear velocity and / or angular velocity of the end effector 40.

[0141] More particularly, referring to Figure 6A , the training stage of the forward prediction model 60b involves a training controller (not shown) configured to interpret the ground truth training dataset D as exemplarily taught in Figures 16 - 18. The dataset D consists of n sequences W containing i data points represented by the 2 - tuple The 2 - tuple consists of a sequence of j successive joint velocity commands 61b and the linear velocity and / or angular velocity 62b of the end effector The entry represents all joint variables controlled by a robot controller (not shown). The sequence may also contain only one entry.

[0142] In practice, the training dataset D is a collection of expert data that reasonably covers different navigations of the intervention device 30. To this end, the diverse dataset training dataset D for learning should include mechanical differences between various types of robots, wear of hardware components, and other system - related factors.

[0143] Reference Figure 6B , the application phase of the forward prediction model 60b involves the continuous positioning controller 50 using the feed - forward prediction model 60b to execute a deep - learning - type algorithm for end - effector motion regression. The forward prediction model 60b is configured to infer the linear velocity and / or angular velocity 62b of the end - effector 40 given an n - vector of the joint velocities 61b of the intervening device 30.

[0144] In one embodiment as Figure 6E shown, the forward prediction model 60b uses a neural network library 160b, which includes an input layer, a hidden layer, and an output layer, and these layers are derived from a combination of one or more fully - connected layers (FCLs) 163b, one or more convolutional layers (CNLs) 164b, one or more recurrent layers (RCLs) 165b, and one or more long short - term memory (LSTM) layers 166b.

[0145] In practice, the combination of these layers is configured to implement the regression of the joint velocities of the intervening device 30 to the linear velocity and / or angular velocity of the end - effector 40.

[0146] In one embodiment for implementing the regression of the joint velocities of the intervening device 30 to the linear velocity and / or angular velocity of the end - effector 40, the neural network library 160b includes a set of N fully - connected layers 163b.

[0147] In a second embodiment for implementing the regression of the joint velocities of the intervening device 30 to the linear velocity and / or angular velocity of the end - effector 40, the neural network library 160b includes a set of N convolutional layers 164b, followed by a set of M fully - connected layers 163b or a set of W recurrent layers 165b or a set of W long short - term memory layers 166b.

[0148] In a third embodiment for implementing the regression of the joint velocities of the intervening device 30 to the linear velocity and / or angular velocity of the end - effector 40, the neural network library 160b includes a set of N convolutional layers 164b, followed by a combination of a set of M fully - connected layers 163b and a set of W recurrent layers 165b or a set of W long short - term memory layers 166b.

[0149] In practice, the fully connected layer 163b may include K neurons, where N, M, W, and K can be any positive integers, and the values can vary according to the embodiments. For example, N can be approximately 8, M can be approximately 2, W can be approximately 2, and K can be approximately 1000. Moreover, the convolutional layer 164b can perform a non-linear transformation, which can be a composite function of operations (such as batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 164b can also include a non-linear function (such as including the rectified non-linear ReLU operation) configured to extract the modified feature map.

[0150] Additionally, in practice, one of the layers 163b or 164b serves as an input layer for a sequence of j successive joint velocities such that the size of the sequence of j successive joint velocities can be ≥1, and one of the layers 163b, 165b, and 166b can serve as an output layer that outputs the linear velocity and angular velocity of the end effector regressed according to the last fully connected layer (e.g., 6 units, 3 units for linear velocity, and 3 units for angular velocity) using a linear or non-linear activation function.

[0151] During training, the predicted linear velocity and angular velocity obtained via the forward velocity prediction model (in the case of a given sequence of joint velocities) are compared with the linear velocity and / or angular velocity 62b from a batch of training data set D (which can be systematically selected or randomly selected from a data memory (not shown)) to continuously update the training weights of the forward prediction model 60b. More specifically, the coefficients of the filter can be initialized with predefined values or arbitrary values. The coefficients of the filter are applied to this batch of training data set D via forward propagation and are adjusted via backpropagation to minimize any output error.

[0152] In an application, the forward prediction model 60b infers the linear velocity and / or angular velocity 62b of the end effector 40 given a sequence of joint velocities 61b of the intervening device 30.

[0153] Still referring to Figure 6E , the neural architecture consists of an input, the previously described neural network library, and an output. The input is a sequence of joint velocities, and the output is the linear velocity and angular velocity that can be regressed according to a fully connected layer with 6 units. The loss function can be the MSE listed below:

[0154] Referring to Figure 6C and Figure 6D, stage S92c of the exemplary interventional procedure 90c uses the robot 230 (FIG. 3A), the TTE probe 240 (FIG. 3A), and the robot controller 101 (FIG. 3G), and includes the robot controller 101 receiving the position of the TTE probe 240 as an n-vector of the joint velocities 61b of the interventional device 30 and sending the n-vector of the joint velocities 61b of the interventional device 30 to the robot 230. In practice, the position of the TTE probe 240 as the n-vector of the joint velocities 61b of the interventional device 30 may be indicated by a user or an external tracking device or guidance system.

[0155] Stage S94c of process 90c involves the robotic controller 101, the forward prediction model 50a, and the display controller 104. The robotic controller 101 stores the n-vector of the joint velocities 61b of the interventional device 30 and transmits it to the forward prediction model 60b that predicts the linear velocity and / or angular velocity 62b of the TTE probe 240. The continuous positioning controller 50c generates a confidence ratio of the prediction derived from the uncertainty of multiple feed-forward iterations of the forward prediction model 60b, which are performed with randomly enabled dropout as is known in the art of the present disclosure. The forward prediction model 60b transmits continuous positioning data 51b to the display controller 104, the continuous positioning data 51b including the predicted navigation pose of the TTE probe 240 derived from the predicted linear velocity and / or angular velocity 62b of the TTE probe 240. And also including the confidence ratio, the display controller 104 then controls the display of the image 105 a of the navigation posture of the TTE probe 240 , the image 106 a of the navigation posture of the TTE probe 240 , and the confidence ratio for the guidance purpose of guiding the end effector 240 to the target posture.

[0156] Figures 7A - 7E The diagram illustrates the training and application of the inverse prediction model 70b of the present disclosure, which is trained on the inverse kinematics of the interventional device 30 (Figure 1) that predicts the positioning motion of the interventional device 30, thereby facilitating the application of the inverse prediction model 70b to the target posture of the end effector 40 during the interventional process to present the predicted positioning motion of the interventional device 30, whereby the continuous positioning controller 50 (Figure 1) can generate continuous positioning commands that control the interventional device 30 to reposition the end effector 40 to the target posture based on the predicted positioning motion of the interventional device 40.

[0157] More specifically, refer to Figure 7A The training phase of the inverse prediction model 70b involves a training controller (not shown) being configured as in Figures 17 - 18 The actual training data set D is interpreted as exemplarily taught in the description of FIG. The data set D consists of two tuples Composed of n sequences W of i data points represented. The 2-tuple consists of the linear and / or angular velocity of the end effector and the successive joint variables collected at sequential time points from t to t + j consists of sequences of. Entries represent all joint variables controlled by a robot controller (not shown).

[0158] In practice, the training dataset D is a collection of expert data that reasonably covers different navigations of the intervention device 30. To this end, the diverse dataset training dataset D for learning should include mechanical differences between various types of robots, wear of hardware components, and other system-related factors.

[0159] Reference Figure 7B , the application phase of the inverse prediction model 70b involves the continuous positioning controller 50 using the inverse prediction model 70b to execute a deep learning type algorithm for joint velocity regression. The inverse prediction model 70b is configured to infer an n-vector of the joint velocity 61b of the intervention device 30 given the linear and / or angular velocity 62b of the end effector 40.

[0160] In one embodiment as Figure 7E shown, the inverse prediction model 70b uses a neural network library 170b, which includes an input layer, a hidden layer, and an output layer, and these layers are derived from a combination of one or more fully connected layers (FCLs) 173b, one or more convolutional layers (CNLs) 174b, one or more recurrent layers (RCLs) 175b, and one or more long short-term memory (LSTM) layers 176b.

[0161] In practice, the combination of these layers is configured to implement the regression of the linear and / or angular velocity of the end effector 40 to the joint velocity of the intervention device 30.

[0162] In one embodiment for implementing the regression of the linear and / or angular velocity of the end effector 40 to the joint velocity of the intervention device 30, the neural network library 170b includes a set of N fully connected layers 173b.

[0163] In a second embodiment for implementing the regression of the linear and / or angular velocity of the end effector 40 to the joint velocity of the intervention device 30, the neural network library 170b includes a set of N convolutional layers 174b, followed by a set of M fully connected layers 173b or a set of W recurrent layers 175b or a set of W long short-term memory layers 176b.

[0164] In a third embodiment for implementing the regression of the linear velocity and / or angular velocity of the end effector 40 to the joint velocities of the intervention device 30, the neural network library 170b includes a set of N convolutional layers 174b, followed by a combination of a set of M fully connected layers 173b and a set of W recurrent layers 175b or a set of W long short-term memory layers 176b.

[0165] In practice, the fully connected layer 173b may include K neurons, where N, M, W, and K can be any positive integers, and the values may vary according to the embodiment. For example, N can be approximately 8, M can be approximately 2, W can be approximately 2, and K can be approximately 1000. Moreover, the convolutional layer 174b can implement a non-linear transformation, which can be a composite function of operations (such as batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 174b can also include a non-linear function (such as including a rectified non-linear ReLU operation) configured to extract a modified feature map.

[0166] Additionally, in practice, one of the layers 173b or 174b serves as an input layer for the input angular velocity and linear velocity and one of the layers 173b, 175b, and 176b can serve as an output layer that outputs a sequence of j successive joint velocities provided as the output from the LSTM layer 176b Alternatively, a single joint velocity can be regressed according to the fully connected layer 173b consisting of m units, each unit corresponding to each connection in the robot controlled by the robot controller. The fully connected layer 173b can have a linear or non-linear activation function.

[0167] During training, the training weights of the inverse prediction model 70b are continuously updated by comparing the predicted sequence of joint velocities (in the case of the linear velocity and angular velocity at a given input) with the ground truth sequence of joint velocities from a batch of training data set D (which can be systematically selected or randomly selected from a data memory (not shown)). More specifically, the coefficients of the filter can be initialized with predefined values or arbitrary values. The coefficients of the filter are applied to this batch of training data set D via forward propagation and are adjusted via backpropagation to minimize any output error.

[0168] In an application, the inverse prediction model 70b infers an n-vector of the joint velocities 61b of the intervention device 30 given the linear velocity and / or angular velocity 62b of the end effector 40.

[0169] Still referring to Figure 7E, the neural architecture of an exemplary embodiment of the inverse velocity model consists of an input, a neural network library, and an output. The input is the angular velocity and the linear velocity and the output is a sequence of joint velocities provided as the output from the LSTM layer Alternatively, a single joint velocity can be regressed according to a fully connected layer consisting of m units, each unit for each connection in the robot controlled by the robot controller. The fully connected layer can have a linear or non - linear activation function.

[0170] Reference Figure 7C and Figure 7D , stage S92d of the exemplary intervention process 90d uses the TTE probe 240 (FIG. 3G) and the navigation controller 103, and includes the navigation controller 103 determining the linear velocity and / or angular velocity 62b of the TTE probe 240 to the target pose and sending the linear velocity and / or angular velocity 62b to the inverse prediction model 70b. In practice, the navigation controller 130 can implement any known guidance algorithm known in the field of the present disclosure.

[0171] Stage S94d of process 90d involves the inverse prediction model 70b and the robot controller 101. The inverse prediction model 70b infers an n - vector of the joint velocities 61b of the intervention device 30 given the linear velocity and / or angular velocity 62b, and the continuous positioning controller 50c transmits a continuous positioning command 52b to the robot controller 101, thereby controlling the positioning of the TTE probe 240 to the target pose via the robot 230 (FIG. 3G) based on the predicted positioning movement of the intervention device 40.

[0172] In practice, the forward prediction model 60a ( Figure 4A ), the inverse prediction model 70a ( Figure 5A ), the forward prediction model 60b ( Figure 6A ), and the inverse prediction model 70b ( Figure 7A ) can use additional auxiliary information, such as images of anatomical structures (e.g., ultrasound images, endoscopic images, or X - ray images), forces measured at the end - effector, and the shape of the robot. Depending on the application, other inputs can also be included, including information from spectral tissue sensing devices, ECG signals or EEG signals, tissue conductivity, or other physiological signals. For example, if a continuum - like robot operates inside the human heart, the features available in the ultrasound image and the electrophysiological signals can improve the positioning of the end - effector relative to the anatomical structure, thus improving the guidance accuracy.

[0173] Reference Figure 8A and Figure 8B, in addition to being shown as trained on the forward kinematics of the intervention device, the forward prediction model 60c is also shown as trained on the sequence Q of joint variables 61a and the end - effector pose T 62a. In addition, the forward prediction model 60c is also shown as trained on the shape 35a, image 35b, and force 35c of the intervention device. Thus, in an application, the forward prediction model 60c will be able to predict the navigation pose of the end - effector based on the sequence Q of joint variables 61a and the shape 35a, image 35b, and force 35c of the intervention device.

[0174] One of ordinary skill in the art will know how to apply the shape 35a, image 35b, and force 35c of the intervention device and any other additional auxiliary information to the inverse prediction model 70a, forward prediction model 60b, and inverse prediction model 70b.

[0175] Figures 9A - 9E Illustrated is the training and application of the forward prediction model 60d of the present disclosure, which is trained on the forward kinematics of the intervention device 30 (FIG. 1) that predicts the navigation pose of the end - effector 40 (FIG. 1) and the shape of the robot, thereby facilitating the application of the forward prediction model 60d to a sequence of successive shapes of the intervention device 30 having embedded OSS technology during an intervention process to present the predicted navigation pose and shape of the end - effector 40. Thus, the continuous positioning controller 50 (FIG. 1) can generate continuous positioning data 51c regarding the information that the intervention device 30 relocates the end - effector 40 to a target pose based on the predicted navigation pose of the end - effector 40.

[0176] More specifically, referring to Figure 9A , in the training phase, a training controller (not shown) is configured to interpret the ground - truth training data set D as exemplarily taught in the description of FIGS. 16 - 18. The data set consists of n sequences W of i data points represented by the 2 - tuple d i =(H i ,H i+1 ). The 2 - tuple consists of a sequence (H i ∈(h t ,h t+1 …h t+k ) of k successive shapes 61d, where h∈(p1…p m ) is a set of m vectors p m that describe both the position and auxiliary shape parameters (e.g., strain, curvature, and twist) of an OSS sensor (e.g., a shape - sensing guidewire) embedded in the intervention device 30 in 3D Euclidean space. The 2 - tuple also consists of a sequence H t+k+1 of k successive shapes 62b that includes a future time point h i+1 ∈(ht+1 , h t+2 …h t+k+1 ) consists of, where h ∈ (p1…p m ) is a set of m vectors p m , which describes both the position and auxiliary shape parameters (e.g., strain, curvature, and twist) of the OSS intervention device 30 in 3D Euclidean space.

[0177] In practice, the training dataset D is a collection of expert data that reasonably covers different navigations of the OSS intervention device 30. To this end, the diverse dataset D for learning should include anatomical structures with different curvatures and motion amplitudes, mechanical differences between various types of robots, wear of hardware components, and other system-related factors (e.g., temperature and humidity of the environment).

[0178] Reference Figure 9B , the application phase of the forward prediction model 60d involves the continuous positioning controller 50d using the forward prediction model 60d to execute a deep learning type algorithm, where the recurrent layer is trained on expert data that reasonably covers different situations. The diverse dataset for learning includes various working conditions (temperature, fiber bending, etc.), different operating motions of the device, and differences in hardware (fibers, interrogators, etc.).

[0179] In one embodiment as Figure 9E shown, the forward prediction model 60d uses a neural architecture that sequentially includes an input layer 163a, a sequence-to-sequence model 263a, an output layer 262a, and an extraction layer 264. This neural architecture is a deep convolutional neural network with a recurrent layer that is configured to infer a future sequence consisting of k shapes Therefore, the last shape in the predicted sequence is used to estimate the position of the OSS intervention device 30 at a future time point.

[0180] During training, the forward prediction model 60d's training weights are continuously updated by comparing the sequence of future shapes i (predicted by the model given an input sequence H ) with the ground truth future sequence H i+1 from the training dataset D (which can be systematically selected or randomly selected from a data memory (not shown)). More specifically, the coefficients of the filter can be initialized with predefined values or arbitrary values. The coefficients of the filter are applied to this batch of the training dataset D via forward propagation and adjusted via backpropagation to minimize any output error.

[0181] During application, the forward prediction model 60d infers a future sequence consisting of k shapes Thus, the last shape in the prediction sequence is used to estimate the position of the OSS intervention device 30 at a future time point.

[0182] In an alternative embodiment as shown in Figure 9F the forward prediction model 60d uses a many-to-one model 263b instead of the sequence-to-sequence model 263a, whereby the last layer will be the last shape

[0183] Referring to Figure 9C and Figure 9D in stage S92e of the exemplary intervention process 90e, the OSS guidewire 332 (FIG. 3F) and the shape sensing controller 103 (FIG. 3F) are used, and it includes the shape sensing controller 103 measuring and storing the shape of the OSS guidewire 332 during the navigation of the end effector 340 to the target pose.

[0184] Stage S94e of process 90e involves the shape sensing controller 103, the forward prediction model 50d, and the display controller 104. The shape sensing controller 103 transmits a sequence 61d of k successive shapes to the forward prediction model 60e, thereby inferring the next shape sequence wherein the last shape is the predicted position of the OSS guidewire 332. The display controller 104 controls the display of the sensed position image 105a of the OSS guidewire 332 for the purpose of guiding the end effector 340 to the target pose.

[0185] Referring to Figure 10A and Figure 10B the imaging prediction model 80a of the present disclosure is trained on expert data that reasonably covers different situations. Given an ultrasound image 81a, the neural network 80a infers the relative position 82a of the image 81a with respect to a reference anatomical structure. As a result, as will be further described herein, the movement between the previous pose of the end effector in an anatomical structure (e.g., the heart) and the current pose of the end effector in the anatomical structure (e.g., the heart) can be calculated.

[0186] Figure 10A Illustrates the batch training of the imaging prediction model 80a. During training, the network continuously updates the weights of the imaging prediction model 80a using 2-tuples from a ground truth dataset that consists of ultrasound images and the relative position of the image with respect to a reference anatomical structure.

[0187] Figure 10BShows real-time inference using the imaging prediction model 80a for a given image 81a, which predicts the relative pose 82a of the end effector n with respect to a reference anatomical structure (e.g., a reference ultrasound image).

[0188] During the training phase, a data acquisition controller (not shown) is configured to receive and interpret information from the robot and the end effector (e.g., an ultrasound device) and store the data in a data storage medium (not shown) in a format defined by the following specification: The training dataset D consists of i data points represented by the 2-tuple d i =(U i ,T i ). The 2-tuple consists of an ultrasound image U i 81a acquired at a certain position T∈SE(3)82a relative to a reference position.

[0189] The training controller is configured to interpret the training dataset D stored on the data storage medium. The dataset D consists of i data points represented by the 2-tuple d i =(U i ,T i ). The 2-tuple consists of the following: an ultrasound image U i 81a of the anatomical structure and the relative motion T i 82a between the current pose of the end effector when acquiring the ultrasound image U i and an arbitrarily selected reference position.

[0190] In one embodiment as Figure 12A shown, the image prediction model 80a uses a neural network library 180a, which includes an input layer, hidden layers, and an output layer, and these layers are derived from a combination of one or more fully connected layers (FCL) 183a, one or more convolutional layers (CNL) 184a, one or more recurrent layers (RCL) 185a, and one or more long short-term memory (LSTM) layers 186a.

[0191] In practice, the combination of these layers is configured to implement the relative positioning of the image U C to the reference image and thereby achieve the pose

[0192] In one embodiment for implementing the relative positioning of the image U C to the reference image and thereby achieve the pose , the neural network library 180a includes a set of N fully connected layers 183a.

[0193] In one for implementing the relative positioning of the image U CRelative positioning to a reference image and thus achieving an attitude In a second embodiment, the neural network library 180a includes a set of N convolutional layers 184a, followed by a set of M fully connected layers 183a or a set of W recurrent layers 185a or a set of W long short-term memory layers 186a.

[0194] In the case of implementing the image U C Relative positioning to a reference image and thus achieving an attitude In a third embodiment, the neural network library 180a includes a set of N convolutional layers 184a, followed by a combination of a set of M fully connected layers 183a and a set of W recurrent layers 185a or a set of W long short-term memory layers 186a.

[0195] In practice, the fully connected layer 183a may include K neurons, where N, M, W, and K can be any positive integers, and the values can vary according to the embodiment. For example, N can be approximately 8, M can be approximately 2, W can be approximately 2, and K can be approximately 1000. Moreover, the convolutional layer 184a can implement a non-linear transformation, which can be a composite function of operations (e.g., batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 184a can also include a non-linear function (e.g., including the rectified non-linear ReLU operation) configured to extract a modified feature map.

[0196] In addition, in practice, one of the layers 183a or 184a serves as an input layer for the input image U C and one of the layers 183a, 185a, and 186a can serve as an output layer for outputting the attitude 182a of the end effector 40 in Cartesian space (e.g., the translation and rotation of the end effector 40 in Cartesian space). The output attitude of the end effector 40 in Cartesian space can be represented as a vector parameterization and / or non-vector parameterization of the position and orientation of a rigid body. More particularly, the parameterization can be in the form of Euler angles, quaternions, matrices, exponential maps, and / or angle axes representing rotation and / or translation (e.g., including the direction and magnitude of translation).

[0197] Moreover, in practice, the output layer can be a non-linear fully connected layer 183a that gradually reduces the high-dimensional output of the last convolutional layer 184a of the neural network library 180a to produce a set of output variables.

[0198] During training, by using the predicted relative motion of the end effector relative to a certain reference anatomical structure obtained by using the image prediction model (given the ultrasound image 161c and using it as an input) The training weights of the image prediction model 80a are continuously updated by comparing with the ground truth relative motion T (which can be systematically or randomly selected from a data memory (not shown)) from a batch of training data set D, which can be systematically or randomly selected from a data memory (not shown). More particularly, the coefficients of the filter can be initialized with predefined values or arbitrary values. The coefficients of the filter are applied to this batch of training data set D via forward propagation and are adjusted via backpropagation to minimize any output error.

[0199] Reference Figure 11A and Figure 11B , the imaging prediction model 80ba of the present disclosure is trained on expert data that reasonably covers different situations. Given an ultrasound image 81a, the neural network 80ba infers the relative position 82a of the image 81a with respect to a reference anatomical structure. As a result, as will be further described herein, the motion between the previous pose of the end effector in an anatomical structure (e.g., the heart) and the current pose of the end effector in the anatomical structure (e.g., the heart) can be calculated.

[0200] Figure 11A Illustrates the batch training of the imaging prediction model 80ba. During training, the network continuously updates the weights of the imaging prediction model 80ba using 2-tuples from a ground truth data set, which consists of an ultrasound image and the relative position of the image with respect to a reference anatomical structure.

[0201] Figure 11B Illustrates real-time inference using the imaging prediction model 80ba given an image 81a, which predicts the linear and angular velocities 83a of the end effector with respect to a reference anatomical structure (e.g., a reference ultrasound image).

[0202] In the training phase, a data acquisition controller (not shown) is configured to receive and interpret information from a robot and an end effector (e.g., an ultrasound device) and store the data in a data storage medium (not shown) in a format defined by the following specification via the data acquisition controller (not shown): The training data set D consists of i data points represented by a 2-tuple d i =(U i ,V i ). The 2-tuple consists of an ultrasound image U i 81a acquired at a certain position T∈SE(3)82a with respect to a reference position obtained from the linear and angular velocity vectors 83a of the end effector.

[0203] The training controller is configured to interpret the training data set D stored on the data storage medium. The data set D consists of a 2-tuple d i=(U i , V i ) and is composed of i data points. The 2-tuple consists of the following: the ultrasound image U of the anatomical structure i 81a and the relative n-vector 83a of the linear and angular velocities of the end effector at the time of acquiring the ultrasound image U i with respect to an arbitrarily selected reference position.

[0204] In one embodiment as shown in Figure 12B , the image prediction model 80b uses a neural network library 180b, which includes an input layer, a hidden layer, and an output layer, and these layers are derived from a combination of one or more fully connected layers (FCL) 183b, one or more convolutional layers (CNL) 184b, one or more recurrent layers (RCL) 185b, and one or more long short-term memory (LSTM) layers 186b.

[0205] In practice, the combination of these layers is configured to implement the relative positioning of the image U C to the reference image and thereby derive the linear and / or angular velocity of the end effector 40.

[0206] In one embodiment for implementing the relative positioning of the image U C to the reference image and thereby derive the linear and / or angular velocity of the end effector 40, the neural network library 180b includes a set of N fully connected layers 183b.

[0207] In a second embodiment for implementing the relative positioning of the image U C to the reference image and thereby derive the linear and / or angular velocity of the end effector 40, the neural network library 180b includes a set of N convolutional layers 184b, followed by a set of M fully connected layers 183b or a set of W recurrent layers 185b or a set of W long short-term memory layers 186b.

[0208] In a third embodiment for implementing the relative positioning of the image U C to the reference image and thereby derive the linear and / or angular velocity of the end effector 40, the neural network library 180b includes a set of N convolutional layers 184b, followed by a combination of a set of M fully connected layers 183b and a set of W recurrent layers 185b or a set of W long short-term memory layers 186b.

[0209] In practice, the fully connected layer 183b may include K neurons, where N, M, W, and K may be any positive integers, and the values may vary according to the embodiments. For example, N may be approximately 8, M may be approximately 2, W may be approximately 2, and K may be approximately 1000. Moreover, the convolutional layer 184b may perform a non-linear transformation, which may be a composite function of operations (such as batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 184b may also include a non-linear function (such as including a rectified non-linear ReLU operation) configured to extract a modified feature map.

[0210] Additionally, in practice, one of the layers 183b or 184b serves as an input layer for the input image U C and one of the layers 183b, 185b, and 186b may serve as an output layer for outputting the linear and angular velocities of the end effector regressed according to the last fully connected layer (e.g., 6 units, 3 units for linear velocity, and 3 units for angular velocity) using a linear or non-linear activation function.

[0211] During training, the training weights of the image prediction model 80b are continuously updated by comparing the predicted linear and angular velocities of the end effector relative to a certain reference anatomical structure (given the ultrasound image 161c as the input) with the true linear and angular velocities of the end effector describing the movement to a certain reference anatomical structure from a batch of training data set D (which may be systematically selected or randomly selected from a data memory (not shown)). More particularly, the coefficients of the filter may be initialized with predefined values or arbitrary values. The coefficients of the filter are applied to this batch of training data set D via forward propagation and are adjusted via backpropagation to minimize any output error.

[0212] Reference Figure 13A and Figure 13B , the continuous positioning controller 50f uses the inverse prediction model 70a, the image prediction model 80a, the subtractor 53, and the control law 54a to perform the closed-loop continuous position control method of the present disclosure as represented by the flowchart 190a.

[0213] In one embodiment of the TEE probe, stage S192a of process 190a includes inserting the TEE probe handle 132 (FIG. 3A) into a robotic controller 100 (FIG. 3B) known in the art of the present disclosure to control the toggling of the handle 132 and the rotation of the TEE probe 130. The TEE transducer 140 (FIG. 3A) is inserted into the body through the esophagus and positioned near the anatomical structure of interest (e.g., the heart) (e.g., positioned as Figure 13Cat the mid-esophageal position shown). The ultrasound image parameters are defined at this target pose of the TEE transducer 140.

[0214] Stage S194a of process 190a includes the image prediction model 90 processing the current ultrasound image 81a (which we will refer to as the previous ultrasound image U f ) to predict the relative position of this image plane with respect to the reference anatomical structure The sonographer uses the ultrasound image to observe the anatomical structure and determine the movement T that the transducer needs to make from its current position d . Alternatively, the movement of the required transducer can be provided from: an external tracking device, a user interface, other imaging modalities registered to the ultrasound image (e.g., an X-ray image registered to a 3D TEE image using EchoNavigator (Philips)).

[0215] Based on T d , the inverse prediction model 70a predicts the joint variables required to move the robot to the desired position 72a. The robot controller 100 receives the joint variables 72a and moves the TEE probe 130 accordingly.

[0216] Stage S196a of process 190a includes the ultrasound transducer reaching another position when acquiring the ultrasound image U c . The image prediction model 90g uses the processing of the current ultrasound image U c to predict the relative positions of two current image planes with respect to the reference anatomical structure As a result, the movement between the previous position and the current position in the anatomical structure (e.g., the heart) can be calculated as follows:

[0217] In the second TEE probe embodiment, as Figure 13D shown, the desired position T can be determined using path selection on the image 201 generated by an external imaging modality registered to the ultrasound image d (e.g., by using methods known in the art (Philips EchoNavigator), X-ray images as well as cone beam CT images can be registered to the ultrasound image).

[0218] Referring to Figure 14A and Figure 14B , the continuous positioning controller 50g uses the inverse prediction model 70b, the image prediction model 80b, the subtractor 53, and the control law 54b to perform the closed-loop continuous position control method of the present disclosure as represented by the flowchart 190b.

[0219] In one embodiment of a TEE probe, stage S192b of process 190b includes inserting the handle 142 of the TEE probe (FIG. 3A) into a robotic controller 100 (FIG. 3B) known in the art of the present disclosure to control the toggling of the handle 142 and the rotation of the TEE probe 140. The TEE transducer 140 (FIG. 3A) is inserted through the esophagus into the body and positioned near the anatomical structure of interest (e.g., the heart) (e.g., positioned at the mid-esophageal location as Figure 14C shown). The ultrasound image parameters are defined in this target pose of the TEE transducer 140.

[0220] Stage S194b of process 190b includes the user desiring to move the transducer in the image space, for example, by selecting a path from point A to point B in the ultrasound image 203 or a transformation between image planes A and B. In this embodiment, the 末端执行器 J 图像 Jacobian 204 transforms the first linear velocity and the first angular velocity 202 defined by the path on the image to the end effector coordinate system. 末端执行器 J 图像 The Jacobian 204 is calculated by using methods known in the art of the present disclosure to know the spatial relationship between the end effector and the image coordinate system.

[0221] Based on V d , the inverse prediction model 70b ( Figure 7B ) predicts the joint velocities 72b required to move the robot to the desired position. The robotic controller 100 receives the joint velocities 72b and moves the TEE probe 130 accordingly.

[0222] Stage S196b of process 190a includes another position when the ultrasound transducer reaches the U c at which to acquire an ultrasound image. The image prediction model 80b processes the current ultrasound image U c 81a to predict the velocity vector 83a of the end effector between points A and B. The image prediction model 80b estimates the function between the Cartesian velocity of the end effector and the velocity in the joint space, i.e., the neural network models the manipulator Jacobian given a 6-vector consisting of the linear velocity and angular velocity of the end effector 71c to predict the n-vector 72c of joint velocities.

[0223] As will be understood by those skilled in the art of the present disclosure, a neural network that models the spatial relationship between images of an anatomical structure (e.g., the heart) requires a large training dataset specific to a given organ.

[0224] In an alternative embodiment, features are directly extracted from the image to verify the position of the transducer. In this embodiment as shown in Figure 14C , the user selects an object on the image 205, e.g., the apical wall, and the system will extract certain prominent features around the object via feature extraction 206. These features (which may include the edges, shape, and size of the object) are first detected using methods known in the art of the present disclosure (e.g., Canny edge detector, morphological operations, etc.), and secondly, the scale-invariant feature transform (SIFT) known in the art of the present disclosure is used to track these features. Finally, the system defines a path between the selected object and the center of the field of view, which path indicates the desired movement on the image. By using SIFT to track prominent features, the continuous position controller 50g ( Figure 14A ) can correct the prediction from the network within closed-loop control.

[0225] More particularly, it is a velocity-based control system for a continuum robot. Once the user selects an object on the image (e.g., the apical wall (see the red dot on the ultrasound image)), the desired movement on the image is identified. The movement is defined by the path between the center of the field of view and the selected object, and the Jacobian 204 can be used to transform this movement into the linear and angular velocities in the end-effector space. Then this Cartesian velocity is sent to the neural network, which will infer the joint velocities. The achieved position will be iteratively verified against the path defined by the continuously tracked object and the center of the field of view.

[0226] In practice, the closed control loop of FIGS. 13 and 14 can be closed using other modalities, e.g., optical shape sensing (OSS), electromagnetic tracking, or an X-ray image registered to the ultrasound image using, e.g., a Philips EchoNavigator.

[0227] Moreover, in practice, the prediction accuracy of the neural network is affected by the configuration of the flexible endoscope. Therefore, first, the position of the transducer relative to the heart is defined using, e.g., neural network g or a Philips HeartModel (which will implicitly define one of the possible configurations). Secondly, a specific set of network weights is loaded into the model according to the detected configuration, thereby improving the prediction accuracy.

[0228] A similar method can be used to guide the user to a position that can provide optimal images and guidance.

[0229] Moreover, one of the most difficult problems in machine / deep learning is being able to access big data in the right format to train a prediction model. More specifically, collecting and constructing training sets and validation sets is very time-consuming and expensive because it requires domain-specific knowledge. For example, to train a prediction model to accurately distinguish between benign and malignant breast tumors, such training requires thousands of ultrasound images annotated by radiology experts and transforming them into a numerical representation that the training algorithm can understand. Additionally, the image dataset may be inaccurate, corrupted, or have noisy labels, all of which factors can lead to inaccurate detection, and collecting large medical datasets may raise ethical and privacy issues as well as many other problems.

[0230] Reference Figure 15 , the training data collection system of the present disclosure uses a shape sensing controller 103 that provides a 3D position vector 233 for each point on an optical shape sensing fiber 332 that is mounted to an endoscope 131 of a robot and a tether-driven manipulator 231. A distal portion of the optical fiber 332 is embedded in a plastic housing that rigidly attaches the optical fiber to the end effector and introduces a certain curvature within the shape.

[0231] For example, Figure 16A and Figure 16B shows a distal end 332d of an optical shape sensing fiber 332 that is embedded in a plastic housing 350 that is rigidly attached to an ultrasound transducer 232 of a manipulator 231 as shown in Figure 16C and Figure 16D . The plastic housing 350 imposes a certain curvature on the shape, thereby enabling the use of template matching algorithms known in the art of the present disclosure to estimate the end effector pose (e.g., a data acquisition sequence 370 as shown in Figure 17 , where α and β on each axis correspond to Figure 15 the knob positions on a TEE handle 132 as shown in

[0232] Returning to FIG. 16, generally, by detecting such a pattern, the pose T ∈ SE(3) of the end effector can be estimated using methods known in the art. The robot controller 100 sends motion commands to the robot, and the robot control is responsible for actuating knobs that tighten or loosen the tethers. By changing the state of the tethers, the position of the end effector changes, as shown in FIGS. 3D and 3E. The data storage controller 190 receives the shape h ∈ (p1…p n ) of the optical fiber, the pose T of the end effector, and the motion commands (i.e., joint positions) q t from the shape sensing controller and the robot controller, respectively. The data is stored on a storage device as a 3-tuple and later used to train the deep convolutional neural network of the present disclosure.

[0233] More specifically, the shape-sensing guidewire 332 is embedded or attached to the continuum robot, and the shape-sensing guidewire 332 uses optical shape sensing (OSS) technology known in the art. OSS uses light along a multi-core optical fiber for device positioning and navigation during surgical intervention. The principle involved utilizes distributed strain measurements in the optical fiber using characteristic Rayleigh backscattering or controlled grating patterns.

[0234] The shape-sensing controller 103 is configured to acquire the shape of the shape-sensing guidewire 322 and estimate the pose T ∈ SE(3) of the end effector rigidly attached to the plastic housing 350, where the plastic housing 350 applies a certain curvature to the guidewire, as described above. The method for estimating the pose T is based on a well-defined curvature and a template matching algorithm known in the field of the present disclosure.

[0235] The data acquisition controller 191 is configured to generate a sequence of motor commands according to a predefined acquisition pattern (e.g., spiral, radial, or square motion, etc.) and send movement commands to the robot controller 100.

[0236] The robot controller 100 is configured to receive the robot position and send a movement signal to the robot. The electric knob robot will tighten / relax the tether, which will cause the movement of the end effector. The robot controller is also configured to receive and interpret information from the data acquisition controller 191 and change the robot position based on the information from the data acquisition controller 191.

[0237] The data storage controller 190 is configured to receive and interpret information from the robot controller 100 and the shape-sensing controller 103 and store the data in a data storage medium (not shown) in a format defined by the following specifications:

[0238] The first specification is to acquire a training data set D for all configurations predefined by the data acquisition controller 191. The data set D consists of a set of n sequences W: D = {W1, W2, …, W n}, where each sequence W n is composed of i data points d i : W n = {d1, d2, …, d i}; and each data point d i from the sequence is defined by a 3-tuple: d i = (T i , H i , Q i ).

[0239] This 3-tuple consists of the end effector pose T ∈ SE(3), a sequence of k successive shapes (e.g., H ∈ (ht , h t+1 … h t+k )) consists of, where h ∈ (p1…p m ) is a set of m vectors p m , which describe the position of the shape-sensing guidewire in 3D Euclidean space, auxiliary shape parameters (e.g., strain, curvature, and twist), and a sequence of j successive joint variables (Q ∈ (q t , q t+1 … q t+j )) collected at sequential time points from t to t + j. For example, the entry q t can be the angles α, β on the control knob collected at time point t.

[0240] Reference Figure 19 , the training data collection method 360 of the present disclosure is performed by the Figure 15 training data collection system.

[0241] Reference Figure 15 and Figure 19 , stage S362 of method 360 includes moving the robot to the origin position by the robot controller 100 using, for example, limit switches or proximity sensors. The distal portion of the shape-sensing guidewire 232 is inserted into a notch 353 provided in the plastic housing 350. The notch 353 will impose a certain curvature on the shape.

[0242] The housing 350 is rigidly attached to the end effector of the continuum-like robot.

[0243] By using a template matching algorithm known in the art during stage S364 of method 360, the shape-sensing controller 103 can now estimate the pose T ∈

[0244] SE(3). Preferably, the coordinate system of the end effector is defined by the template. However, an additional calibration matrix can also be used. When the robot system is still in the origin position, the pose of the end effector in the OSS coordinate system is collected. Each subsequent pose collected during the experiment is estimated relative to this initial position.

[0245] The data acquisition controller 191 generates a motion sequence (i.e., a set of joint variables) according to a predefined acquisition pattern (e.g., Figure 18 pattern 370). The data acquisition controller 191 iteratively sends the motion sequence to the robot controller 100, and the robot controller 100 moves the robot according to the generated joint variables.

[0246] Stage S366 of method 300 includes the data storage controller 190 collecting and storing the data tuple d at each time pointi =(T i , H i , Q i ). Importantly, since H i and Q i are sequential, all previous time points are stored in the memory by the data storage controller 190.

[0247] To facilitate further understanding of the various inventions of the present disclosure, the following description of Figure 19 teaches exemplary embodiments of the continuous positioning controller of the present disclosure. From this description, one of ordinary skill in the art will understand how to apply various aspects of the present disclosure to make and use additional embodiments of the continuous positioning controller of the present disclosure.

[0248] Referring to Figure 19 , the continuous positioning controller 400 includes one or more processors 401, a memory 402, a user interface 403, a network interface 404, and a storage device 405 interconnected via one or more system buses 406.

[0249] Each processor 401 can be any hardware device capable of executing instructions stored in the memory 402 or the storage device or otherwise processing data, as known in the art of the present disclosure or contemplated hereinafter. In a non-limiting example, the processor(s) 401 can include a microprocessor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other similar devices.

[0250] The memory 402 can include various memories, e.g., non-transitory memory and / or static memory as known in the art of the present disclosure or contemplated hereinafter, including but not limited to L1, L2, or L3 cache memories or system memory. In a non-limiting example, the memory 402 can include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read only memory (ROM), or other similar memory devices.

[0251] The user interface 403 can include one or more devices for enabling communication with a user such as an administrator, as known in the art of the present disclosure or contemplated hereinafter. In a non-limiting example, the user interface can include a command line interface or a graphical user interface that can be presented to a remote terminal via the network interface 404.

[0252] The network interface 404 may include one or more devices for implementing communication with other hardware devices as known in the art of the present disclosure or contemplated hereinafter. In a non-limiting example, the network interface 404 may include a network interface card (NIC) configured to communicate according to the Ethernet protocol. Additionally, the network interface 404 may implement a TCP / IP stack for communication according to the TCP / IP protocol. Various alternative or additional hardware or configurations for the network interface 404 will be apparent.

[0253] The storage device 405 may include one or more machine-readable storage media as known in the art of the present disclosure or contemplated hereinafter, including but not limited to read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, or similar storage media. In various non-limiting embodiments, the storage device 405 may store instructions for execution by the processor(s) 401 or data on which the processor(s) 401 may operate. For example, the storage device 405 may store a basic operating system for controlling various basic operations of the hardware. The storage device 405 also stores application modules in the form of executable software / firmware for implementing the various functions of the controller 400a as previously described in the present disclosure, including but not limited to the forward prediction model(s) 60, the inverse prediction model(s) 70, and the imaging prediction model(s) 80 as previously described in the present disclosure.

[0254] In practice, the controller 400 may be installed within an X-ray imaging system 500, an interventional system 501 (e.g., an interventional robotic system), or a stand-alone workstation 502 (e.g., a client workstation or a mobile device such as a tablet) that communicates with the X-ray imaging system 500 and / or the interventional system 501. Alternatively, the components of the controller 400 may be distributed among the X-ray imaging system 500, the interventional system 501, and / or the stand-alone workstation 502.

[0255] Moreover, in practice, additional controllers of the present disclosure, including a shape sensing controller, a data storage controller, and a data acquisition controller, may also each include one or more processors, a memory, a user interface, a network interface, and a storage device interconnected via one or more system buses as arranged in Figure 19 such that the storage device contains applicable application modules of the controller as previously described herein. Alternatively, two or more controllers of the present disclosure may be integrated into a single controller, where the storage device contains applicable application modules of the two or more controllers as previously described herein.

[0256] Referring to FIGS. 1 - 19, those of ordinary skill in the art of the present disclosure will recognize many benefits of the present disclosure.

[0257] In addition, as those of ordinary skill in the art will recognize in view of the teachings provided herein, the structures, elements, components, etc. described in the present disclosure / specification and / or depicted in the accompanying drawings can be implemented in various combinations of hardware and software, and provide functions that can be combined in a single element or multiple elements. For example, the functions of the various structures, elements, components, etc. shown / illustrated / depicted in the drawings can be provided by using dedicated hardware and hardware that can run software associated with appropriate software for additional functions. When provided by a processor, the functions can be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors, some of which can be shared and / or multiplexed. Additionally, the explicit use of the terms "processor" or "controller" should not be construed to refer exclusively to hardware capable of running software, and can implicitly include, without limitation, digital signal processor ("DSP") hardware, memory (e.g., read-only memory ("ROM") for storing software, random access memory ("RAM"), non-volatile storage, etc.), and any unit and / or machine that is substantially capable of (and / or configurable to) execute and / or control processes (including hardware, software, firmware, combinations thereof, etc.).

[0258] Furthermore, all statements herein reciting the principles, aspects, and embodiments of the present invention, as well as specific examples thereof, are intended to cover their structural and functional equivalents. Additionally, such equivalents are intended to include both currently known equivalents and equivalents developed in the future (i.e., any elements developed that can perform the same or substantially similar functions regardless of their structure). Thus, for example, in view of the teachings provided herein, those of ordinary skill in the art should recognize that any block diagrams presented herein can represent a conceptual view of illustrative system components and / or circuits implementing the principles of the present invention. Similarly, in view of the teachings provided herein, those of ordinary skill in the art should recognize that any flowcharts, flow diagrams, etc. can represent various processes that can be substantially represented in a computer-readable storage medium and run by a computer, processor, or other device having processing capabilities, whether or not such a computer or processor is explicitly shown.

[0259] Having described the preferred and exemplary embodiments of the various inventions of the present disclosure, which are intended to be illustrative and not limiting, it should be noted that, in accordance with the teachings provided herein (including the drawings), those skilled in the art can make modifications and variations. Accordingly, it should be understood that changes within the scope of the embodiments disclosed herein can be made to the preferred and exemplary embodiments of the present disclosure.

[0260] In addition, it should be anticipated that corresponding and / or related systems that include and / or implement the devices / systems according to the present disclosure, or corresponding and / or related systems that can be used / implemented in the devices, for example, are also anticipated and considered to be within the scope of the present disclosure. Moreover, corresponding and / or related methods for manufacturing and / or using the devices and / or systems according to the present disclosure are also anticipated and considered to be within the scope of the present disclosure.

Claims

1. A positioning controller (50) for an interventional device (30), the interventional device including an imaging device and at least one joint, the positioning controller (50) comprising: A memory storing: An imaging prediction model (80) configured, using the kinematics of the interventional device, to receive imaging data and output data related to a prediction of a navigation pose of the imaging device (40) or a portion of the interventional device (30) associated with the imaging device, and A control prediction model (70) configured, using the kinematics of the interventional device, to receive error positioning data and output data related to a prediction of a corrective positioning movement of the interventional device (30); And At least one processor in communication with the memory, wherein the at least one processor is configured to: Apply the imaging prediction model (80) to the imaging data generated by the imaging device (40) to present a predicted navigation pose of the imaging device (40) or a predicted navigation pose of a portion of the interventional device associated with the imaging device, Generate error positioning data derived from a difference between a determined target pose and the predicted navigation pose presented by the imaging prediction model, Apply the control prediction model (70) to the generated error positioning data to present a predicted corrective positioning movement of the interventional device (30), wherein the control prediction model (70) is configured to infer an n - vector of joint velocities of the interventional device given a linear velocity and / or an angular velocity of the imaging device (40) or the portion of the interventional device associated with the imaging device to the target pose; and Generate a positioning command for controlling the interventional device (30) to correctively position the imaging device (40) or the portion of the interventional device associated with the imaging device to the target pose based on the predicted corrective positioning movement of the interventional device (30).

2. The positioning controller according to claim 1, wherein, The imaging prediction model (80) is trained or has been trained on a correlation between relative imaging performed by the imaging device (40) and the forward kinematics of the interventional device (30).

3. The positioning controller according to claim 2, wherein, The control prediction model (70) is an inverse prediction model that is trained or has been trained on the inverse kinematics of the interventional device (30).

4. The positioning controller according to any one of claims 1-3, wherein, The imaging device is associated with or born of an end - effector of the interventional device.

5. The positioning controller according to any one of claims 1 to 3, wherein, The positioning controller is further configured to continuously generate the positioning command such that the positioning controller is considered a continuous positioning controller.

6. The positioning controller (50) according to any one of claims 1-3, wherein, The imaging prediction model (80) includes: A neural network library having an input layer and an output layer, the input layer being configured to input the imaging data generated by the imaging device (40), and the output layer being configured to output at least one of translation, rotation, and pivoting of the imaging device (40) derived from a relative positioning of the imaging data to a reference image. Wherein, at least one of the translation, rotation, and pivoting of the imaging device (40) infers the predicted navigation pose of the imaging device (40).

7. The positioning controller (50) according to any one of claims 1-3, wherein, The imaging prediction model (80) includes: A neural network library having an input layer and an output layer, the input layer being configured to input the imaging data generated by the imaging device (40), and the output layer being configured to output at least one of the linear velocity and angular velocity of the imaging device (40) obtained according to the relative positioning of the imaging data to a reference image. Wherein, at least one of the linear velocity and angular velocity of the imaging device (40) infers the predicted navigation pose of the imaging device (40).

8. The positioning controller (50) according to any one of claims 1-3, wherein, The imaging prediction model (80) includes: A feature extractor configured to track the movement of features in the imaging data generated by the imaging device (40); and A motion transformer configured to output at least one of the linear velocity and angular velocity of the imaging device (40) derived from the Jacobian transformation of the movement of the features tracked in the imaging data. Wherein, at least one of the linear velocity and angular velocity of the imaging device (40) infers the predicted navigation pose of the imaging device (40).

9. The positioning controller (50) according to claim 8, wherein, The feature extractor includes: A feature detector configured to detect features from the imaging data generated by the imaging device (40); and A scale-invariant feature transformer configured to track the movement of the detected features.

10. The positioning controller (50) according to any one of claims 1-3, wherein, The control prediction model (70) includes: A neural network library having an input layer and an output layer, the input layer being configured to input the error positioning data derived from the difference pose between the target pose of the imaging device (40) and the predicted navigation pose of the imaging device (40), and the output layer being configured to output at least one of the translation, rotation, and pivoting of the intervention device (30) obtained by regression of the difference pose between the target pose of the imaging device (40) and the predicted navigation pose of the imaging device (40). Wherein, at least one of the translation, rotation, and pivoting of the intervention device (30) infers the predicted correction positioning movement of the intervention device (30).

11. The positioning controller (50) according to any one of claims 1-3, wherein, The control prediction model (70) includes: A neural network library having an input layer and an output layer, the input layer being configured to input the error positioning data derived from the difference movement between at least one of the linear velocity and angular velocity of the imaging device (40) and at least one of the predicted linear velocity and predicted angular velocity of the imaging device (40), and the output layer being configured to output the joint velocity of the intervention device obtained by regression of the difference movement between at least one of the linear velocity and angular velocity of the imaging device (40) and at least one of the predicted linear velocity and predicted angular velocity of the imaging device (40).

12. The positioning controller (50) according to claim 3 Among them, The control prediction model (70) is also trained on at least one navigation parameter of the intervention device (30), and the at least one navigation parameter provides assistance to the inverse kinematics of the intervention device (30) for predicting the positioning movement of the intervention device (30); And wherein, the at least one processor is further configured to apply the inverse prediction model to both the error positioning data of the intervention device (30) and at least one auxiliary navigation parameter to present the predicted positioning movement of the intervention device (30).

13. The positioning controller (50) according to claim 1, Among them, The control prediction model (70) is arranged to: further receive at least one auxiliary navigation parameter of the intervention device (30), and further process the at least one auxiliary navigation parameter to output the prediction result of the corrected positioning movement of the intervention device (30); and wherein, the at least one processor is further configured to apply the control prediction model (70) to both the error positioning data of the intervention device (30) and the at least one auxiliary navigation parameter to present the predicted positioning movement of the intervention device (30).

14. A machine-readable storage medium encoded with instructions, the instructions being executed by at least one processor to instruct an intervention device with the instructions, the intervention device including an imaging device and at least one joint, the machine-readable storage medium including: An imaging prediction model (80), which is configured to receive imaging data and output data related to the prediction of the navigation pose of the imaging device (40) or the part of the intervention device (30) associated with the imaging device by using the kinematics of the intervention device; A control prediction model (70), which is configured to receive error positioning data and output data related to the prediction of the corrected positioning movement of the intervention device (30) by using the kinematics of the intervention device; and Instructions for: Applying the imaging prediction model (80) to the imaging data generated by the imaging device (40) to present the predicted navigation pose of the imaging device (40), Generating error positioning data derived from the difference between the determined target pose and the predicted navigation pose presented by the imaging prediction model, Applying the control prediction model (70) to the error positioning data to present the predicted corrected positioning movement of the intervention device (30), wherein the control prediction model (70) is configured to infer the n-vector of the joint velocity of the intervention device given the linear velocity and / or angular velocity of the imaging device (40) or the part of the intervention device associated with the imaging device to the target pose; and Generating a positioning command for controlling the imaging device (40) to be corrected to the target pose by the intervention device (30) based on the predicted corrected positioning movement of the intervention device (30).

Images