Training data collection for machine learning models

Abstract

A training data collection method for an interventional device (130), the interventional device comprising a portion (140) of the interventional device and a sensor (332) adapted to provide position and / or orientation and / or shape information, wherein at least a portion of the sensor is attached to the device portion (140). The method involves controlling at least one motion variable of the interventional device (130) according to a predefined pattern of data points, and determining an estimate of the pose of the device portion (140) and an estimate of the positioning motion of the interventional device (130) according to shape data derived from the information. The method also involves storing a temporal data sequence for the interventional device (130) derived from the estimated pose of an end effector (140) for each data point, the estimated positioning motion of the interventional device (130) for each data point, and (one or more) motion variables of the interventional device (130) for each data point.

Description

Technical Field

[0001] The present disclosure generally relates to positioning control of a portion of an interventional device (e.g., an end effector of an interventional device) used in an interventional procedure (e.g., minimally invasive surgery, video-assisted thoracic surgery, vascular procedures, endoluminal procedures, deformity correction procedures). The present disclosure can particularly relate to incorporating predictive models in positioning control of such a portion of an interventional device used in an interventional procedure. Background Art

[0002] Continuous (or non-continuous) control - positioning the device part (e.g., end effector) within a certain workspace - is one of the most commonly attempted forms of control in conventional rigid-link robots. By utilizing the discrete rigid-link structure of the robot, precise positioning of the part of the interventional device (e.g., end effector) as needed in structured applications (e.g., manufacturing) can be achieved. However, due to the deformable, fragile nature of soft tissues of human organs and patient safety, there is less demand for the use of rigid-link robots in clinical settings.

[0003] More specifically, biologically inspired robots can produce movements similar to those of snakes, elephants, and octopuses, which can be very effective in manipulating soft anatomical objects. Nevertheless, effective control of robotic structures (and especially effective continuous control) has proven to be extremely difficult to achieve in clinical settings, given that the complexity of continuum (or quasi-continuum) structures for the desired degrees of freedom is difficult to model mathematically or to provide sufficient actuator inputs for consistent control.

[0004] For example, a method for continuous positioning control of an end effector supported by a continuum robot involves modeling and controlling the configuration of the continuum robot, wherein a static model (which is formulated as a set of nonlinear differential equations) attempts to account for the deformation of the gun of the continuum robot due to bending, torsion, and elongation. However, the accuracy of the mathematical modeling is susceptible to changes in the robot's environmental conditions (e.g., temperature and humidity), which affect the mechanical properties of the robot's components, and is susceptible to any manufacturing errors or various workloads.

[0005] As another example, another approach for positioning control of an end effector supported by a robot is to manipulate the robot by projecting a set of allowable motions and a set of allowable forces into the joint space corresponding to the manipulator's control. For example, after inserting the robot into the nasal cavity, the controller adjusts the position of each segment of the robot to increase the difference between the measured generalized forces and the expected generalized forces on each end disc. However, increasing the robot's degrees of freedom to make it more maneuverable has the adverse effect of complicating the robot's kinematics. This can be particularly problematic in the case of continuously controlling a continuum robot.

[0006] Furthermore, even if effective positioning control of the end effector supported by the robot is achieved, improper calibration or use of the robotic system or general wear of the mechanical components may negatively impact the predictive accuracy of the kinematics of the robotic structure. Again, this may be particularly problematic in the case of continuously controlling a continuum robot (or continuum robotic structure).

[0007] WO 2018 / 064394 A1 describes a surgical robotic system for automatically calibrating tubular and flexible surgical tools (e.g., endoscopes). By compensating for the non-ideal behavior of the endoscope, the surgical robotic system is able to accurately model the motion of the endoscope and navigate the endoscope while performing a surgical procedure on a patient. During calibration, the surgical robotic system moves the endoscope to a target position and receives data describing the actual position and / or orientation of the endoscope. The surgical robotic system determines a gain value based at least on the inconsistency between the target position and the actual position.

[0008] US2018 / 055577 A1 describes a method of surgical robot operation, the method comprising moving a surgical instrument through a medium using a surgical robot, wherein the surgical robot attempts to move the surgical instrument to a desired position. The method also includes measuring an actual position of the surgical instrument and calculating a difference between the actual position and the desired position. The actual position of the surgical instrument is adjusted to the desired position using the surgical robot.

[0009] US2018 / 250075 A1 describes a learning / evolutionary system that robotically performs one or more activities of a medical procedure, wherein the medical procedure may include: diagnosing (one or more) medical conditions of a patient, treating (one or more) medical conditions, and robotically diagnosing (one or more) medical conditions of the patient and performing one or more medical procedure activities based on the diagnosis without user intervention.

[0010] US2016 / 349044 A1 describes a shape sensing system comprising a processor coupled to a memory storage device. A prediction model module is stored in the memory storage device and is configured to receive shape sensing measurements and predict a next shape sensing measurement based on current state data for a shape sensing enabled device. Summary of the invention

[0011] Known techniques designed for positioning control of portions of an interventional device (e.g., an end effector) provide limited benefits. Therefore, there remains a need for improved techniques for providing effective positioning control of portions of an interventional device. To this end, the present disclosure teaches feedforward positioning control, feedback positioning control, and data collection. The control is preferably performed continuously.

[0012] To this end, the present invention proposes a data collection system (optionally a training data collection system) according to any one of claims 1 to 12 as a first embodiment.

[0013] As a second embodiment and a third embodiment, the present invention proposes a non-transitory machine-readable storage medium encoded with instructions according to claim 13 and a method for data collection for an interventional device according to claim 14, wherein the data collection is optionally training data collection. The examples listed below that are not within the scope of the claims are used to assist in understanding the present invention.

[0014] Feedforward (preferably continuous) positioning control . The present disclosure also teaches a predictive model method for feedforward (preferably continuous) positioning control of manual or automated navigation positioning of a device portion (e.g., an end effector) supported by an interventional device based on a predictive model configured using the kinematics of the interventional device (optionally trained on these kinematics).

[0015] A further example of feedforward (preferably continuous) positioning control of a device part (e.g., an end effector) of the present disclosure is a (continuous) positioning controller comprising a forward prediction model and / or a control (optionally inverse) prediction model configured using the (optionally forward) kinematics of the interventional device that predicts the navigation posture of the end effector (optionally trained on these forward kinematics of the interventional device) and / or the control (optionally inverse) prediction model configured using the kinematics of the interventional device that predicts the positioning motion of the interventional device (optionally trained on the inverse kinematics of the interventional device that predicts the positioning motion of the interventional device).

[0016] For the description and purposes of the present disclosure, the term "navigation posture" broadly encompasses the posture of the portion of the interventional device (e.g., the end effector of the interventional device) as it is navigated to a spatial position via the interventional device during an interventional procedure, and the term "positioning motion" broadly encompasses any movement of the interventional device used to navigate that portion of the device to a spatial position during the interventional procedure.

[0017] In operation, the continuous positioning controller applies a forward prediction model to commanded positioning motions of the interventional device to render a predicted navigation pose of the end effector and generates positioning data with information regarding positioning of the device portion to a target pose by the interventional device based on the predicted navigation pose of the device portion.

[0018] Alternatively, previously or simultaneously, a (preferably continuous) positioning controller applies an inverse prediction model to a target pose of the part (e.g., an end effector) of the interventional device to render a predicted positioning motion of the interventional device, and generates a (continuous) positioning command to control the interventional device to position the device part to the target pose based on the predicted positioning motion of the interventional device.

[0019] Feedback (preferably continuous) positioning control . The present disclosure also teaches a prediction model method for feedback (preferably continuous) positioning control of manual navigation positioning or (semi-)automatic navigation positioning of an imaging device associated with or attached to a portion of an interventional device (e.g., an end effector of an interventional device) based on the following imaging prediction model: the imaging prediction model utilizes the kinematics of the interventional device (optionally related to the image data) and is configured to receive imaging data from the imaging device as feedback for manual navigation positioning or automatic navigation positioning of the imaging device (or the portion of the interventional device that is linked to the imaging device - e.g., the end effector) to a target pose.

[0020] An example of the present disclosure for feedback (preferably continuous) controlled positioning of an imaging device (or a portion of an interventional device that is linked to the imaging device - for example, an end effector) is a continuous positioning controller that includes an imaging prediction model trained on the correlation of relative imaging performed by the end effector with the forward kinematics of the interventional device that predicts the navigation posture of the end effector. The (continuous) positioning controller may also include a control prediction model that is configured using the kinematics of the interventional device to predict corrective positioning motions of the interventional device. Optionally, the control prediction model is trained or has been trained on the inverse kinematics of the interventional device to output said predictions of corrective positioning motions.

[0021] For the description and purposes of the present disclosure, the term "relative imaging" broadly encompasses the generation of an image of an interventional procedure by an end effector at a given pose relative to a reference image of the interventional procedure.

[0022] In operation, after a device portion (e.g., an end effector) is navigated to a target pose, a (preferably) continuous positioning controller applies an imaging prediction model to imaging data generated by the end effector to present a predicted navigation pose of the device portion (e.g., the end effector), applies a control (or particularly an inverse) prediction model to error positioning data derived from a difference between the target pose of the end effector and the predicted navigation pose of the end effector to present a predicted corrective positioning movement of the interventional device, and generates a continuous positioning command that controls the interventional device to correct the positioning of an imaging device (40) or a portion of the interventional device associated with the imaging device (e.g., the end effector) to the target pose based on the predicted corrective positioning movement of the interventional device.

[0023] (Optionally for training) Data collection . In addition, in order to facilitate the (optionally continuous) positioning control of manual navigation or automatic navigation positioning of a part (e.g., an end effector) of an interventional device via a predictive model that is invariant to environmental differences (e.g., anatomical differences between patients, such as patient size, heart position, etc.), the present disclosure also teaches a (optional training) data collection technique, which is premised on navigating and positioning the interventional device part via a predefined data point pattern and recording the spatial positioning of the interventional device and the posture of the interventional device part at each data acquisition point, thereby collecting (training) data for the predictive model to infer forward kinematics or inverse kinematics of the interventional device.

[0024] One embodiment of the present disclosure for collecting (optionally training) data for a predictive model is a (optionally training) data collection system for an interventional device, the interventional device comprising the portion of the interventional device (e.g., an end effector) and a sensor adapted to provide position and / or orientation and / or shape information, at least a portion of the sensor (332) being attached (optionally in a fixed shape) to the interventional device portion. Such sensors may include markers visible from an imaging system (e.g., an X-ray system, an MRI system), an electromagnetic tracking sensor, a transducer sensor, and / or optical shape sensing provided by / in an optical fiber.

[0025] A specific embodiment of this embodiment is a (optionally trained) data collection system for an interventional device, which includes a portion of the interventional device (e.g., an end effector) and an optical shape sensor, wherein a segment of the optical shape sensor is attached (optionally in a fixed shape) to the end effector.

[0026] The (training) data collection system may use a robot controller, a data acquisition controller, a position determination module (or a shape sensing controller in the specific embodiments mentioned above), and a data storage controller.

[0027] In operation, the data acquisition controller may command the robotic controller to control (one or more) motion variables of the interventional device according to a predefined data point pattern, and the positioning determination module (or optionally a shape sensing controller) is configured to determine the position information based on the position and / or orientation and / or shape information received from the sensor, so as to output an estimate of the pose of the portion of the interventional device portion at each data point of the predefined data point pattern and / or an estimate of the positioning motion of the interventional device. Thus, for the purpose of the estimation, the determined position information is optionally retrieved or derived or extracted or received based on the kinematics or interventional device behavior of the configuration positioning determination module. In a specific embodiment, the positioning determination module may derive or receive derived shape data based on the position and / or orientation and / or shape information to determine the estimation. If the positioning determination module is a shape sensing controller (as in the specific embodiment mentioned above), the positioning determination module controls shape sensing of the optical shape sensor, including the estimation of the pose of the end effector at each data point of the predefined data point pattern and the estimation of the positioning motion of the interventional device at each data point of the predefined data point pattern.

[0028] Said "deriving shape data based on said position and / or orientation and / or shape information" may be implemented according to known techniques for deriving shapes from data provided by "sensors". For example, the positions tracked by the sensors may be a good indication of the general shape of the segment of the interventional device carrying these sensors, and an algorithm (developed more or less based on the distances between the sensors and the possible shapes of the interventional device along that segment) may be developed to derive or reconstruct that shape. Such dynamic tracking of positioning may also give an indication of the orientation of the deformation. Sensors may also provide strain information (e.g., Rayleigh or Bragg grating sensors) which can indicate the local positioning and orientation of the interventional device, from which the shape can be derived and reconstructed (also known techniques).

[0029] An estimate of the pose of an interventional device portion (eg, an end effector) is derived from at least a portion of a sensor attached (optionally in a fixed shape) to the end effector.

[0030] Throughout the process in which the robotic controller can control (one or more) motion variables of the interventional device according to a predefined data point pattern, the data storage controller can receive communication information of the estimated pose of the end effector for each data point from the shape sensing controller, can receive communication information of the estimated positioning motion of the interventional device for each data point from the positioning determination module (or shape sensing controller), and can receive communication information of at least one motion variable of the interventional device for each data point from the robotic controller.

[0031] In response to the communication information, the data storage controller can store a temporal data sequence for the interventional device derived from the estimated pose of the end effector at each data point, the estimated spatial positioning of the interventional device at each data point, and (one or more) motion variables of the interventional device at each data point. The temporal data sequence can serve as training data for any type of machine learning model, particularly for the prediction model of the present disclosure.

[0032] In addition, the data acquisition controller can further command the robotic controller to control (one or more) motion variables of the interventional device according to additional (one or more) predefined data point patterns, whereby the data storage controller generates additional time data series for any type of machine learning model, particularly for the predictive model of the present disclosure.

[0033] Furthermore, for purposes of the description and claims of this disclosure:

[0034] (1) Terms of the art including but not limited to “end effector”, “kinematics”, “position”, “positioning”, “posture”, “achieving posture”, “motion” and “navigation” are interpreted as being known in the art of the present disclosure and exemplarily described in the present disclosure;

[0035] (2) Examples of end effectors include, but are not limited to, intraoperative imaging devices, interventional tools / surgical instruments, and surgical sutures as are known in the art of the present disclosure and contemplated hereinafter;

[0036] (3) The term "intraoperative imaging device" broadly encompasses all imaging devices used to illustrate anatomical objects / regions as known in the art of the present disclosure and contemplated hereinafter. Examples of intraoperative imaging devices include, but are not limited to, transesophageal echocardiography transducers (e.g., X7-2t transducer, Philips), laparoscopic ultrasound transducers (e.g., L10-4lap transducer, Philips), optical cameras, and sensing devices (e.g., tissue spectroscopy sensing sensors, ECG electrodes, probes for electrophysiological mapping);

[0037] (4) Examples of interventional tools / surgical instruments include, but are not limited to, scalpels, cauterizers, ablation devices, needles, forceps, Kirschner wires and associated drivers, endoscopes, awls, screwdrivers, osteotomes, chisels, tapers, curettes, clamps, forceps, periosteum, and J-pins as known in the art of the present disclosure and contemplated hereinafter;

[0038] (5) The term "interventional device" broadly encompasses all devices used to support positioning of an end effector during application as known in the art of the present disclosure and contemplated hereinafter. Examples of interventional devices include, but are not limited to, a continuum flexible robot, a flexible interventionoscope, and a guidewire;

[0039] (6) Examples of flexible robots include, but are not limited to, hyper-redundant robots (e.g., multiple discrete links, serpentine links, or concentric tubes), (e.g., cable-actuated) continuous backbone segment robots, tether-driven robots, gel-like soft robots, and fluid-filled tube-actuated robots as known in the art of the present disclosure and contemplated hereinafter;

[0040] (7) Examples of flexible interventional scopes include, but are not limited to, endoscopes with transesophageal echocardiography (TEE) probes, endoscopes with intracardiac echocardiography (ICE) probes, laparoscopes, and bronchoscopes as known in the art of the present disclosure and contemplated hereinafter;

[0041] (8) The term "prediction model" broadly encompasses all types of models as known in the art of the present disclosure and contemplated hereinafter that predict navigation variables related to the (optional) continuous positioning control of a portion of an interventional device (e.g., an end effector) according to the present disclosure as exemplarily described in the present disclosure. Optionally, these prediction models can be configured to output such prediction results based on a positioning dataset using kinematics. Optionally, the prediction model is trained or has been trained on the kinematics of the interventional device. Examples of prediction models include, but are not limited to, artificial neural networks (e.g., feed-forward convolutional neural networks, recurrent neural networks, long short-term memory networks, autoencoder networks, generative adversarial networks, and many other deep learning neural networks);

[0042] (9) The term "controller" broadly encompasses all structural configurations of a main circuit board or integrated circuit as understood in the art of the present disclosure and exemplarily described in the present disclosure for controlling the application of the various inventive principles of the present disclosure as exemplarily described in the present disclosure. The structural configuration of a controller may include, but is not limited to, processor(s), computer usable / computer readable storage medium(s), operating system(s), application program(s) modules, peripheral device(s) controller(s), slot(s) and port(s), control instructions. The controller may be housed within a workstation or communicatively connected to a workstation;

[0043] (10) the term "application module" broadly encompasses an application program contained within or accessible by a controller, the controller including electronic circuitry (e.g., electronic components and / or hardware) and / or an executable program (e.g., executable software stored on non-transitory computer-readable medium(s) and / or firmware) for executing a particular application program; and

[0044] (11) The terms "signal", "data" and "command" broadly encompass all forms of detectable physical quantities or pulses (e.g., voltage, current, or magnetic field strength) as understood in the art of the present disclosure and exemplarily described in the present disclosure for transmitting information and / or instructions supporting the application of the various inventive principles of the present disclosure as described later in the present disclosure. Signal / data / command communication of the various components of the present disclosure may involve any communication method as known in the art of the present disclosure, including but not limited to signal / data / command transmission / reception via any type of wired or wireless data link and reading of signals / data / commands uploaded to a computer-usable / computer-readable storage medium.

[0045] By reading the following detailed description of various embodiments of the present disclosure in conjunction with the accompanying drawings, the aforementioned embodiments and other embodiments of the present disclosure and the various structures and advantages of the present disclosure will become more apparent. The detailed description and the accompanying drawings are merely illustrative of the present disclosure and are not limiting. The scope of the present disclosure is defined by the claims and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] FIG. 1 illustrates an exemplary embodiment of an interventional device including an end effector as known in the field of the present disclosure.

[0047] Figure 2A Exemplary embodiments of a continuous position controller according to various aspects of the present disclosure are illustrated.

[0048] Figure 2BAn exemplary embodiment of a continuous positioning state machine according to various aspects of the present disclosure is illustrated.

[0049] FIG. 3A illustrates an exemplary embodiment of a transesophageal echocardiography (TEE) probe as is known in the art of the present disclosure.

[0050] 3B illustrates an exemplary embodiment of a handle of a transesophageal echocardiography (TEE) probe as known in the art of the present disclosure.

[0051] 3C-3F illustrate a method as known in the art of the present disclosure. Figure 2A Exemplary movement of the TEE probe.

[0052] 3G illustrates an exemplary embodiment of a robotically manipulated transthoracic echocardiography (TTE) probe as is known in the art of the present disclosure.

[0053] FIG. 3H illustrates an exemplary embodiment of an optical shape sensing continuum robot as is known in the art of the present disclosure.

[0054] Figures 4A-4E A first exemplary embodiment of a forward prediction model and a forward prediction method of the present disclosure is illustrated.

[0055] Figures 5A-5E A first exemplary embodiment of an inverse prediction model and an inverse prediction method of the present disclosure is illustrated.

[0056] Figures 6A-6E A second exemplary embodiment of the forward prediction model and the forward prediction method of the present disclosure is illustrated.

[0057] Figures 7A-7E A second exemplary embodiment of the inverse prediction model and the inverse prediction method of the present disclosure is illustrated.

[0058] Figures 8A-8B A third exemplary embodiment of the forward prediction model and the forward prediction method of the present disclosure is illustrated.

[0059] Figures 9A-9F A fourth exemplary embodiment of the forward prediction model and the forward prediction method of the present disclosure is illustrated.

[0060] Figures 11A-11E A first exemplary embodiment of an image prediction model and an image prediction method of the present disclosure is shown.

[0061] Figures 12A-12E A second exemplary embodiment of the forward prediction model and the forward prediction method of the present disclosure is illustrated.

[0062] Figures 13A-13DThe closed-loop attitude control of the present disclosure is illustrated.

[0063] Figures 14A-14D The closed loop vector speed control of the present disclosure is illustrated.

[0064] Figure 15-18 The training data collection system and method of the present disclosure are illustrated.

[0065] Fig.19 An exemplary embodiment of a continuous positioning controller of the present disclosure is illustrated. DETAILED DESCRIPTION

[0066] The present disclosure is applicable to a wide variety of applications requiring continuous position control of the end effector. Examples of such applications include, but are not limited to, minimally invasive procedures (e.g., endoscopic liver resection, necrosectomy, prostatectomy, etc.), video-assisted thoracic surgery (e.g., lobectomy, etc.), microvascular procedures (e.g., via catheters, sheaths, deployment systems, etc.), micromedical diagnostic procedures (e.g., intraluminal procedures via endoscopes or bronchoscopes), deformity correction procedures (e.g., via Kirschner wires, screwdrivers, etc.), and non-medical applications.

[0067] The present disclosure improves continuous position control of an end effector during such applications by providing predictions of the pose of the end effector and positioning of the motion of an interventional device that can be used to control and / or confirm manual or automated navigation positioning of the end effector.

[0068] To facilitate understanding of the present disclosure, the following description of FIG. 1 teaches an exemplary embodiment of an interventional device including an end effector as known in the art of the present disclosure. Based on the description of FIG. 1 , a person of ordinary skill in the art of the present disclosure will recognize how to apply the present disclosure to make and use additional embodiments of interventional devices including end effectors as known in the art of the present disclosure and contemplated below.

[0069] 1 , in practice, the interventional device 30 supports manual navigation positioning or automatic navigation positioning of the end effector 40 during application as represented by the arrow extending from the end effector 40. Examples of the interventional device 30 include, but are not limited to, a continuum flexible robot, a flexible interventional scope, and a guidewire, and examples of the end effector 40 include, but are not limited to, intraoperative imaging devices, interventional tools / surgical instruments, and surgical sutures.

[0070] In operation, (one or more) navigation commands 31, (one or more) actuation signals 32 and / or (one or more) navigation forces 33 are transmitted to / applied to the interventional device 30, whereby the interventional device 30 translates, rotates and / or pivots according to the (one or more) navigation commands 31, (one or more) actuation signals 32 and / or (one or more) navigation forces 33, thereby navigating the end effector 40 to a target pose (i.e., a position and orientation in an application space).

[0071] For example, FIGS. 3A and 3B show a transesophageal echocardiography (TEE) probe 130 as an embodiment of an interventional device 30 having an imaging end effector 140 that can be inserted through the mouth of a patient P into the esophagus to capture images of the heart of the patient P, and a practitioner (not shown) or a robotic controller 100 ( Figure 2B ) operates the handle 132 of the TEE probe 130 to reposition the TEE probe 130 within the patient P, thereby navigating the imaging end effector 140 to a target posture.

[0072] More specifically, the TEE probe 130 includes a flexible, elongated member 131, a handle 132, and an imaging end effector 140. The flexible, elongated member 131 is structurally arranged and / or otherwise configured to be positioned within a patient's body lumen (e.g., esophagus) in terms of size and / or shape. The imaging end effector 140 is mounted at the distal end of the member 131 and includes one or more ultrasonic transducer elements, whereby the imaging end effector 140 is configured to emit ultrasonic energy toward an anatomical structure (e.g., heart) of the patient P. The ultrasonic energy is reflected by the patient's vascular system and / or tissue structure, whereby the ultrasonic transducer elements in the imaging end effector 140 receive the reflected ultrasonic echo signals. In some embodiments, the TEE probe 130 may include an internal or integrated processing component that is capable of locally processing the ultrasonic echo signals to generate image signals representing the anatomical structure of the patient P being imaged. In practice, (one or more) ultrasonic transducer elements may be arranged to provide a two-dimensional (2D) image or a three-dimensional (3D) image of the anatomical structure of the patient P. As described in greater detail herein, images acquired by TEE probe 130 depend on the insertion depth, rotation, and / or tilt of imaging end effector 140 .

[0073] The handle 132 is coupled to the proximal end of the member 131. The handle 132 includes control elements for navigating the imaging end effector 140 to a target pose. As shown, the handle 132 includes knobs 133 and 134 and a switch 135. The knob 133 bends the member 131 and the imaging end effector 140 along the anterior-posterior plane of the patient P (e.g., the heart). The knob 134 bends the member 131 and the imaging end effector 140 along the left-right plane of the patient P. The switch 135 controls the beamforming performed at the imaging end effector 140 (e.g., adjusting the angle of the imaging plane).

[0074] In a manual navigation embodiment, the practitioner manually turns knobs 133 and 134 and / or manually opens and / or closes switch 135 as needed to navigate imaging end effector 140 to a target pose. The practitioner can receive a display of the image generated by imaging end effector 140 and thereby apply navigation force 33 ( FIG. 1 ) to control knobs 133 and 134 and / or switch 135 on handle 132.

[0075] In an automated navigation embodiment, the robotic system (not shown) may include electrical and / or mechanical components (e.g., motors, rollers, and gears) configured to toggle knobs 133 and 134 and / or open and / or close switches 135, whereby the robotic controller 100 may receive motion control commands 31 ( FIG. 1 ) from a navigation controller (not shown) or an input device (not shown) to control the knobs 133 and 134 and / or switches 135 on the handle 132. Alternatively, the robotic controller 100 may be configured to directly manipulate the TEE probe 130 based on a guidance method implemented by the robotic controller 100 via actuation signals 32 ( FIG. 1 ).

[0076] The TEE probe 130 can be steered in various degrees of freedom. 3C-3F illustrate various mechanisms for steerage of the TEE probe 130.

[0077] 3C is a schematic diagram illustrating that the TEE probe 130 can be manually advanced into the patient's esophagus as indicated by arrow 131b or withdrawn from the patient's esophagus as indicated by arrow 131c. The TEE probe 130 can be manually or automatically rotated to the left (e.g., counterclockwise) or to the right (e.g., clockwise) relative to the longitudinal axis 131a of the TEE probe 130 as indicated by arrows 139a and 139b, respectively. The rotation of the member 131 can be described by a parameter denoted as γ.

[0078] 3D is a schematic diagram illustrating electronically rotating the TEE probe 130 from 0 to 180 degrees (e.g., for beamforming) as indicated by arrows 136a and 136b, e.g., by manual or robotic control of a switch 135 on the handle 132. The rotation of the imaging plane can be described by a parameter denoted as ω.

[0079] 3E is a schematic diagram illustrating bending of TEE probe 130, as indicated by arrows 137a and 137b, along an anteroposterior plane, for example relative to the patient's heart, for example by manual or robotic toggling of knob 134 on handle 132. Bending along the anteroposterior plane can be described by a parameter denoted α.

[0080] 3F is a schematic diagram illustrating bending of TEE probe 130 along a left-right plane, for example relative to the patient's heart, as indicated by arrows 138a and 138b, for example by manually or robotically turning knob 133 on handle 132. Bending along the left-right plane can be described by a parameter denoted β.

[0081] By way of further example of an exemplary embodiment of the interventional device 30 ( FIG. 1 ), FIG. 3G shows a transthoracic echocardiography (TTE) probe 240 configured to capture ultrasound images of anatomical structures of the patient P from outside the body of the patient P and a robot 230 shown for manually or robotically handling the TTE probe 240 to reposition the TTE probe 240 to a target pose (i.e., a position and / or orientation of the TTE probe 240 relative to the patient P). More specifically, the robot 230 includes a plurality of links 231 coupled to a plurality of joints 232 configured to hold the TTE probe 240 and manipulate the TTE probe 240 on an outer surface of the patient P (e.g., surrounding a chest region for imaging the heart).

[0082] In a manual navigation embodiment, a practitioner manually applies navigation forces 33 ( FIG. 1 ) to the link(s) 231 to translate, rotate, and / or pivot the link(s) 231 of the robot 230 to navigate the imaging TTE probe 240 to a target pose. The practitioner may receive a display of images generated by the TTE probe 240 to use as a basis for controlling the links 231 of the robot 230.

[0083] In an automatic navigation embodiment, the robotic system (not shown) includes electrical components and / or mechanical components (e.g., motors, rollers, and gears) configured to control the links 231 of the guide robot 230, whereby the robotic controller 101 receives motion control commands 32 ( FIG. 1 ) in the form of Cartesian velocity parameters or joint velocity parameters from a navigation controller (not shown) or an input device (not shown), thereby controlling the links 231 of the guide robot 230. Alternatively, the robotic controller 101 may be configured to directly manipulate the TTE probe 240 based on a guidance method implemented by the robotic controller 101 via the actuation signals 32 ( FIG. 1 ).

[0084] By way of another example of an exemplary embodiment of the interventional device 30 ( FIG. 1 ), FIG. 3H shows a shape sensing guidewire 332 embedded or attached to a continuum robot 331 having an end effector 340. The shape sensing guidewire 232 incorporates optical shape sensing (OSS) technology known in the art of the present disclosure. More specifically, the shape sensing controller 103 uses light along a multi-core optical fiber 333 of the guidewire 332 for device positioning and navigation during surgery. The principles involved utilize distributed strain measurements made in optical fibers using characteristic Rayleigh backscattering or controlled grating patterns (e.g., fiber Bragg gratings). In practice, when the robot controller 102 or a practitioner (not shown) navigates the continuum robot 331 within the patient P to position the end effector 340 at a target posture, the shape sensing controller 103 acquires the registration shape of the continuum robot 331 via the shape sensing guidewire 332.

[0085] In order to further promote the understanding of the present disclosure, Figure 2A and Figure 2B The description of the present invention respectively teaches exemplary embodiments of the continuous position controller of the present invention and the continuous positioning state machine of the present invention. Figure 2A and Figure 2B With the description above, a person of ordinary skill in the art of the present disclosure will recognize how to apply the present disclosure to make and use additional embodiments of the continuous position controller of the present disclosure and the continuous positioning state machine of the present disclosure.

[0086] In addition, the TEE probe 130 (FIG. 3A), the robot 230 / TTE probe 240 (FIG. 3G), and the continuum robot 331 (FIG. 3H) are used herein as non-limiting examples of an interventional device 30 including an end effector 40 (FIG. 1) to support the description of various embodiments of the continuous position controller of the present disclosure. Nevertheless, a person of ordinary skill in the art of the present disclosure will recognize how to apply the present disclosure to a variety of additional embodiments of the interventional device 30 including the end effector 40.

[0087] Refer to Figure 1. Figure 2A and Figure 2B , the interventional device 30 of the present disclosure including the end effector 40 and the continuous positioning controller 50 represents a continuous positioning state machine 90 .

[0088] In particular, if Figure 2B As shown, state S92 of the continuous positioning state machine 90 includes navigation of the interventional device 30 including the end effector 40 according to a specific application (e.g., minimally invasive procedures, video-assisted chest surgery, microvascular procedures, micromedical diagnostic procedures, or deformity correction procedures). In practice, the application may involve manual navigation positioning or automatic navigation positioning of the end effector 40 to a target pose, whereby the application combines imaging guidance (e.g., image segmentation, image registration, path planning, etc.), interventional device tracking (e.g., electromagnetic, optical, or shape sensing), and / or any other navigation technology for positioning the end effector 40 to a target pose.

[0089] Execution of state S92 of the continuous positioning state machine 90 results in the generation of navigation data 34 and optional generation of auxiliary data 35. Typically, in practice, the navigation data 34 will be transmitted to / applied to the interventional device 30 in the form of navigation command(s) 31, actuation signal(s) 32, and / or navigation force(s) 33, and the auxiliary data 35 will be in the form of image(s) of the interventional device 30 and / or end effector 40, operating characteristics of the interventional device 30 (e.g., shape, strain, distortion, temperature, etc.), and operating characteristics of the end effector 40 (e.g., posture, force, etc.).

[0090] In response thereto, state S94 of the continuous positioning state machine 90 includes continuous positioning control according to state S92 by the continuous position controller 50 that navigates the interventional tool 30 and the end effector 40. To this end, the continuous position controller 50 uses the forward prediction model 60 of the present disclosure, the inverse prediction model 70 of the present disclosure, and / or the imaging prediction model 80 of the present disclosure.

[0091] In practice, as will be further explained in the present disclosure, the forward prediction model 60 can be any type of machine learning model or equivalent (e.g., a neural network) suitable for use with a particular type of interventional device 30 used in the particular type of application being implemented for regressing positioning motions of the interventional device 30 to a navigation pose of the end effector 40, whereby the forward prediction model 60 is trained on forward kinematics of the interventional device 30 that predicts the navigation pose of the end effector 40.

[0092] In operation, the forward prediction model 60 inputs the navigation data 34 (and auxiliary data 35, if transmitted) associated with the manual navigation or automatic navigation of the interventional device 30, thereby predicting the navigation pose of the end effector 40 corresponding to the navigation of the interventional device 30 and outputting continuous positioning data 51 of information about positioning the end effector 40 to a target pose by the interventional device 30 based on the predicted navigation pose of the end effector 40. The continuous positioning data 51 can be used as a control operation in state S92 to determine the accuracy and / or perform recalibration of the manual navigation or automatic navigation of the interventional device 30 to position the end effector 40 to the target pose.

[0093] In practice, as will be further explained in the present disclosure, the inverse prediction model 70 can be any type of machine learning model or equivalent (e.g., a neural network) suitable for use in a particular type of interventional device 30 for use in the particular type of application being implemented for regressing a target pose of the end effector 40 to the positioning motion of the interventional device 30, whereby the inverse prediction model 60 is trained on the inverse kinematics of the interventional device 30 that predicts the positioning motion of the interventional device 30.

[0094] In operation, the inverse prediction model 70 inputs the navigation data 34 (and auxiliary data 35, if transmitted) associated with the target pose of the end effector 40, thereby predicting the positioning motion of the interventional device 30 for positioning the end effector 40 to the target pose, and the inverse prediction model 70 outputs continuous positioning commands 52 for positioning the end effector 40 to the target pose by the interventional device 30 based on the predicted positioning motion of the interventional device 30. The continuous positioning commands 52 can be used as a control operation in state S92 to perform manual navigation or automatic navigation of the interventional device 30 to position the end effector 40 to the target pose.

[0095] In practice, as will be further explained in the present disclosure, the imaging prediction model 80 can be any type of machine learning model or equivalent (e.g., a neural network or a scale-invariant feature transformation network, whereby the inverse prediction model 60 is trained on the correlation of relative imaging performed by the end effector 40 to the forward kinematics of the interventional device 30 that predicts the navigation pose of the end effector 40) suitable for the particular type of end effector 40 used in the particular type of application being implemented.

[0096] In operation, the imaging prediction model 60 inputs auxiliary data 35 in the form of images generated by the end effector 40 in one or more poses, thereby predicting the navigation pose of the end effector 40 as feedback data for information about corrective positioning of the end effector 40 to a target pose by the interventional device 30. The feedback data is used in a closed loop of state S94 to generate a difference between the target pose of the end effector 40 and the predicted navigation pose of the end effector 40, whereby the inverse prediction model 70 can input the difference to predict the corrective positioning motion of the interventional device 30 to reposition the end effector 30 to the target pose.

[0097] In practice, embodiments of the continuous positioning controller 50 may use the forward prediction model 60 , the inverse prediction model 70 , and / or the imaging prediction model 80 .

[0098] For example, an embodiment of the continuous positioning controller 50 may use only the forward prediction model 60 to facilitate displaying the accuracy of manual or automated navigation of the interventional device 30 to position the end effector 40 to a target pose.

[0099] More specifically, a user interface is provided to display an image of the attempted navigation of the end effector 40 to the target pose and an image of the predicted navigation pose of the end effector 40 obtained by the forward prediction model 60. The confidence ratio of the prediction is shown to the user. To evaluate the prediction uncertainty, multiple feed-forward iterations of the forward prediction model 60 are performed with dropout randomly enabled, as is known in the art of the present disclosure.

[0100] As another example, an embodiment of the continuous positioning controller 50 may use only the inverse prediction model 70 to command manual or automated navigation of the interventional device 30 to position the end effector 40 to a target pose.

[0101] As another example, an embodiment of the continuous positioning controller 50 may use only the imaging prediction model 80 to provide feedback data that informs the operation of the interventional device 30 to correct the positioning of the end effector 40 to a target pose.

[0102] As another example, an embodiment of the continuous positioning controller 50 may use the forward prediction model 60 and the inverse prediction model 70 to command manual navigation or automatic navigation of the interventional device 30 to position the end effector 40 to a target pose and display the accuracy of the manual navigation or automatic navigation of the interventional device 30 to position the end effector 40 to a target pose.

[0103] As another example, an embodiment of the continuous positioning controller 50 may use the inverse prediction model 70 and the imaging prediction model 80 to command manual or automatic navigation of the interventional device 30 to position the end effector 40 to a target posture and provide feedback data that provides information for the following operations: the interventional device 30 corrects the positioning of the end effector 40 to the target posture.

[0104] To further facilitate understanding of the present disclosure, the following description of FIGS. 4-14 teaches exemplary embodiments of the forward prediction model, the reverse prediction model, and the imaging prediction model of the present disclosure. Based on the description of FIGS. 4-14, a person of ordinary skill in the art of the present disclosure will recognize how to apply the present disclosure to make and use additional embodiments of the forward prediction model, the reverse prediction model, and the imaging prediction model of the present disclosure.

[0105] Figures 4A-4E The diagram illustrates the training and application of a forward prediction model 60a of the present disclosure, which is trained on the forward kinematics of an interventional device 30 (FIG. 1) that predicts the navigation posture of an end effector 40 (FIG. 1), thereby facilitating the application of the forward prediction model 60a to command positioning motions of the interventional device 30 during the interventional procedure to present the predicted navigation posture of the end effector 40, whereby a continuous positioning controller 50 (FIG. 1) can generate continuous positioning data 51 of information about positioning the end effector 40 to a target posture by the interventional device 30 based on the predicted navigation posture of the end effector 40.

[0106] More specifically, refer to Figure 4A The training phase of the forward prediction model 60a involves a training controller (not shown) configured to interpret a ground truth training data set D as exemplarily taught in the subsequent description of FIGS. 16-18 of the present disclosure. The data set D consists of a 2-tuple d i =(T i , Q i ) represents n sequences W of i data points. The 2-tuple consists of the end effector posture (T∈SE(3)) 62a and the sequence of j consecutive joint variables (Q∈(q t ,q t+1 ...q t+j ))61a. Entry q t represents all joint variables controlled by the robot controller (not shown).

[0107] In practice, the training dataset D is a collection of expert data that reasonably covers different navigations of the interventional device 30. To this end, the diverse dataset training dataset D used for learning should include manufacturing differences between various types of robots, performance characteristics, wear of hardware components, and other system-related factors and system-irrelevant factors (e.g., temperature or humidity of the environment in which the robot operates).

[0108] refer to Figure 4B , the application phase of the forward prediction model 60a involves the continuous positioning controller 50 using the feedforward prediction model 60a to execute a deep learning type algorithm to regress the command position motion of the interventional device 30 to the navigation pose of the end effector 40. The forward prediction model 60a is configured to infer the pose of the end effector 40 given a sequence Q of j consecutive joint variables 61b of the interventional device 30 (e.g., parameters α, β as shown in FIG. 3B ) 62b.

[0109] In such Figure 4E In one embodiment shown, the forward prediction model 60a uses a neural network library 160a, which includes an input layer, a hidden layer, and an output layer, which are derived from a combination of one or more fully connected layers (FCL) 163a, one or more convolutional layers (CNL) 164a, one or more recurrent layers (RCL) 165a, and one or more long short-term memory (LSTM) layers 166a.

[0110] In practice, the combination of these layers is configured to implement the joint variables Q to pose The return of.

[0111] In order to implement the joint variable Q to pose In one embodiment of regression, the neural network library 160a includes a set of N fully connected layers 163a.

[0112] In order to implement the joint variable Q to pose In the second embodiment of regression, the neural network library 160a includes a set of N convolutional layers 164a, followed by a set of M fully connected layers 163a or a set of W recurrent layers 165a or a set of W long short-term memory layers 166a.

[0113] In order to implement the joint variable Q to pose In the third embodiment of regression, the neural network library 160a includes a set of N convolutional layers 164a, followed by a set of M fully connected layers 163a in combination with a set of W recurrent layers 165a or a set of W long short-term memory layers 166a.

[0114] In practice, the fully connected layer 163a may include K neurons, where N, M, W, K may be any positive integers and the values ​​may vary according to the embodiment. For example, N may be about 8, M may be about 2, W may be about 2, and K may be about 1000. Moreover, the convolutional layer 164a may implement a nonlinear transformation, which may be a composite function of operations (e.g., batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 164a may also include a nonlinear function configured to extract a rectified feature map (e.g., including a rectified nonlinear ReLU operation).

[0115] In addition, in practice, one of the layers 163a or 164a acts as an input layer for inputting a sequence 161a of joint variables Q, whereby the size of the sequence of joint variables Q can be ≥ 1, and one of the layers 163a, 165a, and 166a can act as an output layer for outputting a posture 162a of the end effector 40 in Cartesian space (e.g., the translation and rotation of the end effector 40 in Cartesian space). The output posture of the end effector 40 in Cartesian space can be represented as a vector parameterization and / or non-vector parameterization of the position and orientation of the rigid body. More specifically, the parameterization can be in the form of Euler angles, quaternions, matrices, exponential mappings, and / or angular axes representing rotations and / or translations (e.g., including the direction and magnitude of the translation).

[0116] Moreover, in practice, the output layer may be a non-linear fully connected layer 163a that gradually downscales the high-dimensional output of the last convolutional layer 164a of the neural network bank 160a to produce a set of output variables.

[0117] During training, we forward predict the output of the model (given an input sequence Q) by Compared with the ground-truth end-effector pose T from a batch of training datasets D i The training weights of the forward prediction model 60a are continuously updated by comparing the weights of the filter (which can be systematically selected or randomly selected from a data storage (not shown)). More specifically, the coefficients of the filter can be initialized with predefined values ​​or arbitrary values. The coefficients of the filter are applied to the batch of training data sets D via forward propagation and are adjusted via back propagation to minimize any output error.

[0118] In application, the forward prediction model 60a infers the pose of the end effector 40 given a sequence Q of j consecutive joint variables 62a. 62a.

[0119] Still reference Figure 4EDepending on the complexity of the task, the neural architecture of an exemplary embodiment of the forward prediction model 60a has a unique number of layers, which is defined as predicting the pose of the end effector 40 given a sequence Q of j consecutive joint variables 61a. The loss function for training this neural architecture can be defined as the sum of the Euclidean distances between the translation component and the rotation component, e.g.

[0120] refer to Figure 4C and Figure 4D , stage S92a of the exemplary interventional procedure 90a uses a TEE probe 130 (FIG. 3A) and a robotic controller 100 (FIG. 3B), and includes the robotic controller 100 receiving the position of the TEE probe 130 as a joint variable and sending a movement signal to a handle 132 (FIG. 3A) of the TEE probe 130. By using a motorized knob, the handle 132 tightens / relaxes the tether of the TEE probe 130, which will cause movement of the end effector 140 (FIGs. 3C-3F) to a target pose. In practice, the position of the TEE probe 130 as a joint variable can be indicated by a user or an external tracking device or guidance system.

[0121] Stage S94a of process 90a involves the robot controller 100, the forward prediction model 50a, and the display controller 104. The robot controller 100 stores the sequence (Q) of successive joint variables 61a and transmits it to the predicted navigation pose of the end effector 140. The continuous positioning controller 50a generates a confidence ratio of the prediction derived from the uncertainty of multiple feed-forward iterations of the forward prediction model 60a, which are performed with randomly enabled dropout as is known in the art of the present disclosure. The forward prediction model 50a will include the predicted navigation pose of the end effector 140 The continuous positioning data 51a and the confidence ratio are transmitted to the display controller 104, which in turn controls the display of the image 105a of the navigation posture of the end effector 140, the image 106a of the navigation posture of the end effector 140 and the confidence ratio for the guidance purpose of guiding the end effector 140 to the target posture.

[0122] Figures 5A-5E 1 illustrates the training and application of an inverse prediction model 70a of the present disclosure, which is trained on the inverse kinematics of an interventional device 30 (FIG. 1) that predicts the positioning motion of the interventional device 30, thereby facilitating the application of the inverse prediction model 70a to a target pose (T) 71a of the end effector 40 to render the joint variable motion (q) 72 of the interventional device 30 during an interventional procedure.t , whereby the continuous positioning controller 50 ( FIG. 1 ) may generate continuous positioning commands for controlling the interventional device 30 to reposition the end effector 40 to a target pose based on the predicted positioning motion of the interventional device 40 .

[0123] More specifically, refer to Figure 5A The training phase of the inverse prediction model 70a involves a training controller (not shown) being configured as in Figure 17-18 The actual training data set D is interpreted as exemplarily taught in the description of FIG. The data set D consists of a 2-tuple d i =(T i , Q i ) represents n sequences W of i data points. This 2-tuple consists of the end effector posture (T∈SE(3))71a and the sequence of j consecutive joint variables collected at sequential time points from t to t+j (Q∈(q t ,q t+1 ...q t+j ))72a. Entry q t represents all joint variables controlled by the robot controller (not shown). The variable j can also be equal to 1, which means that the prediction model will be trained to infer a single set of joint variables.

[0124] In practice, the training dataset D is a collection of expert data that reasonably covers different navigations of the interventional device 30. To this end, the diverse dataset training dataset D used for learning should include mechanical differences between various types of robots, wear of hardware components, and other system-related factors.

[0125] refer to Figure 5B The application phase of the inverse prediction model 70a involves the continuous positioning controller 50 executing a deep learning type algorithm using the inverse prediction model 70a to perform motion regression. The inverse prediction model 70a is configured to infer j successive joint variables 72 t (e.g., the parameters α, β as shown in FIG. 3B ) To achieve the posture (T∈SE(3)) 71t of the end effector 40.

[0126] In such Figure 5E In one embodiment shown, the inverse prediction model 70a uses a neural network library 170a, which includes an input layer, a hidden layer, and an output layer, which are derived from a combination of one or more fully connected layers (FCL) 173a, one or more convolutional layers (CNL) 174a, one or more recurrent layers (RCL) 175a, and one or more long short-term memory (LSTM) layers 176a.

[0127] In practice, the combination of these layers is configured to implement the pose T into joint variables The return of.

[0128] In the implementation of posture T to joint variables In one embodiment of regression, the neural network library 170a includes a set of N fully connected layers 173a.

[0129] In the implementation of posture T to joint variables In the second embodiment of regression, the neural network library 170a includes a set of N convolutional layers 174a, followed by a set of M fully connected layers 173a or a set of W recurrent layers 175a or a set of W long short-term memory layers 176a.

[0130] In the implementation of posture T to joint variables In the third embodiment of regression, the neural network library 170a includes a set of N convolutional layers 174a, followed by a set of M fully connected layers 173a in combination with a set of W recurrent layers 175a or a set of W long short-term memory layers 176a.

[0131] In practice, the fully connected layer 173a may include K neurons, where N, M, W, K may be any positive integers and the values ​​may vary according to the embodiment. For example, N may be about 8, M may be about 2, W may be about 2, and K may be about 1000. Moreover, the convolutional layer 174a may implement a nonlinear transformation, which may be a composite function of operations (e.g., batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 174a may also include a nonlinear function configured to extract a rectified feature map (e.g., including a rectified nonlinear ReLU operation).

[0132] In addition, in practice, one of the layers 173a or 174a serves as an input layer for inputting a posture 171a of the end effector 40 in Cartesian space (e.g., a translation and rotation of the end effector 40 in Cartesian space), and one of the layers 173a, 175a, and 176a may serve as an output layer for outputting a sequence 172a of joint variables Q, whereby the size of the sequence of joint variables Q may be ≥ 1. The input posture of the end effector 40 in Cartesian space may be represented as a vector parameterization and / or a non-vector parameterization of the position and orientation of the rigid body. More specifically, the parameterization may be in the form of Euler angles, quaternions, matrices, exponential mappings, and / or angular axes representing rotations and / or translations (e.g., including the direction and magnitude of the translation).

[0133] Moreover, in practice, the output layer can be a non-linear fully connected layer 173a, which gradually reduces the high-dimensional output of the last convolutional layer 174a of the neural network bank 170a to produce a set of output variables.

[0134] During training, the output from the inverse prediction model is obtained by (given the ground-truth end-effector pose T as input) and the true sequence Q from a batch of training dataset D i The training weights of the inverse prediction model 70a are continuously updated by comparing the filter coefficients (which can be systematically selected or randomly selected from a data storage (not shown)). More specifically, the coefficients of the filter can be initialized with predefined values ​​or arbitrary values. The coefficients of the filter are applied to the batch of training data sets D via forward propagation and are adjusted via back propagation to minimize any output error.

[0135] In application, the inverse prediction model 70a infers a sequence of j consecutive joint variables 72a (e.g., parameters α, β as shown in FIG. 3B ) To achieve the provided posture (T∈SE(3)) 71b of the end effector 40.

[0136] Still reference Figure 5E Depending on the complexity of the task, the neural architecture of the exemplary embodiment of the inverse prediction model 70a shown in the figure has a unique number of layers, which are defined as a sequence of predicting j consecutive joint variables 72b (e.g., parameters α, β as shown in FIG. 3B ) The pose (T∈SE(3)) of the end effector 40 is obtained as specified in the output layer 179a of 71b. The mean square error (MSE) can be used as a loss function.

[0137] refer to Figure 5C and Figure 5D , stage S92b of the exemplary interventional procedure 90b uses the TEE probe 130 (FIG. 3A) and the navigation controller 103, and includes the navigation controller 103 determining the target pose of the end effector 140 and sending the target pose 71a to the inverse prediction model 70a. In practice, the navigation controller 130 may implement any known guidance algorithm known in the art of the present disclosure.

[0138] Stage S94A of process 90b involves the inverse prediction model 70a and the robot controller 100. The inverse prediction model 70a infers a sequence of j successive joint variables 72a (eg, parameters α, β as shown in FIG. 3B ) To achieve the posture (T∈SE(3)) 71a of the end effector 40, and the continuous positioning controller 50b communicates with the command controller 104 to control the interventional device 30 via the end effector 40 to position the end effector 50 to the target posture based on the predicted positioning movement of the interventional device 40.

[0139] Figures 6A-6E The diagram illustrates the training and application of a forward prediction model 60b of the present disclosure, which is trained on the forward kinematics of an interventional device 30 (FIG. 1) that predicts the navigation posture of an end effector 40 (FIG. 1), thereby facilitating the application of the forward prediction model 60a to a command n vector of the joint velocities of the interventional device 30 to present a predicted linear velocity and / or angular velocity of the end effector 40 during the interventional procedure, whereby a continuous positioning controller 50 (FIG. 1) can generate continuous positioning data 51 of information regarding repositioning of the end effector 40 by the interventional device 30 to a target posture based on the predicted linear velocity and / or angular velocity of the end effector 40.

[0140] More specifically, refer to Fig. 6A The training phase of the forward prediction model 60b involves a training controller (not shown) configured to interpret a ground truth training dataset D as exemplarily taught in FIGS. 16-18. The dataset D consists of a 2-tuple The 2-tuple consists of a command sequence of j consecutive joint velocities 61b and the linear velocity and / or angular velocity 62b of the end effector Composition. Item Represents all joint variables controlled by the robot controller (not shown). A sequence may also contain only one entry.

[0141] In practice, the training dataset D is a collection of expert data that reasonably covers different navigations of the interventional device 30. To this end, the diverse dataset training dataset D used for learning should include mechanical differences between various types of robots, wear of hardware components, and other system-related factors.

[0142] refer to Figure 6B The application phase of the forward prediction model 60b involves the continuous positioning controller 50 executing a deep learning type algorithm for end effector motion regression using the feedforward prediction model 60b. The forward prediction model 60b is configured to infer the linear velocity and / or angular velocity 62b of the end effector 40 given the n-vector of joint velocities 61b of the interventional device 30.

[0143] In such Fig. 6EIn one embodiment shown, the forward prediction model 60b uses a neural network library 160b, which includes an input layer, a hidden layer, and an output layer, which are derived from a combination of one or more fully connected layers (FCL) 163b, one or more convolutional layers (CNL) 164b, one or more recurrent layers (RCL) 165b, and one or more long short-term memory (LSTM) layers 166b.

[0144] In practice, the combination of these layers is configured to implement a regression of the joint velocities of the interventional device 30 to the linear velocity and / or angular velocity of the end effector 40 .

[0145] In one embodiment for implementing the regression of the joint velocity of the interventional device 30 to the linear velocity and / or angular velocity of the end effector 40, the neural network library 160b includes a set of N number of fully connected layers 163b.

[0146] In a second embodiment for implementing the regression of the joint velocity of the interventional device 30 to the linear velocity and / or angular velocity of the end effector 40, the neural network library 160b includes a set of N convolutional layers 164b followed by a set of M fully connected layers 163b or a set of W recurrent layers 165b or a set of W long short-term memory layers 166b.

[0147] In a third embodiment for implementing the regression of the joint velocity of the interventional device 30 to the linear velocity and / or angular velocity of the end effector 40, the neural network library 160b includes a set of N convolutional layers 164b, followed by a set of M fully connected layers 163b in combination with a set of W recurrent layers 165b or a set of W long short-term memory layers 166b.

[0148] In practice, the fully connected layer 163b may include K neurons, where N, M, W, K may be any positive integers and the values ​​may vary according to the embodiment. For example, N may be about 8, M may be about 2, W may be about 2, and K may be about 1000. Moreover, the convolutional layer 164b may implement a nonlinear transformation, which may be a composite function of operations (e.g., batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 164b may also include a nonlinear function configured to extract a rectified feature map (e.g., including a rectified nonlinear ReLU operation).

[0149] Additionally, in practice, one of the layers 163b or 164b serves as a sequence for inputting j consecutive joint velocities The input layer of j consecutive joint velocities The size of can be ≥ 1, and one of the layers 163b, 165b and 166b can serve as an output layer for outputting the linear velocity and angular velocity of the end effector regressed according to the last fully connected layer (e.g., 6 units, 3 units for linear velocity, and 3 units for angular velocity) using a linear or nonlinear activation function

[0150] During training, the predicted linear and angular velocities are obtained by passing them through the forward velocity prediction model (given a sequence of joint velocities) The training weights of the forward prediction model 60b are continuously updated by comparing with linear velocities and / or angular velocities 62b from a batch of training data sets D (which may be systematically selected or randomly selected from a data memory (not shown)). More specifically, the coefficients of the filter may be initialized with predefined values ​​or arbitrary values. The coefficients of the filter are applied to the batch of training data sets D via forward propagation and are adjusted via back propagation to minimize any output error.

[0151] In application, the forward prediction model 60 b infers the linear velocity and / or angular velocity 62 b of the end effector 40 given a sequence of joint velocities 61 b ​​of the interventional device 30 .

[0152] Still reference Fig. 6E , the neural architecture consists of an input, the previously described neural network library, and an output. The input is a sequence of joint velocities, and the output is the linear and angular velocities that can be regressed according to a fully connected layer with 6 units. The loss function can be the MSE listed below:

[0153]

[0154] refer to Figure 6C and Fig.6D , stage S92c of the exemplary interventional procedure 90c uses the robot 230 (FIG. 3A), the TTE probe 240 (FIG. 3A), and the robot controller 101 (FIG. 3G), and includes the robot controller 101 receiving the position of the TTE probe 240 as an n-vector of the joint velocities 61b of the interventional device 30 and sending the n-vector of the joint velocities 61b of the interventional device 30 to the robot 230. In practice, the position of the TTE probe 240 as the n-vector of the joint velocities 61b of the interventional device 30 may be indicated by a user or an external tracking device or guidance system.

[0155] Stage S94c of process 90c involves the robotic controller 101, the forward prediction model 50a, and the display controller 104. The robotic controller 101 stores the n-vector of the joint velocities 61b of the interventional device 30 and transmits it to the forward prediction model 60b that predicts the linear velocity and / or angular velocity 62b of the TTE probe 240. The continuous positioning controller 50c generates a confidence ratio of the prediction derived from the uncertainty of multiple feed-forward iterations of the forward prediction model 60b, which are performed with randomly enabled dropout as is known in the art of the present disclosure. The forward prediction model 60b transmits continuous positioning data 51b to the display controller 104, the continuous positioning data 51b including the predicted navigation pose of the TTE probe 240 derived from the predicted linear velocity and / or angular velocity 62b of the TTE probe 240. And also including the confidence ratio, the display controller 104 then controls the display of the image 105 a of the navigation posture of the TTE probe 240 , the image 106 a of the navigation posture of the TTE probe 240 , and the confidence ratio for the guidance purpose of guiding the end effector 240 to the target posture.

[0156] Figures 7A-7E The diagram illustrates the training and application of the inverse prediction model 70b of the present disclosure, which is trained on the inverse kinematics of the interventional device 30 (Figure 1) that predicts the positioning motion of the interventional device 30, thereby facilitating the application of the inverse prediction model 70b to the target posture of the end effector 40 during the interventional process to present the predicted positioning motion of the interventional device 30, whereby the continuous positioning controller 50 (Figure 1) can generate continuous positioning commands that control the interventional device 30 to reposition the end effector 40 to the target posture based on the predicted positioning motion of the interventional device 40.

[0157] More specifically, refer to Fig. 7A The training phase of the inverse prediction model 70b involves a training controller (not shown) being configured as in Figure 17-18 The actual training data set D is interpreted as exemplarily taught in the description of FIG. The data set D consists of two tuples The 2-tuple consists of the linear velocity and / or angular velocity of the end effector and the successive joint variables collected at sequential time points starting from t to t+j. The sequence of entries represents all joint variables controlled by the robot controller (not shown).

[0158] In practice, the training dataset D is a collection of expert data that reasonably covers different navigations of the interventional device 30. To this end, the diverse dataset training dataset D used for learning should include mechanical differences between various types of robots, wear of hardware components, and other system-related factors.

[0159] refer to Figure 7B The application phase of the inverse prediction model 70b involves the continuous positioning controller 50 executing a deep learning type algorithm for joint velocity regression using the inverse prediction model 70b. The inverse prediction model 70b is configured to infer the n-vector of the joint velocities 61b of the interventional device 30 given the linear velocity and / or angular velocity 62b of the end effector 40.

[0160] In such Fig. 7E In one embodiment shown, the inverse prediction model 70b uses a neural network library 170b, which includes an input layer, a hidden layer, and an output layer, which are derived from a combination of one or more fully connected layers (FCL) 173b, one or more convolutional layers (CNL) 174b, one or more recurrent layers (RCL) 175b, and one or more long short-term memory (LSTM) layers 176b.

[0161] In practice, the combination of these layers is configured to implement a regression of the linear velocity and / or angular velocity of the end effector 40 to the joint velocities of the interventional device 30 .

[0162] In one embodiment for implementing the regression of the linear velocity and / or angular velocity of the end effector 40 to the joint velocities of the interventional device 30, the neural network library 170b includes a set of N number of fully connected layers 173b.

[0163] In a second embodiment for implementing the regression of the linear velocity and / or angular velocity of the end effector 40 to the joint velocity of the interventional device 30, the neural network library 170b includes a set of N convolutional layers 174b followed by a set of M fully connected layers 173b or a set of W recurrent layers 175b or a set of W long short-term memory layers 176b.

[0164] In a third embodiment for implementing the regression of the linear velocity and / or angular velocity of the end effector 40 to the joint velocity of the interventional device 30, the neural network library 170b includes a set of N convolutional layers 174b, followed by a combination of a set of M fully connected layers 173b and a set of W recurrent layers 175b or a set of W long short-term memory layers 176b.

[0165] In practice, the fully connected layer 173b may include K neurons, where N, M, W, K may be any positive integers and the values ​​may vary depending on the embodiment. For example, N may be about 8, M may be about 2, W may be about 2, and K may be about 1000. Moreover, the convolutional layer 174b may implement a nonlinear transformation, which may be a composite function of operations (e.g., batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 174b may also include a nonlinear function configured to extract a rectified feature map (e.g., including a rectified nonlinear ReLU operation).

[0166] Additionally, in practice, one of the layers 173b or 174b acts as a buffer for inputting angular velocity and linear velocity. The input layer of LSTM layer 173b, 175b and 176b may serve as an output layer for outputting a sequence of j consecutive joint velocities provided as an output from LSTM layer 176b. Alternatively, individual joint velocities may be regressed from a fully connected layer 173b consisting of m units, one for each joint in the robot controlled by the robot controller. The fully connected layer 173b may have a linear or non-linear activation function.

[0167] During training, we pass (the linear velocity and angular velocity at a given input The predicted sequence of joint velocities The actual sequence of joint velocities from a batch of training datasets D The training weights of the inverse prediction model 70b are continuously updated by comparing the filter coefficients (which can be systematically selected or randomly selected from a data storage (not shown)). More specifically, the coefficients of the filter can be initialized with predefined values ​​or arbitrary values. The coefficients of the filter are applied to the batch of training data sets D via forward propagation and are adjusted via back propagation to minimize any output error.

[0168] In application, the inverse prediction model 70 b infers the n-vector of the joint velocities 61 b ​​of the interventional device 30 given the linear and / or angular velocities 62 b of the end effector 40 .

[0169] Still reference Fig. 7E The neural architecture of an exemplary embodiment of the inverse velocity model consists of an input, a neural network library, and an output. The input is the angular velocity and the linear velocity And the output is a sequence of joint velocities provided as output from the LSTM layer Alternatively, individual joint velocities can be regressed from a fully connected layer consisting of m units, one for each joint in the robot controlled by the robot controller. The fully connected layer can have linear or nonlinear activation functions.

[0170] refer to Figure 7C and Fig.7D , stage S92d of the exemplary interventional procedure 90d uses the TTE probe 240 (FIG. 3G) and the navigation controller 103, and includes the navigation controller 103 determining the linear velocity and / or angular velocity 62b of the TTE probe 240 to the target pose and sending the linear velocity and / or angular velocity 62b to the inverse prediction model 70b. In practice, the navigation controller 130 may implement any known guidance algorithm known in the art of the present disclosure.

[0171] Stage S94d of process 90d involves the inverse prediction model 70b and the robot controller 101. The inverse prediction model 70b infers the n-vector of the joint velocity 61b of the interventional device 30 given the linear velocity and / or angular velocity 62b, and the continuous positioning controller 50c transmits the continuous positioning command 52b to the robot controller 101, thereby controlling the positioning of the TTE probe 240 to the target pose via the robot 230 (FIG. 3G) based on the predicted positioning motion of the interventional device 40.

[0172] In practice, the forward prediction model 60a ( Figure 4A ), reverse prediction model 70a( Figure 5A ), forward prediction model 60b( Fig. 6A ) and the reverse prediction model 70b( Fig. 7A ) can use additional auxiliary information, such as images of the anatomical structure (e.g., ultrasound images, endoscopic images, or X-ray images), forces measured at the end effector, and the shape of the robot. Depending on the application, other inputs may also be included, including information from spectral tissue sensing devices, ECG signals or EEG signals, tissue conductivity, or other physiological signals. For example, if a continuum robot is operating inside a human heart, features available in ultrasound images as well as electrophysiological signals can improve the positioning of the end effector relative to the anatomical structure, thereby improving the guidance accuracy.

[0173] refer to Fig. 8A and Figure 8BIn addition to being shown as being trained on the forward kinematics of the interventional device, the forward prediction model 60c is shown as being trained on the sequence Q of joint variables 61a and the end effector pose T 62a. In addition, the forward prediction model 60c is shown as being trained on the shape 35a, image 35b and force 35c of the interventional device. Therefore, in application, the forward prediction model 60c will be able to predict the navigation pose of the end effector based on the sequence Q of joint variables 61a and the shape 35a, image 35b and force 35c of the interventional device.

[0174] A person of ordinary skill in the art will know how to apply the shape 35a, image 35b and force 35c of the interventional device and any other additional auxiliary information to the inverse prediction model 70a, the forward prediction model 60b and the inverse prediction model 70b.

[0175] Figures 9A-9E The diagram illustrates the training and application of the forward prediction model 60d of the present disclosure, which is trained on the forward kinematics of the interventional device 30 (Figure 1) that predicts the navigation pose of the end effector 40 (Figure 1) and the shape of the robot, thereby facilitating the application of the forward prediction model 60d to a sequence of successive shapes of the interventional device 30 with embedded OSS technology during the interventional process to present the predicted navigation pose and shape of the end effector 40, whereby the continuous positioning controller 50 (Figure 1) can generate continuous positioning data 51c of information about repositioning the end effector 40 to a target pose by the interventional device 30 based on the predicted navigation pose of the end effector 40.

[0176] More specifically, refer to Fig.9A In the training phase, a training controller (not shown) is configured to interpret a real-world training data set D as exemplarily taught in the description of FIGS. 16-18 . The data set consists of a 2-tuple d i =(H i , H i+1 ) represents n sequences W of i data points. This 2-tuple consists of a sequence of k consecutive shapes 61d (H i ∈(h t ,h t+ 1...h t+k ), where h∈(p1...p m ) is a set of m vectors p m , which describes both the position of the OSS sensor (e.g., shape sensing guidewire) embedded in the interventional device 30 in 3D Euclidean space and auxiliary shape parameters (e.g., strain, curvature, and twist). This 2-tuple is also composed of a future time point h t+k+1 A sequence H of k consecutive shapes 62b i+1 ∈(ht+1 ,h t+2 …h t+k+1 ), where hv∈(p1...p m ) is a set of m vectors p m , which describes both the position of the OSS interventional device 30 in 3D Euclidean space and the auxiliary shape parameters (eg, strain, curvature and distortion).

[0177] In practice, the training dataset D is a collection of expert data that reasonably covers different navigations of the OSS interventional device 30. To this end, the diverse dataset training dataset D for learning should include anatomical structures with different curvatures, motion amplitudes, mechanical differences between various types of robots, wear of hardware components, and other system-related factors (e.g., temperature and humidity of the environment).

[0178] refer to Fig. 9B , the application phase of the forward prediction model 60d involves the continuous positioning controller 50d using the forward prediction model 60d to perform a deep learning type algorithm, where the recurrent layers are trained on expert data that reasonably covers different situations. The diverse data set used for learning includes various operating conditions (temperature, fiber bending, etc.), different operating motions of the equipment, and differences in hardware (fiber, interrogator, etc.).

[0179] In such Fig.9E In one embodiment shown, the forward prediction model 60d uses a neural architecture that sequentially includes an input layer 163a, a sequence-to-sequence model 263a, an output layer 262a, and an extraction layer 264. The neural architecture is a deep convolutional neural network with a recurrent layer configured to infer a future sequence consisting of k shapes. So using the last shape in the prediction sequence To estimate the position of the OSS intervention device 30 at a future time point.

[0180] In training, by (given an input sequence H i The sequence of future shapes predicted by the model with the true future sequence H from the training dataset D i+1 The training weights of the forward prediction model 60d are continuously updated by comparing the weights of the filter (which can be systematically selected or randomly selected from a data storage (not shown)). More specifically, the coefficients of the filter can be initialized with predefined values ​​or arbitrary values. The coefficients of the filter are applied to the batch of training data sets D via forward propagation and are adjusted via back propagation to minimize any output error.

[0181] In the application, the forward prediction model 60d infers a future sequence consisting of k shapes So using the last shape in the prediction sequence To estimate the position of the OSS intervention device 30 at a future time point.

[0182] In such Fig.9F In the alternative embodiment shown, the forward prediction model 60d uses a many-to-one model 263b instead of a sequence-to-sequence model 263a, whereby the last layer will be the last shape

[0183] refer to Fig. 9C and Fig.9D , stage S92e of the exemplary interventional procedure 90e uses the OSS guidewire 332 (FIG. 3F) and the shape sensing controller 103 (FIG. 3F), and includes the shape sensing controller 103 measuring and storing the shape of the OSS guidewire 332 during navigation of the end effector 340 to a target pose.

[0184] Stage S94e of process 90e involves shape sensing controller 103, forward prediction model 50d and display controller 104. Shape sensing controller 103 transmits sequence 61d of k consecutive shapes to forward prediction model 60e, thereby inferring the next shape sequence Among them, the last shape is the predicted position of the OSS guidewire 332. The display controller 104 controls the display of the sensed position image 105a of the OSS guidewire 332 for the purpose of guiding the end effector 340 to a target posture.

[0185] refer to Fig.11A and Fig. 11B , the imaging prediction model 80a of the present disclosure is trained on expert data that reasonably covers different situations. Given an ultrasound image 81a, the neural network 80a infers the relative position 82a of the image 81a relative to a reference anatomical structure. As a result, as will be further described herein, the motion between the previous pose of the end effector in the anatomical structure (e.g., the heart) and the current pose of the end effector in the anatomical structure (e.g., the heart) can be calculated.

[0186] Fig.11A Batch training of the imaging prediction model 80a is shown. During training, the network continuously updates the weights of the imaging prediction model 80a using 2-tuples from a ground-truth dataset consisting of an ultrasound image and the relative position of the image with respect to a reference anatomical structure.

[0187] Fig. 11BReal-time inference using an imaging prediction model 80a is shown that predicts a relative pose 82a of an end effector n relative to a reference anatomy (eg, a reference ultrasound image) given an image 81a.

[0188] During the training phase, a data acquisition controller (not shown) is configured to receive and interpret information from the robot and the end effector (e.g., an ultrasonic device) and store the data on a data storage medium (not shown) in a format defined by the following specifications: The training data set D is composed of a 2-tuple d i =(U i , T i ) is composed of i data points represented by . This 2-tuple consists of an ultrasound image U acquired at a certain position T∈SE(3)82a relative to the reference position i 81a composition.

[0189] The training controller is configured to interpret a training data set D stored on a data storage medium. The data set D is composed of a 2-tuple d i =(U i , T i ) is composed of i data points represented by . The 2-tuple consists of the following items: the ultrasound image U of the anatomical structure i 81a and collecting ultrasound images U i The relative motion T between the current posture of the end effector and an arbitrarily selected reference position i 82a.

[0190] In such Fig.11E In one embodiment shown, the image prediction model 80a uses a neural network library 180a, which includes an input layer, a hidden layer, and an output layer, which are derived from a combination of one or more fully connected layers (FCL) 183a, one or more convolutional layers (CNL) 184a, one or more recurrent layers (RCL) 185a, and one or more long short-term memory (LSTM) layers 186a.

[0191] In practice, the combination of these layers is configured to implement the image U C Relative positioning to the reference image and thus the pose

[0192] In the implementation of image U C Relative positioning to the reference image and thus the pose In one embodiment, the neural network library 180a includes a set of N fully connected layers 183a.

[0193] In the implementation of image U CRelative positioning to the reference image and thus the pose In a second embodiment, the neural network library 180a includes a set of N convolutional layers 184a followed by a set of M fully connected layers 183a or a set of W recurrent layers 185a or a set of W long short term memory layers 186a.

[0194] In the implementation of image U C Relative positioning to the reference image and thus the pose In a third embodiment, the neural network library 180a includes a set of N convolutional layers 184a, followed by a combination of a set of M fully connected layers 183a and a set of W recurrent layers 185a or a set of W long short-term memory layers 186a.

[0195] In practice, the fully connected layer 183a may include K neurons, where N, M, W, K may be any positive integers and the values ​​may vary according to the embodiment. For example, N may be about 8, M may be about 2, W may be about 2, and K may be about 1000. Moreover, the convolutional layer 184a may implement a nonlinear transformation, which may be a composite function of operations (e.g., batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 184a may also include a nonlinear function configured to extract a rectified feature map (e.g., including a rectified nonlinear ReLU operation).

[0196] In addition, in practice, one of the layers 183a or 184a acts as a C The input layer of the layer 183a, 185a and 186a can serve as an output layer for outputting the posture 182a of the end effector 40 in the Cartesian space (e.g., the translation and rotation of the end effector 40 in the Cartesian space). The output posture of the end effector 40 in the Cartesian space can be represented as a vector parameterization and / or non-vector parameterization of the position and orientation of the rigid body. More specifically, the parameterization can be in the form of Euler angles, quaternions, matrices, exponential mappings, and / or angular axes representing rotations and / or translations (e.g., including the direction and magnitude of the translation).

[0197] Moreover, in practice, the output layer can be a non-linear fully connected layer 183a, which gradually reduces the high-dimensional output of the last convolutional layer 184a of the neural network bank 180a to produce a set of output variables.

[0198] During training, the predicted relative motion of the end effector relative to a reference anatomical structure is obtained by using the image prediction model (given the ultrasound image 161c and taking it as input) The training weights of the image prediction model 80a are continuously updated by comparison with the true relative motion T from a batch of training data sets D (which may be systematically or randomly selected from a data memory (not shown)). More specifically, the coefficients of the filter may be initialized with predefined values ​​or arbitrary values. The coefficients of the filter are applied to the batch of training data sets D via forward propagation and adjusted via back propagation to minimize any output error.

[0199] refer to Fig. 12A and Fig. 12B , the imaging prediction model 80ba of the present disclosure is trained on expert data that reasonably covers different situations. Given an ultrasound image 81a, the neural network 80ba infers the relative position 82a of the image 81a relative to a reference anatomical structure. As a result, as will be further described herein, the motion between the previous pose of the end effector in the anatomical structure (e.g., the heart) and the current pose of the end effector in the anatomical structure (e.g., the heart) can be calculated.

[0200] Fig. 12A Batch training of the imaging prediction model 80ba is shown. During training, the network continuously updates the weights of the imaging prediction model 80ba using 2-tuples from a ground-truth dataset consisting of an ultrasound image and the relative position of the image with respect to a reference anatomical structure.

[0201] Fig. 12B Real-time inference is shown using an imaging prediction model 80ba that predicts the linear and angular velocities 83a of the end effector relative to a reference anatomy (eg, a reference ultrasound image) given an image 81a.

[0202] During the training phase, a data acquisition controller (not shown) is configured to receive and interpret information from the robot and the end effector (e.g., an ultrasonic device) and store the data on a data storage medium (not shown) in a format defined by the following specifications: The training data set D is composed of a 2-tuple d i =(U i , V i ) is composed of i data points represented by . This 2-tuple consists of an ultrasound image U acquired at a certain position T∈SE(3)82a relative to a reference position obtained by the vector 83a of the linear velocity and angular velocity of the end effector i 81a composition.

[0203] The training controller is configured to interpret a training data set D stored on a data storage medium. The data set D is composed of a 2-tuple d i=(U i , V i ) is composed of i data points represented by . The 2-tuple consists of the following items: the ultrasound image U of the anatomical structure i 81a and collecting ultrasound images U i The relative vector 83a of the linear velocity and angular velocity of the end effector and an arbitrarily selected reference position.

[0204] In such Fig.12E In one embodiment shown, the image prediction model 80b uses a neural network library 180b, which includes an input layer, a hidden layer, and an output layer, which are derived from a combination of one or more fully connected layers (FCL) 183b, one or more convolutional layers (CNL) 184b, one or more recurrent layers (RCL) 185b, and one or more long short-term memory (LSTM) layers 186b.

[0205] In practice, the combination of these layers is configured to implement the image U C The relative positioning to the reference image and the linear velocity and / or angular velocity of the end effector 40 are derived therefrom.

[0206] In the implementation of image U C In one embodiment of the invention, the neural network library 180b includes a set of N fully connected layers 183b for determining the relative position of the end effector 40 to a reference image and deriving the linear velocity and / or angular velocity of the end effector 40 therefrom.

[0207] In the implementation of image U C In a second embodiment of the relative positioning to a reference image and thereby deriving the linear velocity and / or angular velocity of the end effector 40, the neural network library 180b includes a group of N convolutional layers 184b, followed by a group of M fully connected layers 183b or a group of W recurrent layers 185b or a group of W long short-term memory layers 186b.

[0208] In the implementation of image U C In a third embodiment of the relative positioning to a reference image and thereby deriving the linear velocity and / or angular velocity of the end effector 40, the neural network library 180b includes a group of N convolutional layers 184b, followed by a group of M fully connected layers 183b in combination with a group of W recurrent layers 185b or a group of W long short-term memory layers 186b.

[0209] In practice, the fully connected layer 183b may include K neurons, where N, M, W, K may be any positive integers and the values ​​may vary depending on the embodiment. For example, N may be about 8, M may be about 2, W may be about 2, and K may be about 1000. Moreover, the convolutional layer 184b may implement a nonlinear transformation, which may be a composite function of operations (e.g., batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 184b may also include a nonlinear function configured to extract a rectified feature map (e.g., including a rectified nonlinear ReLU operation).

[0210] In addition, in practice, one of the layers 183b or 184b acts as a C The input layer of , and one of the layers 183b, 185b and 186b can serve as an output layer for outputting the linear velocity and angular velocity of the end effector regressed according to the last fully connected layer (e.g., 6 units, 3 units for linear velocity, and 3 units for angular velocity) using a linear or nonlinear activation function

[0211] During training, the training weights of the image prediction model 80b are continuously updated by comparing the predicted linear and angular velocities of the end effector relative to a reference anatomical structure (given an ultrasound image 161c as input) with the linear and angular velocities of the end effector describing the actual motion to the reference anatomical structure from a batch of training data sets D (which may be systematically selected or randomly selected from a data memory (not shown)). More specifically, the coefficients of the filter may be initialized with predefined values ​​or arbitrary values. The coefficients of the filter are applied to the batch of training data sets D via forward propagation and are adjusted via back propagation to minimize any output error.

[0212] refer to Fig.13A and Fig. 13B , the continuous positioning controller 50f uses the inverse prediction model 70a, the image prediction model 80a, the subtractor 53 and the control law 54a to perform the closed-loop continuous position control method of the present disclosure as represented by the flowchart 190a.

[0213] In one TEE probe embodiment, stage S192a of process 190a includes inserting the TEE probe handle 132 (FIG. 3A) into a robotic controller 100 (FIG. 3B) as known in the art of the present disclosure to control the toggle of the handle 132 and the rotation of the TEE probe 130. The TEE transducer 140 (FIG. 3A) is inserted into the body through the esophagus and positioned near the anatomical structure of interest (e.g., the heart) (e.g., positioned at a position such as Fig. 13CThe ultrasound image parameters are defined at this target pose of the TEE transducer 140.

[0214] The stage S194a of the process 190a includes the image prediction model 90 processing the current ultrasound image 81a (which we will refer to as the previous ultrasound image U f ) to predict the relative position of the image plane with respect to the reference anatomical structure The sonographer uses the ultrasound image to visualize the anatomy and decide what movement the transducer needs to make from its current position. d Alternatively, the required motion of the transducer may be provided from an external tracking device, a user interface, other imaging modalities registered to the ultrasound image (eg, an X-ray image registered to a 3D TEE image using EchoNavigator (Philips)).

[0215] Based on T d , the inverse prediction model 70a predicts the joint variables required to move the robot to the desired position 72a. The robotic controller 100 receives the joint variables 72a and moves the TEE probe 130 accordingly.

[0216] The stage S196a of the process 190a includes the ultrasonic transducer reaching the acquisition ultrasonic image U c The image prediction model 90g uses the current ultrasound image U c The relative positions of the two current image planes with respect to the reference anatomical structure are predicted by As a result, the motion between the previous position and the current position in the anatomical structure (eg, the heart) can be calculated as follows:

[0217] In the second TEE probe embodiment, as Fig.13D As shown, the desired position T can be determined by path selection on an image 201 generated by an external imaging modality registered to an ultrasound image. d (For example, X-ray images as well as cone beam CT images can be registered to ultrasound images using methods known in the art (Philips EchoNavigator).

[0218] refer to Fig.14A and Fig. 14B , the continuous positioning controller 50g uses the inverse prediction model 70b, the image prediction model 80b, the subtractor 53 and the control law 54b to perform the closed-loop continuous position control method of the present disclosure as represented by the flowchart 190b.

[0219] In one TEE probe embodiment, stage S192b of process 190b includes inserting a TEE probe handle 142 (FIG. 3A) into a robotic controller 100 (FIG. 3B) as known in the art of the present disclosure to control the movement of the handle 142 and the rotation of the TEE probe 140. The TEE transducer 140 (FIG. 3A) is inserted into the body through the esophagus and positioned near the anatomical structure of interest (e.g., the heart) (e.g., positioned at a position such as Fig. 14C The ultrasound image parameters are defined at this target pose of the TEE transducer 140.

[0220] Stage S194b of process 190b includes the user's desire to move the transducer in image space, for example by selecting a path from point A to point B in ultrasound image 203 or a transformation between image planes A and B. In this embodiment, the user uses 末端执行器 J 图像 The Jacobian 204 transforms the first linear velocity and the first angular velocity 202 defined by the path on the image into the end effector coordinate system, 末端执行器 J 图像 The Jacobian 204 is calculated by knowing the spatial relationship between the end effector and the image coordinate system using methods known in the art of the present disclosure.

[0221] Based on V d , reverse prediction model 70b( Figure 7B ) predicts the joint velocities required to move the robot to the desired position 72b. The robot controller 100 receives the joint velocity 72b and moves the TEE probe 130 accordingly.

[0222] Stage S196b of process 190a includes the ultrasonic transducer reaching the U for acquiring the ultrasonic image. c The image prediction model 80b processes the current ultrasound image U c 81a to predict the velocity vector 83a of the end effector between point A and point B. The image prediction model 80b estimates the function between the Cartesian velocity of the end effector and the velocity in the joint space, i.e., the neural network models the manipulator Jacobian given a 6-vector consisting of the linear and angular velocities of the end effector 71c to predict the n-vector 72c of the joint velocities.

[0223] As will be appreciated by those skilled in the art of the present disclosure, a neural network that models spatial relationships between images of an anatomical structure (eg, the heart) requires a large training data set specific to a given organ.

[0224] In an alternative embodiment, features are extracted directly from the image to verify the location of the transducer. Fig. 14C In the illustrated embodiment, the user selects an object on the image 205, e.g., the apex wall, and the system extracts certain salient features around the object via feature extraction 206. These features (which may include the edges, shape, and size of the object) are first detected using methods known in the art of the present disclosure (e.g., Canny edge detector, morphological operations, etc.), and secondly tracked using a scale-invariant feature transform (SIFT) known in the art of the present disclosure. Finally, the system defines a path between the selected object and the center of the field of view that indicates the desired motion on the image. By tracking the salient features using SIFT, the continuous position controller 50g ( Fig.14A ) is able to correct predictions from the network within a closed-loop control.

[0225] More particular is the velocity-based control system of the continuum-like robot. Once the user selects an object on the image (e.g., the apical wall (see red dot on the ultrasound image)), the desired motion on the image is identified. The motion is defined by the path between the center of the field of view and the selected object, which can be transformed into linear and angular velocities in the end effector space using the Jacobian 204. This Cartesian velocity is then sent to a neural network, which infers the joint velocities. The achieved position will be iteratively verified against the path defined by the continuously tracked object and the center of the field of view.

[0226] In practice, the closed control loop of Figures 13 and 14 may be closed using other modalities, such as optical shape sensing (OSS), electromagnetic tracking, or X-ray images registered to ultrasound images using, for example, the Philips EchoNavigator.

[0227] Moreover, in practice, the prediction accuracy of the neural network will be affected by the configuration of the flexible endoscope. Therefore, the position of the transducer relative to the heart is first defined using, for example, Neural Network g or Philips HeartModel (which will implicitly define one of the possible configurations). Secondly, a specific set of network weights is loaded into the model according to the detected configuration, thereby improving the prediction accuracy.

[0228] Similar methods can be used to guide the user to a location that provides optimal images and guidance.

[0229] Furthermore, one of the most difficult issues in machine / deep learning is being able to access big data in the right format to train predictive models. More specifically, collecting and building training and validation sets is very time-consuming and expensive because it requires domain-specific knowledge. For example, in order to train a predictive model to accurately distinguish benign breast tumors from malignant breast tumors, such training requires thousands of ultrasound images annotated by radiologists and transformed into numerical representations that the training algorithm can understand. Additionally, image datasets may be inaccurate, corrupted, or labeled with noise, all of which can lead to inaccurate detection, and the collection of large medical datasets can raise ethical and privacy issues, among many other issues.

[0230] refer to Fig.15 , the training data collection system of the present disclosure uses a shape sensing controller 103 that provides a 3D position vector 233 for each point on an optical shape sensing fiber 332 mounted to the robot's endoscope 131 and tether-driven manipulator 231. The distal portion of the optical fiber 332 is embedded in a plastic housing that rigidly connects the fiber to the end effector and introduces a certain curvature into the shape.

[0231] For example, Fig.16A and Fig. 16B The distal end 332d of the optical shape sensing fiber 332 is shown embedded in a plastic housing 350 which is rigidly attached to a Fig. 16C and Fig.16D The ultrasonic transducer 232 of the manipulator 231 is shown. The plastic housing 350 imposes a certain curvature on the shape, thereby enabling the use of template matching algorithms known in the art of the present disclosure to estimate the end effector pose (e.g., as Fig.17 The data acquisition sequence 370 is shown, where α and β on each axis correspond to Fig.15 The knob position on the TEE handle 132 is shown).

[0232] Referring back to FIG. 16 , in general, by detecting this pattern, the pose of the end effector T∈SE(3) can be estimated using methods known in the art. The robot controller 100 sends motion commands to the robot, which controls the actuator knobs responsible for tightening or loosening the tether. By changing the state of the tether, the position of the end effector changes, as shown in FIG. 3D and FIG. 3E . The data storage controller 190 receives the shape h∈(p1...p n ), the pose T of the end effector, and the motion command (i.e., joint position) q t The data is stored as 3-tuples on a storage device and later used to train the deep convolutional neural network of the present disclosure.

[0233] More specifically, a shape sensing guidewire 332 is embedded or attached to the continuum robot, and the shape sensing guidewire 332 uses optical shape sensing (OSS) technology known in the art. OSS uses light along multi-core optical fibers for device positioning and navigation during surgical interventions. The principles involved utilize distributed strain measurements made in optical fibers using characteristic Rayleigh backscattering or controlled grating patterns.

[0234] The shape sensing controller 103 is configured to acquire the shape of the shape sensing guidewire 322 and estimate the pose T∈SE(3) of the end effector rigidly attached to the plastic housing 350, which imposes a certain curvature in the guidewire as described above. The method for estimating the pose T is based on a well-defined curvature and a template matching algorithm known in the art of the present disclosure.

[0235] The data acquisition controller 191 is configured to generate a sequence of motor commands according to a predefined acquisition pattern (eg, spiral, radial or square motion, etc.) and send movement commands to the robotic controller 100 .

[0236] The robot controller 100 is configured to receive the robot position and send movement signals to the robot. Using the electric knob, the robot will tighten / relax the tether, which will cause the movement of the end effector. The robot controller is also configured to receive and interpret information from the data acquisition controller 191 and change the robot position based on the information from the data acquisition controller 191.

[0237] The data storage controller 190 is configured to receive and interpret information from the robotic controller 100 and the shape sensing controller 103 and store the data on a data storage medium (not shown) in a format defined by the following specifications:

[0238] The first specification is to collect a training data set D for all configurations predefined by the data collection controller 191. The data set D consists of a set of n sequences W: D = {W1, W2, ..., W n}, each sequence W n From i data points d i Composition: W n ={d1, d2, ..., d i}; and each data point d from the sequence i are defined by a 3-tuple: i =(T i , H i , Q i ).

[0239] The 3-tuple consists of the end-effector pose T∈SE(3), a sequence of k consecutive shapes (e.g., H∈(ht ,h t+1 ,...h t+k )),where h∈(p1...o m ) is a set of m vectors p m , which describes the position of the shape sensing wire in 3D Euclidean space, auxiliary shape parameters (e.g., strain, curvature, and twist), and a sequence of j consecutive joint variables acquired at sequential time points starting from t to t+j (Q∈(q t ,q t+1 ...q t+j )). For example, entry q t It can be the angles α, β on the control knob acquired at time point t.

[0240] refer to Fig.19 The training data collection method 360 of the present disclosure is composed of Fig.15 The training data collection system is used to perform the task.

[0241] refer to Fig.15 and Fig.19 , stage S362 of method 360 includes moving the robot to the origin position using, for example, limit switches or proximity sensors by the robot controller 100. The distal end portion of the shape sensing guidewire 232 is inserted in a recess 353 provided in the plastic housing 350. The recess 353 will impose a certain curvature on the shape.

[0242] The housing 350 is rigidly attached to the end effector of the continuum robot.

[0243] By using a template matching algorithm known in the art of the present disclosure during stage S364 of method 360, the shape sensing controller 103 is now able to estimate the pose T∈SE(3) of the end effector. Preferably, the coordinate system of the end effector is defined by the template, however, additional calibration matrices can also be used. While the robotic system is still in the origin position, the pose of the end effector in the OSS coordinate system is acquired. Each subsequent pose acquired during the experiment is estimated relative to this initial position.

[0244] The data collection controller 191 collects data according to a predefined collection pattern (e.g., Fig.18 The data acquisition controller 191 iteratively sends the motion sequence to the robot controller 100, and the robot controller 100 moves the robot according to the generated joint variables.

[0245] Stage S366 of method 300 includes the data storage controller 190 collecting and storing data tuples d at each time point. i =(Ti , H i , Q i ). Importantly, due to H i and Q i are sequential, so previous time points are saved in memory by the data storage controller 190.

[0246] In order to facilitate further understanding of the various inventions of the present disclosure, Fig.19 The description teaches exemplary embodiments of the continuous positioning controller of the present disclosure. Based on this description, one of ordinary skill in the art will understand how to apply various aspects of the present disclosure to make and use additional embodiments of the continuous positioning controller of the present disclosure.

[0247] refer to Fig.19 The continuous positioning controller 400 includes one or more processors 401 , a memory 402 , a user interface 403 , a network interface 404 , and a storage device 405 interconnected via one or more system buses 406 .

[0248] Each processor 401 may be any hardware device capable of executing instructions stored in memory 402 or storage or otherwise processing data as known in the art of the present disclosure or contemplated hereinafter. In a non-limiting example, the processor(s) 401 may include a microprocessor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other similar device.

[0249] The memory 402 may include various memories, such as non-transitory memory and / or static memory as known in the art of the present disclosure or contemplated below, including but not limited to L1, L2 or L3 cache memory or system memory. In non-limiting examples, the memory 402 may include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read-only memory (ROM), or other similar memory devices.

[0250] User interface 403 may include one or more devices for enabling communication with users such as administrators as known in the art of the present disclosure or contemplated hereinafter. In a non-limiting example, the user interface may include a command line interface or a graphical user interface that may be presented to a remote terminal via network interface 404.

[0251] The network interface 404 may include one or more devices for implementing communication with other hardware devices as known in the art of the present disclosure or contemplated below. In a non-limiting example, the network interface 404 may include a network interface card (NIC) configured to communicate according to the Ethernet protocol. In addition, the network interface 404 may implement a TCP / IP stack for communication according to the TCP / IP protocol. Various alternative or additional hardware or configurations for the network interface 404 will be apparent.

[0252] The storage device 405 may include one or more machine-readable storage media as known in the art of the present disclosure or contemplated below, including but not limited to read-only memory (ROM), random access memory (RAM), disk storage media, optical storage media, flash memory devices, or similar storage media. In various non-limiting embodiments, the storage device 405 may store instructions for execution by (one or more) processors 401 or data on which (one or more) processors 401 can operate. For example, the storage device 405 may store a basic operating system for controlling various basic operations of the hardware. The storage device 405 also stores application modules in the form of executable software / firmware for implementing various functions of the controller 400a as previously described in the present disclosure, including but not limited to (one or more) forward prediction models 60, (one or more) reverse prediction models 70, and (one or more) imaging prediction models 80 as previously described in the present disclosure.

[0253] In practice, the controller 400 may be installed in the X-ray imaging system 500, the interventional system 501 (e.g., an interventional robotic system), or an independent workstation 502 (e.g., a client workstation or a mobile device such as a tablet computer) that communicates with the X-ray imaging system 500 and / or the interventional system 501. Alternatively, components of the controller 400 may be distributed among the X-ray imaging system 500, the interventional system 501, and / or the independent workstation 502.

[0254] Moreover, in practice, the additional controllers of the present disclosure including the shape sensing controller, the data storage controller, and the data acquisition controller may also each include the following: Fig.19 One or more processors, memories, user interfaces, network interfaces, and storage devices interconnected via one or more system buses are arranged in the controller, whereby the storage device contains applicable application modules of the controller as previously described herein. Alternatively, two or more controllers of the present disclosure may be integrated into a single controller, wherein the storage device contains applicable application modules of the two or more controllers as previously described herein.

[0255] Those of ordinary skill in the art of the present disclosure will recognize the numerous benefits of the present disclosure with reference to FIGS. 1-19 .

[0256] In addition, as will be appreciated by those of ordinary skill in the art in view of the teachings provided herein, the structures, elements, components, etc. described in the present disclosure / specification and / or depicted in the accompanying drawings can be implemented in various combinations of hardware and software and provide functionality that can be combined in a single element or multiple elements. For example, the functionality of the various structures, elements, components, etc. shown / illustrated / depicted in the accompanying drawings can be provided by using dedicated hardware and hardware capable of running software associated with appropriate software for additional functionality. When provided by a processor, the functionality can be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors (some of which can be shared and / or multiplexed). In addition, the explicit use of the term "processor" or "controller" should not be interpreted as referring exclusively to hardware capable of running software, and can implicitly include, without limitation, digital signal processor ("DSP") hardware, memory (e.g., read-only memory ("ROM"), random access memory ("RAM"), non-volatile storage, etc. for storing software), and substantially any unit and / or machine (including hardware, software, firmware, combinations thereof, etc.) capable of (and / or configurable to) execute and / or control a process.

[0257] In addition, all statements of the principles, aspects and embodiments of the present invention and its specific examples recorded herein are intended to cover their structural and functional equivalents. In addition, such equivalents are intended to include currently known equivalents and equivalents developed in the future (i.e., any element developed that can perform the same or substantially similar function regardless of its structure). Therefore, for example, in view of the teachings provided herein, it should be appreciated by those of ordinary skill in the art that any block diagram presented herein can represent a conceptual view of an illustrative system component and / or circuit that implements the principles of the present invention. Similarly, in view of the teachings provided herein, it should be appreciated by those of ordinary skill in the art that any flow chart, flow chart, etc. can represent various processes, which can be substantially represented in a computer-readable storage medium and run in this way by a computer, processor or other device having processing power, regardless of whether such a computer or processor is explicitly shown.

[0258] Having described various preferred and exemplary embodiments of the invention of the present disclosure (which embodiments are intended to be illustrative and not limiting), it should be noted that modifications and variations can be made by those skilled in the art in accordance with the teachings provided herein (including the accompanying drawings). Therefore, it should be understood that changes can be made in / to the preferred and exemplary embodiments of the present disclosure within the scope of the embodiments disclosed herein.

[0259] In addition, it is contemplated that corresponding and / or related systems that include and / or implement the apparatus / system according to the present disclosure or that can be used / implemented in the apparatus, for example, are also contemplated and considered to be within the scope of the present disclosure. Moreover, corresponding and / or related methods for making and / or using the apparatus and / or system according to the present disclosure are also contemplated and considered to be within the scope of the present disclosure.

Claims

1. A training data collection system for an interventional device (130), the interventional device comprising a device portion and a sensor (332) adapted to provide position and / or orientation and / or shape information, at least a portion of the sensor (332) being attached to the device portion, the training data collection system comprising: a robotic controller (100) configured to control at least one motion variable of the interventional device (130); a data acquisition controller (191) configured to command the robotic controller (100) to control the at least one motion variable of the interventional device (130) according to a predefined data point pattern; A positioning determination module is configured to: determining position information based on position and / or orientation and / or shape information received from said sensor, At each data point of the predefined data point pattern, a pose of the device part and a positioning movement of the interventional device (130) are estimated based on the position information, and outputting an estimated pose of the device portion and an estimated positioning motion of the interventional device (130) at each data point of the predefined data point pattern; as well as Data storage controller (190), wherein throughout the process of the robotic controller (100) controlling the at least one motion variable of the interventional device (130) according to the predefined data point pattern, the data storage controller (190) is configured to: receive communication information of the estimated pose of the device part at each data point from the positioning determination module, receive communication information of the estimated positioning motion of the interventional device (130) at each data point from the positioning determination module, and receive communication information of the at least one motion variable of the interventional device (130) at each data point from the robotic controller (100), and Wherein, the data storage controller (190) is further configured to: store a time data series for the interventional device (130) derived based on the estimated posture of the device part at each data point, the estimated positioning motion of the interventional device (130) at each data point, and the at least one motion variable of the interventional device (130) at each data point, wherein the time data series is configured to train a model for predicting navigation variables related to continuous positioning control of the device part of the interventional device.

2. The training data collection system according to claim 1, wherein: At least a portion of the sensor (332) is within the guidewire.

3. The training data collection system according to claim 1, wherein: The predefined data point pattern includes one of a spiral data point pattern, a radial data point pattern, or a square data point pattern.

4. The training data collection system according to claim 1, wherein: The position determination module is configured to generate the estimate of the pose of the device part based on matching a curvature shape of the sensor (332) attached to the device part to a template.

5. The training data collection system according to claim 1, wherein: The data storage controller is configured to continuously command the robotic controller (100) to continuously control the at least one motion variable of the interventional device (130) according to the predefined data point pattern.

6. The training data collection system according to claim 1, wherein: For each data point, the time data series includes: said estimated pose of said device part, a sequence of successive shapes of the sensor (332), and A sequence of successive joint variations of the interventional device (130).

7. The training data collection system according to claim 1, wherein: The positioning determination module is configured to receive shape data or to derive shape data based on the position and / or orientation and / or shape information received from the sensor to determine the estimated result of the posture of the device part and the estimated result of the positioning movement of the interventional device (130) at each data point of the predefined data point pattern.

8. The training data collection system according to claim 1, wherein: The sensor includes a marker visible in an imaging system to be used to visualize a region of interest including the interventional device (130), an electromagnetic tracking sensor, a transducer sensor and / or an optical shape sensing element provided by / in an optical fiber.

9. The training data collection system according to claim 1, wherein: The sensor (332) is an optical shape sensor (332) or includes an optical shape sensor, and at least a portion of the optical shape sensor attached to the device part is a segment of the optical shape sensor (332), wherein the positioning determination module is a shape sensing controller or includes a shape sensing controller, the shape sensing controller is configured to control shape sensing of the optical shape sensor (332), including the estimation of the posture of the device part at each data point of the predefined data point pattern and the estimation of the positioning movement of the interventional device (130) at each data point of the predefined data point pattern, and wherein the shape sensing controller is configured to derive the estimation result of the posture of the device part based on the segment of the optical shape sensor (332) attached to the device part.

10. The training data collection system according to claim 1, wherein: At least part of the sensor (332) is attached to the device part in a fixed shape.

11. The training data collection system according to claim 1, wherein: The device part is an end effector of the interventional device (130) or includes an end effector of the interventional device.

12. The training data collection system according to any one of claims 1 to 10, further comprising the interventional device (130).

13. A non-transitory machine-readable storage medium encoded with instructions for instructing an interventional device (130) to perform the following operations, the interventional device comprising a device portion and a sensor (332) adapted to provide position and / or orientation and / or shape information, at least a portion of the sensor (332) being attached to the device portion, the operations comprising: controlling at least one motion variable of the interventional device (130) according to a predefined data point pattern; estimating a pose of the device part and a positioning movement of the interventional device (130) at each data point of the predefined data point pattern based on position information determined from position and / or orientation and / or shape information received from the sensor; and Throughout the process of controlling the at least one motion variable of the interventional device (130) according to the predefined data point pattern, a time data series for the interventional device (130) derived based on the estimated posture of the device portion at each data point, the estimated positioning motion of the interventional device (130) at each data point, and the at least one motion variable of the interventional device (130) at each data point is stored, wherein the time data series is configured to train a model for predicting navigation variables associated with continuous positioning control of the device portion of the interventional device.

14. A training data collection method for an interventional device (130), the interventional device comprising a device portion and a sensor (332) adapted to provide position and / or orientation and / or shape information, at least a portion of the sensor (332) being attached to the device portion, the training data collection method comprising: controlling at least one motion variable of the interventional device (130) according to a predefined data point pattern via a robotic controller (100) and a data acquisition controller (191); estimating a pose of the device part and a positioning movement of the interventional device (130) at each data point of the predefined data point pattern based on position information determined from position and / or orientation and / or shape information received from the sensor; and Throughout the process of controlling the at least one motion variable of the interventional device (130) according to the predefined data point pattern, a time data series for the interventional device (130) derived based on the estimated posture of the device part at each data point, the estimated positioning motion of the interventional device (130) at each data point, and the at least one motion variable of the interventional device (130) at each data point is stored via a data storage controller (190), wherein the time data series is configured to train a model for predicting navigation variables associated with continuous positioning control of the device part of the interventional device.

Images