Feedforward continuous positioning control of the end effector

Abstract

A positioning controller (50) for positioning control of an interventional device (30) includes a forward prediction model (60) and / or an inverse control prediction model (70), and the interventional device includes a portion (40) of the interventional device. In operation, the controller (50) can apply the forward prediction model (60) to a commanded positioning movement of the interventional device (30) to present a predicted navigation pose of the end effector (40), and generate positioning data regarding information that the portion (40) of the interventional device (30) positions to a target pose based on the predicted navigation pose of the portion (40) of the interventional device. Alternatively, before or subsequently, the controller (50) can apply the control prediction model (70) to the target pose of the portion (40) of the interventional device to present a predicted positioning movement of the interventional device (30), and generate a positioning command for controlling that the device portion (40) of the interventional device (30) positions to the target pose based on the predicted positioning movement of the interventional device (30).

Description

Technical Field

[0001] The present disclosure generally relates to position control of portions of an intervention device (e.g., an end effector of an intervention device) used in an intervention procedure (e.g., minimally invasive surgery, video-assisted thoracic surgery, vascular procedure, endoluminal procedure, deformity correction procedure). The present disclosure can particularly relate to incorporating a prediction model in the position control of such portions of an intervention device used in an intervention procedure. Background Art

[0002] Continuous (or non - continuous) control - positioning the device portion (e.g., end effector) within a certain workspace - is one of the most commonly attempted forms of control in conventional rigid - link robots. By leveraging the discrete rigid - link structure of the robot, precise positioning of the said portion of the intervention device (e.g., end effector) can be achieved as needed in structured applications (e.g., manufacturing). However, due to the deformable and fragile nature of the soft tissues of human organs and patient safety, there is less need to use rigid - link robots in a clinical environment.

[0003] More specifically, biologically inspired robots can generate motions similar to those of snakes, elephants, and octopuses, which can be very effective in manipulating soft anatomical objects. Nevertheless, given the complexity of the continuum (or quasi - continuum) structure for the desired degrees of freedom, it is difficult to model mathematically or to provide sufficient actuator inputs to achieve consistent control. Therefore, effective control (and particularly effective continuous control) of the robotic structure in a clinical environment has proven extremely difficult to achieve.

[0004] For example, a method for continuous position control of an end effector supported by a continuum robot involves modeling and controlling the configuration of the continuum robot, where a static model (formulated as a set of non - linear differential equations) attempts to account for the gun - like deformation of the continuum robot due to bending, torsion, and elongation. However, the accuracy of the mathematical modeling is susceptible to changes in the environmental conditions of the robot (e.g., temperature and humidity), which can affect the mechanical properties of the robot components, and is also susceptible to any manufacturing errors or various workloads.

[0005] As another example, another method for positioning control of a robot-supported end effector is to manipulate the robot by projecting a set of allowable motions and a set of allowable forces into the joint space corresponding to the operator's control. For example, after inserting the robot into the nasal cavity, the controller adjusts the position of each segment of the robot to increase the difference between the measured generalized force and the expected generalized force on each end plate. However, increasing the degrees of freedom of the robot to make it more maneuverable has the adverse effect of complicating the robot kinematics. This can be particularly problematic in the case of continuously controlling a continuum robot.

[0006] In addition, even if effective positioning control of the robot-supported end effector is achieved, incorrect calibration or use of the robot system or general wear of mechanical components will have a negative impact on the prediction accuracy of the kinematics of the robot structure. Similarly, this can be particularly problematic in the case of continuously controlling a continuum robot (or continuum robot structure).

[0007] US2014 / 046128 A1 describes applying a control method to a surgical robot system that includes a slave robot and a master robot. The slave robot has a robot arm to which a main surgical tool and an auxiliary surgical tool are coupled. The master robot has a master manipulator for manipulating the robot arm. The control method includes: acquiring data on the motion of the master manipulator, predicting a basic motion to be performed by the operator based on the acquired motion data and the result of learning a plurality of motions constituting a surgical task, and adjusting the auxiliary surgical tool based on the predicted basic motion so as to correspond to the operator's basic motion.

[0008] US2017 / 172662 A1 describes a system that includes a quantitative three-dimensional (Q3D) endoscope configured to image a field of view and a processor that generates a Q3D model of the scene and identifies target instruments and structures. The processor is configured to: display the scene seen from a virtual field of view of the instrument, determine a no-fly zone around the target, determine a predicted path for the instrument, or provide 3D tracking of the instrument.

[0009] WO 2011 / 123669 A1 describes a control system for robot control with user guidance of a medical device.

[0010] US2013 / 211244 A1 describes detecting and predicting the position and trajectory of a surgical tool. A radiography imaging system can capture an image of the surgical tool inside or near a patient. An associated processing circuit can process the image to determine and predict the position, orientation, and trajectory of the tool based on a 3D model of the tool, geometric calculation results and mathematical models describing the movement and deformation of the surgical tool inside the patient.

[0011] US2016 / 349044 A1 describes a shape sensing system that includes a processor coupled to a memory storage device. A prediction model module is stored in the memory storage device and is configured to receive shape sensing measurements and predict the next shape sensing measurement based on current state data for the shape sensing enabled device. SUMMARY OF THE INVENTION

[0012] Known techniques designed to provide position control for parts of an intervention device (e.g., an end effector) provide limited benefits. Accordingly, there is still a need for improved techniques for providing effective position control for these parts of an intervention device. To that end, the present disclosure teaches feedforward position control, feedback position control, and data collection. This control is preferably performed continuously.

[0013] To that end, the present invention presents a position controller for an intervention device as a first embodiment, the intervention device including an imaging device.

[0014] As a second and third embodiment, the present invention presents a non-transitory (or transitory) machine-readable storage medium encoded with instructions and a position method executable by a position controller for an intervention device. Examples listed below that are not within the scope of the claims are used to assist in understanding the present invention.

[0015] Feedforward (preferably continuous) positioning control The present disclosure also teaches a prediction model method for feedforward (preferably continuous) position control of manual or automatic navigation positioning of a device part (e.g., an end effector) supported by an intervention device based on a prediction model configured (optionally trained on) the kinematics of the intervention device.

[0016] Another embodiment of the present disclosure for feedforward (preferably continuous) position control of a device part (e.g., an end effector) is a (continuous) position controller that includes a forward prediction model and / or a control (optionally inverse) prediction model, the forward prediction model being configured (optionally trained on) the (optional forward) kinematics of the intervention device that predicts the navigation pose of the end effector, and / or the control (optionally inverse) prediction model being configured (optionally trained on the inverse kinematics of the intervention device that predicts the positioning movement of the intervention device).

[0017] For the purposes of the description and the present disclosure, the term "navigation pose" broadly encompasses the pose of the portion of the intervention device (e.g., the end effector of the intervention device) during an intervention procedure when being navigated to a spatial position via the intervention device, and the term "positioning movement" broadly encompasses any movement of the intervention device during an intervention procedure for navigating the device portion to a spatial position.

[0018] In operation, a continuous positioning controller applies a forward prediction model to a commanded positioning movement of the intervention device to present a predicted navigation pose of the end effector and generate positioning data regarding the positioning of the device portion to a target pose by the intervention device based on the predicted navigation pose of the device portion.

[0019] Alternatively, antecedently or concurrently, (preferably continuous) the positioning controller applies an inverse prediction model to the target pose of the portion of the intervention device (e.g., the end effector) to present a predicted positioning movement of the intervention device and generate a (continuous) positioning command for controlling the positioning of the device portion to the target pose by the intervention device based on the predicted positioning movement of the intervention device.

[0020] Feedback (preferably continuous) positioning control . The present disclosure also teaches a predictive model method for (preferably continuous) positioning control of feedback of manual navigation positioning or (semi-)automatic navigation positioning of an imaging device associated with or attached to a portion of an intervention device (e.g., the end effector of the intervention device) based on the following imaging prediction model: the imaging prediction model is configured to receive imaging data from the imaging device as feedback of manual navigation positioning or automatic navigation positioning of the imaging device (or the portion of the intervention device linked to the imaging device - e.g., the end effector) to a target pose, using the kinematics of the intervention device (optionally related to image data).

[0021] An example of the present disclosure for (preferably continuous) control positioning of feedback of an imaging device (or the portion of the intervention device linked to the imaging device - e.g., the end effector) is a continuous positioning controller that includes an imaging prediction model trained on the correlation of relative imaging performed by the end effector with the forward kinematics of the intervention device predicting the navigation pose of the end effector. The (continuous) positioning controller may also include a control prediction model configured using the kinematics of the intervention device predicting a corrective positioning movement of the intervention device. Optionally, the control prediction model is trained or has been trained on the inverse kinematics of the intervention device to output the prediction of the corrective positioning movement.

[0022] For purposes of the description and for this disclosure, the term "relative imaging" broadly encompasses the generation of an image of an intervention procedure by an end effector in a given pose relative to a reference image of the intervention procedure.

[0023] In operation, after a device portion (e.g., an end effector) has been navigated to a target pose, a (preferably continuous) positioning controller applies an imaging prediction model to imaging data generated by the end effector to present a predicted navigation pose of the device portion (e.g., the end effector), applies a control (or specifically, an inverse) prediction model to error positioning data derived in terms of a difference between the target pose of the end effector and the predicted navigation pose of the end effector to present a predicted corrective positioning movement of the intervention device, and generates, based on the predicted corrective positioning movement of the intervention device, a continuous positioning command for controlling the intervention device to correctively position an imaging device (40) or a portion of the intervention device associated with the imaging device (e.g., the end effector) to the target pose.

[0024] Training data collection Additionally, to facilitate (optionally continuous) positioning control of manual or automated navigation positioning of a portion of an intervention device (e.g., an end effector) via a prediction model invariant to environmental differences (e.g., anatomical differences between patients, such as patient body size, heart position, etc.), this disclosure also teaches an (optionally trained) data collection technique, which presumes navigation positioning of the intervention device portion via a predefined pattern of data points and recording of the spatial positioning of the intervention device and the pose of the intervention device portion at each data acquisition point, thereby collecting (training) data for the prediction model to infer the forward kinematics or inverse kinematics of the intervention device.

[0025] One example of this disclosure for collecting (optionally training) data for a prediction model is an (optionally trained) data collection system for an intervention device that includes the portion of the intervention device (e.g., the end effector) and a sensor adapted to provide position and / or orientation and / or shape information, with at least a portion of the sensor (332) attached (optionally in a fixed shape) to the intervention device portion. Such sensors can include markers visible from an imaging system (e.g., an X-ray system, an MRI system), an electromagnetic tracking sensor, a transducer sensor, and / or optical shape sensing provided by / within an optical fiber.

[0026] A specific example of this example is an (optionally trained) data collection system for an intervention device that includes a portion of the intervention device (e.g., the end effector) and an optical shape sensor, wherein segments of the optical shape sensor are attached (optionally in a fixed shape) to the end effector.

[0027] (Training) data collection systems can use a robot controller, a data acquisition controller, a positioning determination module (or a shape sensing controller in the specific examples mentioned above), and a data storage controller.

[0028] In operation, the data acquisition controller can command the robot controller to control the (one or more) motion variables of the intervention device according to a predefined data point pattern, and the positioning determination module (or shape sensing controller) is configured to determine position information based on the position and / or orientation and / or shape information received from the sensors, so as to output an estimated result of the posture of the part of the intervention device at each data point of the predefined data point pattern and / or an estimated result of the positioning movement of the intervention device. Therefore, for the purpose of the estimation, the determined position information is optionally retrieved or derived or extracted or received based on the kinematics of the configured positioning determination module or the behavior of the intervention device. In a specific example, the positioning determination module can derive or receive derived shape data according to the position and / or orientation and / or shape information to determine the estimated result. If the positioning determination module is a shape sensing controller (as in the specific example mentioned above), the positioning determination module controls the shape sensing of the optical shape sensor, including the estimation of the posture of the end effector at each data point of the predefined data point pattern and the estimation of the positioning movement of the intervention device at each data point of the predefined data point pattern.

[0029] The "deriving shape data according to the position and / or orientation and / or shape information" can be implemented according to known techniques for deriving shapes from data provided by "sensors". For example, the positions tracked by the sensors can well indicate the general shape of the segments of the intervention device carrying these sensors, and algorithms (more or less developed according to the distances between the sensors and the possible shapes of the intervention device along the segment) can be developed to derive or reconstruct the shape. This dynamic tracking of the position can also give an indication of the deformed orientation. The sensors can also provide strain information (e.g., Rayleigh or Bragg grating sensors), which can indicate the local positioning and orientation of the intervention device, and based on which the shape can be derived and reconstructed (also a known technique).

[0030] The estimated result of the posture of the intervention device part (e.g., the end effector) is derived based on at least part of the sensors (optionally in a fixed shape) attached to the end effector.

[0031] Throughout the process, the robotic controller can control the (one or more) motion variables of the intervention device according to a predefined pattern of data points. The data storage controller can receive communication information regarding the estimated pose of the end effector for each data point from the shape sensing controller, can receive communication information regarding the estimated positioning motion of the intervention device for each data point from the positioning determination module (or the shape sensing controller), and can receive communication information regarding at least one motion variable of the intervention device for each data point from the robotic controller.

[0032] In response to this communication information, the data storage controller can store a time data series for the intervention device derived from the estimated pose of the end effector at each data point, the estimated spatial positioning of the intervention device at each data point, and the (one or more) motion variables of the intervention device at each data point. This time data series can serve as training data for any type of machine learning model, particularly the prediction model of the present disclosure.

[0033] Additionally, the data acquisition controller can further command the robotic controller to control the (one or more) motion variables of the intervention device according to an additional (one or more) predefined pattern of data points, whereby the data storage controller generates an additional time data series for any type of machine learning model, particularly the prediction model of the present disclosure.

[0034] Moreover, for the description and claims of the present disclosure:

[0035] (1) Terms in the art including but not limited to "end effector", "kinematics", "position", "positioning", "pose", "achieved pose", "motion", and "navigation" are to be construed as known in the field of the present disclosure and exemplarily described in the present disclosure;

[0036] (2) Examples of end effectors include but are not limited to intraoperative imaging devices, intervention tools / surgical instruments, and surgical sutures as known in the field of the present disclosure and contemplated hereinafter;

[0037] (3) The term "intraoperative imaging device" broadly encompasses all imaging devices for depicting anatomical objects / regions as known in the field of the present disclosure and contemplated hereinafter. Examples of intraoperative imaging devices include but are not limited to transesophageal echocardiography transducers (e.g., X7-2t transducer, Philips), laparoscopic ultrasound transducers (e.g., L10-4lap transducer, Philips), optical cameras, and sensing devices (e.g., tissue spectral sensing sensors, ECG electrodes, probes for electrophysiological mapping);

[0038] (4) Examples of intervention tools / surgical instruments include, but are not limited to, scalpels, cautery devices, ablation devices, needles, forceps, Kirschner wires and associated drivers, endoscopes, awls, screwdrivers, osteotomes, chisels, cone rods, curettes, clamps, pliers, periosteal and J-shaped needles, as known in the art of the present disclosure and contemplated hereinafter;

[0039] (5) The term "intervention device" broadly encompasses all devices known in the art of the present disclosure and contemplated hereinafter for supporting the positioning of the end effector during application. Examples of intervention devices include, but are not limited to, continuum flexible robots, flexible endoscopes, and guide wires;

[0040] (6) Examples of flexible robots include, but are not limited to, hyper-redundant robots (e.g., multiple discrete links, serpentine links, or concentric tubes), continuum backbone robots (e.g., cable-actuated), tether-driven robots, gel-like soft robots, and fluid-filled tube-actuated robots, as known in the art of the present disclosure and contemplated hereinafter;

[0041] (7) Examples of flexible endoscopes include, but are not limited to, endoscopes of transesophageal echocardiography (TEE) probes, endoscopes of intracardiac echocardiography (ICE) probes, laparoscopes, and bronchoscopes, as known in the art of the present disclosure and contemplated hereinafter;

[0042] (8) The term "prediction model" broadly encompasses all types of models known in the art of the present disclosure and contemplated hereinafter for predicting navigation variables related to the (optional) continuous positioning control of parts of an intervention device (e.g., the end effector) as exemplarily described in the present disclosure. Optionally, these prediction models can be configured to output such prediction results based on a positioning data set using kinematics. Optionally, the prediction model is trained or has been trained on the kinematics of the intervention device. Examples of prediction models include, but are not limited to, artificial neural networks (e.g., feedforward convolutional neural networks, recurrent neural networks, long short-term memory networks, autoencoder networks, generative adversarial networks, and many other deep learning neural networks);

[0043] (9) The term "controller" broadly encompasses all structural configurations of a main circuit board or integrated circuit for controlling the application of various inventive principles of the present disclosure, as understood in the field of the present disclosure and as exemplarily described in the present disclosure. The structural configuration of the controller may include, but is not limited to, (one or more) processors, (one or more) computer-usable / computer-readable storage media, an operating system, (one or more) application modules, (one or more) peripheral device controllers, (one or more) slots and (one or more) ports, and control instructions. The controller may be housed within a workstation or communicatively connected to the workstation;

[0044] (10) The term "application module" broadly encompasses application programs contained within or accessible by the controller, the controller including an electronic circuit (e.g., electronic components and / or hardware) and / or an executable program for executing a specific application program (e.g., executable software stored on (one or more) non-transitory (transitory) computer-readable media and / or firmware); and

[0045] (11) The terms "signal", "data", and "command" broadly encompass all forms of detectable physical quantities or pulses (e.g., voltage, current, or magnetic field strength) for transmitting information and / or instructions for supporting the application of various inventive principles of the present disclosure, as understood in the field of the present disclosure and as exemplarily described in the present disclosure. The signal / data / command communication of the various components of the present disclosure may involve any communication method known in the field of the present disclosure, including but not limited to signal / data / command transmission / reception via any type of wired or wireless data link and reading of signals / data / commands uploaded to computer-usable / computer-readable storage media.

[0046] By reading the following detailed description of the various embodiments of the present disclosure in conjunction with the accompanying drawings, the foregoing and other embodiments of the present disclosure, as well as the various structures and advantages of the present disclosure, will become more apparent. The detailed description and the drawings are merely illustrative of the present disclosure and not limiting, and the scope of the present disclosure is defined by the claims and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] FIG. 1 illustrates an exemplary embodiment of an intervention device including an end effector, as known in the field of the present disclosure.

[0048] Figure 2A Illustrates an exemplary embodiment of a continuous position controller in accordance with various aspects of the present disclosure.

[0049] Figure 2BIllustrates an exemplary embodiment of a continuous positioning state machine according to various aspects of the present disclosure.

[0050] FIG. 3A illustrates an exemplary embodiment of a transesophageal echocardiography (TEE) probe as known in the art of the present disclosure.

[0051] FIG. 3B illustrates an exemplary embodiment of a handle of a transesophageal echocardiography (TEE) probe as known in the art of the present disclosure.

[0052] FIGS. 3C - 3F illustrate exemplary movements of a TEE probe as known in the art of the present disclosure Figure 2A thereof.

[0053] FIG. 3G illustrates an exemplary embodiment of a robotically - manipulated transthoracic echocardiography (TTE) probe as known in the art of the present disclosure.

[0054] FIG. 3H illustrates an exemplary embodiment of an optically - shaped - sensing continuum robot as known in the art of the present disclosure.

[0055] Figures 4A - 4E Illustrates a first exemplary embodiment of the forward prediction model and forward prediction method of the present disclosure.

[0056] Figures 5A - 5E Illustrates a first exemplary embodiment of the inverse prediction model and inverse prediction method of the present disclosure.

[0057] Figures 6A - 6E Illustrates a second exemplary embodiment of the forward prediction model and forward prediction method of the present disclosure.

[0058] Figures 7A - 7E Illustrates a second exemplary embodiment of the inverse prediction model and inverse prediction method of the present disclosure.

[0059] Figures 8A - 8B Illustrates a third exemplary embodiment of the forward prediction model and forward prediction method of the present disclosure.

[0060] Figures 9A - 9F Illustrates a fourth exemplary embodiment of the forward prediction model and forward prediction method of the present disclosure.

[0061] Figures 10A - 10B Illustrates an image prediction model and an image prediction method.

[0062] Figures 11A - 11B Shows a first exemplary embodiment of the image prediction model and image prediction method of the present disclosure.

[0063] Figures 12A - 12BIllustrates a second exemplary embodiment of the forward prediction model and the forward prediction method of the present disclosure.

[0064] Figures 13A - 13D Illustrates the closed-loop attitude control of the present disclosure.

[0065] Figures 14A - 14D Illustrates the closed-loop vector velocity control of the present disclosure.

[0066] Figures 15 - 18 Illustrates the training data collection system and method of the present disclosure.

[0067] Figure 19 Illustrates an exemplary embodiment of the continuous positioning controller of the present disclosure. Detailed Description

[0068] The present disclosure is applicable to a wide variety of applications that require continuous position control of an end effector. Examples of such applications include, but are not limited to, minimally invasive procedures (e.g., endoscopic hepatectomy, necrosectomy, prostatectomy, etc.), video-assisted thoracic surgery (e.g., lobectomy, etc.), microvascular procedures (e.g., via catheter, sheath, deployment system, etc.), micro medical diagnostic procedures (e.g., intraluminal procedures via endoscope or bronchoscope), deformity correction procedures (e.g., via Kirschner wire, screwdriver, etc.), and non-medical applications.

[0069] The present disclosure improves the continuous position control of the end effector during such applications by providing a prediction of the attitude of the end effector and a positioning of the movement of an intervention device that can be used to control and / or verify the manual or automatic navigated positioning of the end effector.

[0070] To facilitate understanding of the present disclosure, the following description of FIG. 1 teaches an exemplary embodiment of an intervention device including an end effector as known in the art of the present disclosure. From the description of FIG. 1, one of ordinary skill in the art of the present disclosure will recognize how to apply the present disclosure to make and use additional embodiments of an intervention device including an end effector as known in the art of the present disclosure and contemplated hereinafter.

[0071] Referring to FIG. 1, in practice, the intervention device 30 supports the manual or automatic navigated positioning of the end effector 40 during the application as represented by the arrow extending from the end effector 40. Examples of the intervention device 30 include, but are not limited to, continuum flexible robots, flexible endoscopes, and guide wires, and examples of the end effector 40 include, but are not limited to, intraoperative imaging devices, intervention tools / surgical instruments, and surgical sutures.

[0072] In operation, one or more navigation commands 31, one or more actuation signals 32, and / or one or more navigation forces 33 are transmitted to / interposed on the intervention device 30, whereby the intervention device 30 translates, rotates, and / or pivots in accordance with the one or more navigation commands 31, the one or more actuation signals 32, and / or the one or more navigation forces 33 to navigate the end effector 40 to a target pose (i.e., a position and orientation in the application space).

[0073] For example, FIGS. 3A and 3B illustrate an embodiment of a transesophageal echocardiography (TEE) probe 130 as the intervention device 30 having an imaging end effector 140 that can be inserted through the mouth of a patient P and into the esophagus to capture an image of the patient P's heart, and a practitioner (not shown) or a robotic controller 100 ( Figure 2B ) operates the handle 132 of the TEE probe 130 to reposition the TEE probe 130 within the patient P to navigate the imaging end effector 140 to a target pose.

[0074] More specifically, the TEE probe 130 includes a flexible elongate member 131, a handle 132, and an imaging end effector 140. The flexible elongate member 131 is structurally arranged and / or otherwise configured in terms of size and / or shape to be positioned within a body lumen (e.g., the esophagus) of the patient. The imaging end effector 140 is mounted at the distal end of the member 131 and includes one or more ultrasound transducer elements, whereby the imaging end effector 140 is configured to transmit ultrasound energy toward the anatomy (e.g., the heart) of the patient P. The ultrasound energy is reflected by the patient's vasculature and / or tissue structure, whereby the ultrasound transducer elements in the imaging end effector 140 receive the reflected ultrasound echo signals. In some embodiments, the TEE probe 130 may include internal or integrated processing components that can locally process the ultrasound echo signals to generate image signals representative of the anatomy of the patient P being imaged. In practice, the one or more ultrasound transducer elements may be arranged to provide a two-dimensional (2D) image or a three-dimensional (3D) image of the anatomy of the patient P. As described in more detail herein, the image acquired by the TEE probe 130 depends on the insertion depth, rotation, and / or tilt of the imaging end effector 140.

[0075] The handle 132 is coupled to the proximal end of the member 131. The handle 132 includes control elements for navigating the imaging end effector 140 to a target pose. As shown, the handle 132 includes knobs 133 and 134 and a switch 135. The knob 133 causes the member 131 and the imaging end effector 140 to bend along the anterior-posterior plane of the patient P (e.g., the heart). The knob 134 causes the member 131 and the imaging end effector 140 to bend along the left-right plane of the patient P. The switch 135 controls beamforming (e.g., adjusting the angle of the imaging plane) performed at the imaging end effector 140.

[0076] In a manual navigation embodiment, a practitioner manually toggles the knobs 133 and 134 and / or manually opens and / or closes the switch 135 as needed to navigate the imaging end effector 140 to a target pose. The practitioner may receive a display of an image generated by the imaging end effector 140 and thereby apply a navigation force 33 (FIG. 1) to control the knobs 133 and 134 and / or the switch 135 on the handle 132.

[0077] In an automatic navigation embodiment, a robotic system (not shown) may include electrical and / or mechanical components (e.g., motors, rollers, and gears) configured to toggle the knobs 133 and 134 and / or open and / or close the switch 135, whereby the robotic controller 100 may receive a motion control command 31 (FIG. 1) from a navigation controller (not shown) or an input device (not shown) to control the knobs 133 and 134 and / or the switch 135 on the handle 132. Alternatively, the robotic controller 100 may be configured to directly manipulate the TEE probe 130 based on a guidance method implemented by the robotic controller 100 via an actuation signal 32 (FIG. 1).

[0078] The TEE probe 130 can be manipulated in various degrees of freedom. FIGS. 3C-3F illustrate various mechanisms for manipulating the TEE probe 130.

[0079] FIG. 3C is a schematic diagram illustrating that the TEE probe 130 can be manually advanced into the esophagus of a patient as shown by arrow 131b or withdrawn from the esophagus of the patient as shown by arrow 131c. The TEE probe 130 can be manually or automatically rotated counterclockwise (e.g., to the left) or clockwise (e.g., to the right) relative to the longitudinal axis 131a of the TEE probe 130 as shown by arrows 139a and 139b, respectively. The rotation of the member 131 can be described by a parameter represented as γ.

[0080] FIG. 3D is a schematic illustration of electronically rotating the TEE probe 130 from 0 degrees to 180 degrees (e.g., for beamforming) as shown by arrows 136a and 136b, e.g., by manual or robotic control of a switch 135 on the handle 132. Rotation of the imaging plane can be described by a parameter represented as ω.

[0081] FIG. 3E is a schematic illustration of bending the TEE probe 130 as shown by arrows 137a and 137b, e.g., along an anterior-posterior plane relative to the patient's heart, e.g., by manually or robotically toggling a knob 134 on the handle 132. Bending along the anterior-posterior plane can be described by a parameter represented as α.

[0082] FIG. 3F is a schematic illustration of bending the TEE probe 130 as shown by arrows 138a and 138b, e.g., along a left-right plane relative to the patient's heart, e.g., by manually or robotically toggling a knob 133 on the handle 132. Bending along the left-right plane can be described by a parameter represented as β.

[0083] As another example of an exemplary embodiment of the intervention device 30 (FIG. 1), FIG. 3G shows a transthoracic echocardiography (TTE) probe 240 configured to capture an ultrasonic image of the anatomy of a patient P from outside the body of the patient P and a robot 230 shown as being for manually or robotically manipulating the TTE probe 240 to reposition the TTE probe 240 to a target pose (i.e., the positioning and / or orientation of the TTE probe 240 relative to the patient P). More specifically, the robot 230 includes a plurality of links 231 coupled to a plurality of joints 232, the plurality of joints 232 being configured to hold the TTE probe 240 and manipulate the TTE probe 240 on the outer surface of the patient P (e.g., around the chest region for imaging the heart).

[0084] In a manual navigation embodiment, a practitioner manually applies a navigation force 33 (FIG. 1) to the link(s) 231, thereby translating, rotating, and / or pivoting the link(s) 231 of the robot 230 to navigate the imaging TTE probe 240 to the target pose. The practitioner may receive a display of the image generated by the TTE probe 240 to use as a basis for controlling the link(s) 231 of the robot 230.

[0085] In an automatic navigation embodiment, a robotic system (not shown) includes electrical and / or mechanical components (e.g., motors, rollers, and gears) configured to control the linkage 231 of the guiding robot 230, whereby the robot controller 101 receives a motion control command 32 (FIG. 1) in the form of Cartesian velocity parameters or joint velocity parameters from a navigation controller (not shown) or an input device (not shown) to control the linkage 231 of the guiding robot 230. Alternatively, the robot controller 101 may be configured to directly manipulate the TTE probe 240 via an actuation signal 32 (FIG. 1) based on a guiding method implemented by the robot controller 101.

[0086] As a further example of an exemplary embodiment of the access device 30 (FIG. 1A), FIG. 3H shows a shape-sensing guidewire 332 embedded in or attached to a continuum robot 331 having an end effector 340. The shape-sensing guidewire 332 incorporates optical shape sensing (OSS) technology known in the field of the present disclosure. More specifically, the shape-sensing controller 103 uses light from the multi-core optical fiber 333 along the guidewire 332 for device positioning and navigation during a surgical procedure. The principles involved utilize distributed strain measurements in an optical fiber using characteristic Rayleigh backscattering or a controlled grating pattern (e.g., a fiber Bragg grating). In practice, when the robot controller 102 or a practitioner (not shown) navigates the continuum robot 331 within the patient P to position the end effector 340 in a target pose, the shape-sensing controller 103 acquires the registered shape of the continuum robot 331 via the shape-sensing guidewire 332.

[0087] To further facilitate understanding of the present disclosure, the following descriptions of Figure 2A and Figure 2B teach exemplary embodiments of the continuous position controller of the present disclosure and the continuous positioning state machine of the present disclosure, respectively. In accordance with the descriptions of Figure 2A and Figure 2B a person of ordinary skill in the art of the present disclosure will recognize how to apply the present disclosure to make and use additional embodiments of the continuous position controller of the present disclosure and the continuous positioning state machine of the present disclosure.

[0088] Additionally, the TEE probe 130 (FIG. 3A), the robot 230 / TTE probe 240 (FIG. 3G), and the continuum robot 331 (FIG. 3H) are used herein as non-limiting examples of the access device 30 including the end effector 40 (FIG. 1) to support the description of various embodiments of the continuous position controller of the present disclosure. Nevertheless, a person of ordinary skill in the art of the present disclosure will recognize how to apply the present disclosure to a wide variety of additional embodiments of the access device 30 including the end effector 40.

[0089] Referring to FIGS. 1, Figure 2A and Figure 2B , the intervention device 30 of the present disclosure including the end effector 40 and the continuous positioning controller 50 represents a continuous positioning state machine 90.

[0090] In particular, as Figure 2B shown, the state S92 of the continuous positioning state machine 90 includes the navigation of the intervention device 30 including the end effector 40 according to a specific application (e.g., a minimally invasive procedure, a video-assisted thoracic surgery, a microvascular procedure, a micro medical diagnosis procedure, or a deformity correction procedure). In practice, the application may involve manual navigation positioning or automatic navigation positioning of the end effector 40 to a target pose, whereby the application incorporates imaging guidance (e.g., image segmentation, image registration, path planning, etc.), intervention device tracking (e.g., electromagnetic, optical, or shape sensing), and / or any other navigation technique for positioning the end effector 40 to the target pose.

[0091] The execution of the state S92 of the continuous positioning state machine 90 causes the generation of navigation data 34 and optionally the generation of auxiliary data 35. Generally, in practice, the navigation data 34 will be transmitted to / imposed on the intervention device 30 in the form of one or more navigation commands 31, one or more actuation signals 32, and / or one or more navigation forces 33, and the auxiliary data 35 will be in the form of one or more images of the intervention device 30 and / or the end effector 40, the operating characteristics of the intervention device 30 (e.g., shape, strain, twist, temperature, etc.), and the operating characteristics of the end effector 40 (e.g., pose, force, etc.).

[0092] In response thereto, the state S94 of the continuous positioning state machine 90 includes continuous positioning control by the continuous position controller 50 that navigates the intervention tool 30 and the end effector 40 according to the state S92. To this end, the continuous position controller 50 uses the forward prediction model 60 of the present disclosure, the inverse prediction model 70 of the present disclosure, and / or the imaging prediction model 80 of the present disclosure.

[0093] In practice, as will be further explained in the present disclosure, the forward prediction model 60 can be any type of machine learning model or equivalent (e.g., a neural network) suitable for regressing the positioning movement of the intervention device 30 to the navigation pose of the end effector 40 for a specific type of intervention device 30 used in a specific type of application being implemented, whereby the forward prediction model 60 is trained on the forward kinematics of the intervention device 30 predicting the navigation pose of the end effector 40.

[0094] In operation, the forward prediction model 60 inputs navigation data 34 (and auxiliary data 35, if transmitted) associated with manual or automatic navigation of the intervention device 30, thereby predicting the navigation pose of the end effector 40 corresponding to the navigation of the intervention device 30 and outputting continuous positioning data 51 regarding information on positioning the end effector 40 to a target pose by the intervention device 30 based on the predicted navigation pose of the end effector 40. The continuous positioning data 51 can be used in state S92 as a control operation to determine accuracy and / or perform recalibration of the manual or automatic navigation of the intervention device 30 to position the end effector 40 to the target pose.

[0095] In practice, as will be further explained in the present disclosure, the inverse prediction model 70 can be any type of machine learning model or equivalent (e.g., a neural network) suitable for a particular type of intervention device 30 used in a particular type of application being implemented for regressing the target pose of the end effector 40 to the positioning movement of the intervention device 30, whereby the inverse prediction model 60 is trained in the inverse kinematics of the intervention device 30 for predicting the positioning movement of the intervention device 30.

[0096] In operation, the inverse prediction model 70 inputs navigation data 34 (and auxiliary data 35, if transmitted) associated with the target pose of the end effector 40, thereby predicting the positioning movement of the intervention device 30 for positioning the end effector 40 to the target pose, and the inverse prediction model 70 outputs continuous positioning commands 52 for positioning the end effector 40 to the target pose by the intervention device 30 based on the predicted positioning movement of the intervention device 30. The continuous positioning commands 52 can be used in state S92 as a control operation to perform manual or automatic navigation of the intervention device 30 to position the end effector 40 to the target pose.

[0097] In practice, as will be further explained in the present disclosure, the imaging prediction model 80 can be any type of machine learning model or equivalent (e.g., a neural network or a scale-invariant feature transform network) suitable for a particular type of end effector 40 used in a particular type of application being implemented for regressing the relative imaging performed by the end effector 40 to the navigation pose of the end effector 40, whereby the inverse prediction model 60 is trained in the correlation between the relative imaging performed by the end effector 40 and the forward kinematics of the intervention device 30 for predicting the navigation pose of the end effector 40.

[0098] In operation, the imaging prediction model 60 inputs the auxiliary data 35 in the form of an image generated by the end effector 40 in one or more poses, thereby predicting the navigation pose of the end effector 40 as feedback data regarding the information that the intervention device 30 corrects the positioning of the end effector 40 to the target pose. The feedback data is used in the closed loop of state S94 to generate the difference between the target pose of the end effector 40 and the predicted navigation pose of the end effector 40, whereby the inverse prediction model 70 can input this difference to predict the correction positioning movement of the intervention device 30 to reposition the end effector 30 to the target pose.

[0099] In practice, embodiments of the continuous positioning controller 50 may use the forward prediction model 60, the inverse prediction model 70, and / or the imaging prediction model 80.

[0100] For example, an embodiment of the continuous positioning controller 50 may use only the forward prediction model 60 to facilitate the display of the accuracy of the manual or automatic navigation of the intervention device 30 to position the end effector 40 to the target pose.

[0101] More particularly, a user interface is provided to display an image of the attempted navigation of the end effector 40 to the target pose and an image of the predicted navigation pose of the end effector 40 obtained by the forward prediction model 60. The predicted confidence ratio is shown to the user. To evaluate the prediction uncertainty, as is known in the art of the present disclosure, multiple feed-forward iterations of the forward prediction model 60 are performed with dropout randomly enabled.

[0102] As another example, an embodiment of the continuous positioning controller 50 may use only the inverse prediction model 70 to command the manual or automatic navigation of the intervention device 30 to position the end effector 40 to the target pose.

[0103] As another example, an embodiment of the continuous positioning controller 50 may use only the imaging prediction model 80 to provide feedback data giving the following operation, the intervention device 30 corrects the positioning of the end effector 40 to the target pose.

[0104] As another example, an embodiment of the continuous positioning controller 50 may use the forward prediction model 60 and the inverse prediction model 70, thereby commanding the manual or automatic navigation of the intervention device 30 to position the end effector 40 to the target pose and displaying the accuracy of the manual or automatic navigation of the intervention device 30 to position the end effector 40 to the target pose.

[0105] As another example, an embodiment of the continuous positioning controller 50 can use an inverse prediction model 70 and an imaging prediction model 80 to command manual or automatic navigation of the intervention device 30 to position the end effector 40 in a target pose and provide feedback data that gives information on the following operation: correcting the positioning of the end effector 40 to the target pose by the intervention device 30.

[0106] To further facilitate understanding of the present disclosure, the following description of FIGS. 4-14 teaches exemplary embodiments of the forward prediction model, inverse prediction model, and imaging prediction model of the present disclosure. Based on the description of FIGS. 4-14, those of ordinary skill in the art of the present disclosure will recognize how to apply the present disclosure to make and use additional embodiments of the forward prediction model, inverse prediction model, and imaging prediction model of the present disclosure.

[0107] Figures 4A - 4E Illustrated is the training and application of the forward prediction model 60a of the present disclosure, which is trained on the forward kinematics of the intervention device 30 (FIG. 1) that predicts the navigation pose of the end effector 40 (FIG. 1), thereby facilitating the application of the forward prediction model 60a to the commanded positioning movement of the intervention device 30 during the intervention process to present the predicted navigation pose of the end effector 40. Thus, the continuous positioning controller 50 (FIG. 1) can generate continuous positioning data 51 regarding the information of positioning the end effector 40 to the target pose by the intervention device 30 based on the predicted navigation pose of the end effector 40.

[0108] More particularly, referring to Figure 4A , the training phase of the forward prediction model 60a involves a training controller (not shown) configured to interpret the ground truth training data set D as exemplarily taught in the subsequent description of FIGS. 16-18 of the present disclosure. The data set D consists of n sequences W of i data points represented by the 2-tuple d i =(T i , Q i ). The 2-tuple consists of the end effector pose (T∈SE(3)) 62a and a sequence (Q∈(q t , q t+1 …q t+j )) 61a of j successive joint variables collected at sequential time points starting from t to t + j. The entry q t represents all joint variables controlled by a robot controller (not shown).

[0109] In practice, the training dataset D is a collection of expert data that reasonably covers different navigations of the intervention device 30. To this end, the diverse dataset training dataset D for learning should include manufacturing differences between various types of robots, performance characteristics, wear of hardware components, and other system-related and system-unrelated factors (e.g., temperature or humidity of the environment in which the robot operates).

[0110] Reference Figure 4B , the application phase of the forward prediction model 60a involves the continuous positioning controller 50 using the feedforward prediction model 60a to execute a deep learning type algorithm to regress the commanded position movement of the intervention device 30 to the navigation pose of the end effector 40. The forward prediction model 60a is configured to infer the pose of the end effector 40 given a sequence Q of j successive joint variables 61b of the intervention device 30 (e.g., parameters α, β as shown in FIG. 3B).

[0111] In an embodiment as Figure 4E shown, the forward prediction model 60a uses a neural network library 160a that includes an input layer, a hidden layer, and an output layer, which are derived from a combination of one or more fully connected layers (FCLs) 163a, one or more convolutional layers (CNLs) 164a, one or more recurrent layers (RCLs) 165a, and one or more long short-term memory (LSTM) layers 166a.

[0112] In practice, the combination of these layers is configured to implement the regression of the joint variable Q to the pose .

[0113] In one embodiment for implementing the regression of the joint variable Q to the pose , the neural network library 160a includes a set of N fully connected layers 163a.

[0114] In a second embodiment for implementing the regression of the joint variable Q to the pose , the neural network library 160a includes a set of N convolutional layers 164a, followed by a set of M fully connected layers 163a or a set of W recurrent layers 165a or a set of W long short-term memory layers 166a.

[0115] In a third embodiment for implementing the regression of the joint variable Q to the pose , the neural network library 160a includes a set of N convolutional layers 164a, followed by a combination of a set of M fully connected layers 163a and a set of W recurrent layers 165a or a set of W long short-term memory layers 166a.

[0116] In practice, the fully connected layer 163a may include K neurons, where N, M, W, and K can be any positive integers, and the values can vary according to the embodiments. For example, N can be approximately 8, M can be approximately 2, W can be approximately 2, and K can be approximately 1000. Moreover, the convolutional layer 164a can perform a non-linear transformation, which can be a composite function of operations (such as batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 164a can also include a non-linear function (such as including the rectified non-linear ReLU operation) configured to extract the modified feature map.

[0117] Additionally, in practice, one of the layers 163a or 164a serves as the input layer for the sequence 161a of input joint variables Q, whereby the size of the sequence of joint variables Q can be ≥1, and one of the layers 163a, 165a, and 166a can serve as the output layer for the pose 162a of the end effector 40 in the Cartesian space (e.g., the translation and rotation of the end effector 40 in the Cartesian space). The output pose of the end effector 40 in the Cartesian space can be represented as a vector parameterization and / or non-vector parameterization of the position and orientation of a rigid body. More particularly, the parameterization can be in the form of Euler angles, quaternions, matrices, exponential maps, and / or angle axes representing rotation and / or translation (e.g., including the direction and magnitude of translation).

[0118] Moreover, in practice, the output layer can be the non-linear fully connected layer 163a, which gradually shrinks the high-dimensional output of the last convolutional layer 164a of the neural network library 160a to produce a set of output variables.

[0119] During training, by comparing the output inferred by the forward prediction model (given the input sequence Q) i with the ground truth end effector pose T

[0120] from a batch of training data set D

[0121] Still referring to Figure 4E, depending on the complexity of the task, the neural architecture of an exemplary embodiment of the forward prediction model 60a has a unique number of layers, which is specified in the output layer 169a that is defined to predict the pose of the end effector 40 given a sequence Q of j successive joint variables 61a The loss function for training such a neural architecture can be defined as the sum of the Euclidean distances between the translational component and the rotational component. For example,

[0122] Reference Figure 4C and Figure 4D , stage S92a of the exemplary intervention process 90a uses the TEE probe 130 (FIG. 3A) and the robot controller 100 (FIG. 3B), and includes the robot controller 100 receiving the position of the TEE probe 130 as a joint variable and sending a movement signal to the handle 132 (FIG. 3A) of the TEE probe 130. By using an electric knob, the handle 132 tightens / relaxes the tether of the TEE probe 130, which causes the end effector 140 (FIGS. 3C - 3F) to move to the target pose. In practice, the position of the TEE probe 130 as a joint variable can be indicated by the user or an external tracking device or guiding system.

[0123] Stage S94a of process 90a involves the robot controller 100, the forward prediction model 50a, and the display controller 104. The robot controller 100 stores the sequence (Q) of successive joint variables 61a and transmits it to the forward prediction model 60a that predicts the navigation pose of the end effector 140 The continuous positioning controller 50a generates a prediction confidence ratio derived from the uncertainty of multiple feed - forward iterations of the forward prediction model 60a, and such multiple feed - forward iterations of the forward prediction model 60a are performed with randomly enabled dropout as is known in the art of the present disclosure. The forward prediction model 50a transmits the continuous positioning data 51a including the predicted navigation pose of the end effector 140 and the confidence ratio to the display controller 104, and the display controller 104 in turn controls the display of the image 105a of the navigation pose of the end effector 140, the image 106a of the navigation pose of the end effector 140, and the confidence ratio for the guiding purpose of guiding the end effector 140 to the target pose.

[0124] Figures 5A - 5EIllustrated is the training and application of the inverse prediction model 70a of the present disclosure, which is trained on the inverse kinematics of the intervention device 30 (FIG. 1) that predicts the positioning movement of the intervention device 30, so as to facilitate the application of the inverse prediction model 70a to the target pose (T) 71a of the end effector 40 during the intervention process to present the joint variable movement (q) 72t of the intervention device 30, whereby the continuous positioning controller 50 (FIG. 1) can generate a continuous positioning command for repositioning the end effector 40 to the target pose by the intervention device 30 based on the predicted positioning movement of the intervention device 40.

[0125] More specifically, referring to Figure 5A , the training phase of the inverse prediction model 70a involves a training controller (not shown) configured to interpret the ground truth training dataset D as exemplarily taught in the description of Figures 17 - 18 . The dataset D consists of n sequences W containing i data points represented by the 2-tuple d i = (T i , Q i ). The 2-tuple consists of the end effector pose (T ∈ SE(3)) 71a and a sequence (Q ∈ (q t , q t+1 ... q t+j )) 72a of j successive joint variables collected at sequential time points starting from t to t + j. The entry q t represents all joint variables controlled by a robot controller (not shown). The variable j can also be equal to 1, which means that the prediction model will be trained to infer a single set of joint variables.

[0126] In practice, the training dataset D is a collection of expert data that reasonably covers different navigations of the intervention device 30. To this end, the diverse dataset training dataset D for learning should include mechanical differences between various types of robots, wear of hardware components, and other system-related factors.

[0127] Referring to Figure 5B , the application phase of the inverse prediction model 70a involves the continuous positioning controller 50 using the inverse prediction model 70a to perform a deep learning type algorithm for motion regression. The inverse prediction model 70a is configured to infer a sequence t of j successive joint variables 72 (e.g., the parameters α, β as shown in FIG. 3B) to reach the pose (T ∈ SE(3)) 71t of the end effector 40.

[0128] In as Figure 5EIn one illustrated embodiment, the inverse prediction model 70a uses a neural network library 170a, which includes an input layer, a hidden layer, and an output layer, and these layers are derived from a combination of one or more fully connected layers (FCLs) 173a, one or more convolutional layers (CNLs) 174a, one or more recurrent layers (RCLs) 175a, and one or more long short-term memory (LSTM) layers 176a.

[0129] In practice, the combination of these layers is configured to implement the regression of pose T to joint variables .

[0130] In one embodiment for implementing the regression of pose T to joint variables , the neural network library 170a includes a set of N fully connected layers 173a.

[0131] In a second embodiment for implementing the regression of pose T to joint variables , the neural network library 170a includes a set of N convolutional layers 174a, followed by a set of M fully connected layers 173a or a set of W recurrent layers 175a or a set of W long short-term memory layers 176a.

[0132] In a third embodiment for implementing the regression of pose T to joint variables , the neural network library 170a includes a set of N convolutional layers 174a, followed by a combination of a set of M fully connected layers 173a and a set of W recurrent layers 175a or a set of W long short-term memory layers 176a.

[0133] In practice, the fully connected layer 173a may include K neurons, where N, M, W, and K can be any positive integers, and the values can vary according to the embodiment. For example, N can be approximately 8, M can be approximately 2, W can be approximately 2, and K can be approximately 1000. Moreover, the convolutional layer 174a can perform a non-linear transformation, which can be a composite function of operations (such as batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 174a can also include a non-linear function (such as including a rectified non-linear ReLU operation) configured to extract modified feature maps.

[0134] Additionally, in practice, one of layers 173a or 174a serves as an input layer for the pose 171a of the input end effector 40 in Cartesian space (e.g., translation and rotation of the end effector 40 in Cartesian space), and one of layers 173a, 175a, and 176a can serve as an output layer for the sequence 172a of output joint variables Q, whereby the size of the sequence of joint variables Q can be ≥1. The input pose of the end effector 40 in Cartesian space can be represented as a vector parameterization and / or non-vector parameterization of the position and orientation of a rigid body. More particularly, the parameterization can be in the form of Euler angles, quaternions, matrices, exponential maps, and / or angle axes representing rotation and / or translation (e.g., including the direction and magnitude of translation).

[0135] Moreover, in practice, the output layer can be a non-linear fully connected layer 173a that progressively shrinks the high-dimensional output of the last convolutional layer 174a of the neural network library 170a to produce a set of output variables.

[0136] During training, by comparing the output from the inverse prediction model (with the given ground truth end effector pose T as input) i with the ground truth sequence Q from a batch of training dataset D

[0137] which can be systematically selected or randomly selected from a data memory (not shown), the training weights of the inverse prediction model 70a are continuously updated. More particularly, the coefficients of the filter can be initialized with predefined values or arbitrary values. The coefficients of the filter are applied to this batch of training dataset D via forward propagation and are adjusted via backpropagation to minimize any output error. In application, the inverse prediction model 70a infers a sequence of j successive joint variables 72a (e.g., parameters α, β as shown in FIG. 3B)

[0138] Still referring to Figure 5E , depending on the complexity of the task, the neural architecture of the exemplary embodiment of the inverse prediction model 70a as shown has a unique number of layers, which is specified in the output layer 179a that is defined to predict a sequence of j successive joint variables 72b (e.g., parameters α, β as shown in FIG. 3B) to reach the pose (T∈SE(3)) 71b of the end effector 40. The mean squared error (MSE) can be used as the loss function.

[0139] Referring to Figure 5C and Figure 5D, stage S92b of the exemplary intervention process 90b uses the TEE probe 130 (FIG. 3A) and the navigation controller 103, and includes the navigation controller 103 determining the target pose of the end effector 140 and sending the target pose 71a to the inverse prediction model 70a. In practice, the navigation controller 130 may implement any known guidance algorithm known in the field of the present disclosure.

[0140] Stage S94A of process 90b involves the inverse prediction model 70a and the robot controller 100. The inverse prediction model 70a infers a sequence of j successive joint variables 72a (e.g., parameters α, β as shown in FIG. 3B) to reach the pose (T ∈ SE(3)) 71a of the end effector 40, and the continuous positioning controller 50b communicates with the command controller 104 to control the intervention device 30 via the end effector 40 to position the end effector 50 to the target pose based on the predicted positioning movement of the intervention device 40.

[0141] Figures 6A - 6E Illustrated is the training and application of the forward prediction model 60b of the present disclosure, which is trained on the forward kinematics of the intervention device 30 (FIG. 1) predicting the navigation pose of the end effector 40 (FIG. 1), so as to facilitate the application of the forward prediction model 60a to the command n vector of the joint velocity of the intervention device 30 during the intervention process to present the predicted linear velocity and / or angular velocity of the end effector 40, whereby the continuous positioning controller 50 (FIG. 1) can generate continuous positioning data 51 regarding the information of repositioning the end effector 40 to the target pose by the intervention device 30 based on the predicted linear velocity and / or angular velocity of the end effector 40.

[0142] More particularly, referring to Figure 6A , the training stage of the forward prediction model 60b involves a training controller (not shown) configured to interpret the ground truth training data set D as exemplarily taught in FIGS. 16 - 18. The data set D consists of n sequences W containing i data points represented by the 2 - tuple The 2 - tuple consists of a sequence of j successive joint velocities 61b and the linear velocity and / or angular velocity of the end effector The entry represents all joint variables controlled by a robot controller (not shown). The sequence may also contain only one entry.

[0143] In practice, the training data set D is a collection of expert data that reasonably covers different navigations of the intervention device 30. To this end, the diverse data set D for learning should include mechanical differences between various types of robots, wear of hardware components, and other system - related factors.

[0144] Reference Figure 6B In the application phase of the forward prediction model 60b, the continuous positioning controller 50 uses the feed-forward prediction model 60b to perform a deep learning type algorithm for end effector motion regression. The forward prediction model 60b is configured to infer the linear velocity and / or angular velocity 62b of the end effector 40 given an n-vector of the joint velocity 61b of the intervening device 30.

[0145] In one embodiment as shown Figure 6E The forward prediction model 60b uses a neural network library 160b, which includes an input layer, a hidden layer, and an output layer, and these layers are derived from a combination of one or more fully connected layers (FCLs) 163b, one or more convolutional layers (CNLs) 164b, one or more recurrent layers (RCLs) 165b, and one or more long short-term memory (LSTM) layers 166b.

[0146] In practice, the combination of these layers is configured to implement the regression of the joint velocity of the intervening device 30 to the linear velocity and / or angular velocity of the end effector 40.

[0147] In one embodiment for implementing the regression of the joint velocity of the intervening device 30 to the linear velocity and / or angular velocity of the end effector 40, the neural network library 160b includes a set of N fully connected layers 163b.

[0148] In a second embodiment for implementing the regression of the joint velocity of the intervening device 30 to the linear velocity and / or angular velocity of the end effector 40, the neural network library 160b includes a set of N convolutional layers 164b, followed by a set of M fully connected layers 163b or a set of W recurrent layers 165b or a set of W long short-term memory layers 166b.

[0149] In a third embodiment for implementing the regression of the joint velocity of the intervening device 30 to the linear velocity and / or angular velocity of the end effector 40, the neural network library 160b includes a set of N convolutional layers 164b, followed by a combination of a set of M fully connected layers 163b and a set of W recurrent layers 165b or a set of W long short-term memory layers 166b.

[0150] In practice, the fully connected layer 163b may include K neurons, where N, M, W, and K can be any positive integers, and the values may vary according to the embodiments. For example, N can be approximately 8, M can be approximately 2, W can be approximately 2, and K can be approximately 1000. Moreover, the convolutional layer 164b may perform a non-linear transformation, which may be a composite function of operations (such as batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 164b may also include a non-linear function (such as including a rectified non-linear ReLU operation) configured to extract a modified feature map.

[0151] Additionally, in practice, one of the layers 163b or 164b serves as an input layer for a sequence of j successive joint velocities such that the size of the sequence of j successive joint velocities can be ≥1, and one of the layers 163b, 165b, and 166b can serve as an output layer that outputs the linear and angular velocities of the end effector regressed according to the last fully connected layer (e.g., 6 units, 3 units for linear velocity, and 3 units for angular velocity) using a linear or non-linear activation function.

[0152] During training, the predicted linear and angular velocities obtained via the forward velocity prediction model (in the case of a given sequence of joint velocities) are compared with the linear and / or angular velocities 62b from a batch of training data set D (which can be systematically selected or randomly selected from a data memory (not shown)) to continuously update the training weights of the forward prediction model 60b. More specifically, the coefficients of the filter can be initialized with predefined values or arbitrary values. The coefficients of the filter are applied to this batch of training data set D via forward propagation and are adjusted via backpropagation to minimize any output error.

[0153] In an application, the forward prediction model 60b infers the linear and / or angular velocities 62b of the end effector 40 in the case of a given sequence of joint velocities 61b of the intervening device 30.

[0154] Still referring to Figure 6E , the neural architecture consists of an input, the previously described neural network library, and an output. The input is a sequence of joint velocities, and the output is the linear and angular velocities that can be regressed according to a fully connected layer with 6 units. The loss function can be the MSE listed below:

[0155]

[0156] Referring to Figure 6C andFigure 6D , stage S92c of the exemplary interventional procedure 90c uses the robot 230 (FIG. 3A), the TTE probe 240 (FIG. 3A), and the robot controller 101 (FIG. 3G), and includes the robot controller 101 receiving the position of the TTE probe 240 as an n-vector of the joint velocities 61b of the interventional device 30 and sending the n-vector of the joint velocities 61b of the interventional device 30 to the robot 230. In practice, the position of the TTE probe 240 as the n-vector of the joint velocities 61b of the interventional device 30 may be indicated by a user or an external tracking device or guidance system.

[0157] Stage S94c of process 90c involves the robotic controller 101, the forward prediction model 50a, and the display controller 104. The robotic controller 101 stores the n-vector of the joint velocities 61b of the interventional device 30 and transmits it to the forward prediction model 60b that predicts the linear velocity and / or angular velocity 62b of the TTE probe 240. The continuous positioning controller 50c generates a confidence ratio of the prediction derived from the uncertainty of multiple feed-forward iterations of the forward prediction model 60b, which are performed with randomly enabled dropout as is known in the art of the present disclosure. The forward prediction model 60b transmits continuous positioning data 51b to the display controller 104, the continuous positioning data 51b including the predicted navigation pose of the TTE probe 240 derived from the predicted linear velocity and / or angular velocity 62b of the TTE probe 240. And also including the confidence ratio, the display controller 104 then controls the display of the image 105 a of the navigation posture of the TTE probe 240 , the image 106 a of the navigation posture of the TTE probe 240 , and the confidence ratio for the guidance purpose of guiding the end effector 240 to the target posture.

[0158] Figures 7A - 7E The diagram illustrates the training and application of the inverse prediction model 70b of the present disclosure, which is trained on the inverse kinematics of the interventional device 30 (Figure 1) that predicts the positioning motion of the interventional device 30, thereby facilitating the application of the inverse prediction model 70b to the target posture of the end effector 40 during the interventional process to present the predicted positioning motion of the interventional device 30, whereby the continuous positioning controller 50 (Figure 1) can generate continuous positioning commands that control the interventional device 30 to reposition the end effector 40 to the target posture based on the predicted positioning motion of the interventional device 40.

[0159] More specifically, refer to Figure 7A The training phase of the inverse prediction model 70b involves a training controller (not shown) being configured as in Figures 17 - 18 The actual training data set D is interpreted as exemplarily taught in the description of FIG. The data set D consists of two tuples composed of n sequences W of i data points represented. The 2-tuple consists of the linear and / or angular velocity of the end effector and successive joint variables collected at sequential time points from t to t + j consisting of the sequence of represents all joint variables controlled by a robot controller (not shown).

[0160] In practice, the training dataset D is a collection of expert data that reasonably covers different navigations of the intervention device 30. To this end, the diverse dataset training dataset D for learning should include mechanical differences between various types of robots, wear of hardware components, and other system-related factors.

[0161] Reference Figure 7B , the application phase of the inverse prediction model 70b involves the continuous positioning controller 50 using the inverse prediction model 70b to perform a deep learning type algorithm for joint velocity regression. The inverse prediction model 70b is configured to infer an n-vector of joint velocities 61b of the intervention device 30 given the linear and / or angular velocity 62b of the end effector 40.

[0162] In one embodiment as shown in Figure 7E , the inverse prediction model 70b uses a neural network library 170b that includes an input layer, a hidden layer, and an output layer, which are derived from a combination of one or more fully connected layers (FCLs) 173b, one or more convolutional layers (CNLs) 174b, one or more recurrent layers (RCLs) 175b, and one or more long short-term memory (LSTM) layers 176b.

[0163] In practice, the combination of these layers is configured to implement the regression of the linear and / or angular velocity of the end effector 40 to the joint velocity of the intervention device 30.

[0164] In one embodiment for implementing the regression of the linear and / or angular velocity of the end effector 40 to the joint velocity of the intervention device 30, the neural network library 170b includes a set of N fully connected layers 173b.

[0165] In a second embodiment for implementing the regression of the linear and / or angular velocity of the end effector 40 to the joint velocity of the intervention device 30, the neural network library 170b includes a set of N convolutional layers 174b, followed by a set of M fully connected layers 173b or a set of W recurrent layers 175b or a set of W long short-term memory layers 176b.

[0166] In a third embodiment for implementing the regression of the linear velocity and / or angular velocity of the end effector 40 to the joint velocities of the intervention device 30, the neural network library 170b includes a set of N convolutional layers 174b, followed by a combination of a set of M fully connected layers 173b and a set of W recurrent layers 175b or a set of W long short-term memory layers 176b.

[0167] In practice, the fully connected layer 173b may include K neurons, where N, M, W, and K can be any positive integers, and the values can vary according to the embodiment. For example, N can be approximately 8, M can be approximately 2, W can be approximately 2, and K can be approximately 1000. Moreover, the convolutional layer 174b can implement a non-linear transformation, which can be a composite function of operations (such as batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 174b can also include a non-linear function (such as including the rectified non-linear ReLU operation) configured to extract a modified feature map.

[0168] Additionally, in practice, one of the layers 173b or 174b serves as an input layer for the input angular velocity and linear velocity and one of the layers 173b, 175b, and 176b can serve as an output layer that outputs a sequence of j successive joint velocities provided as the output from the LSTM layer 176b Alternatively, a single joint velocity can be regressed based on the fully connected layer 173b consisting of m units, each unit corresponding to each connection in the robot controlled by the robot controller. The fully connected layer 173b can have a linear or non-linear activation function.

[0169] During training, the training weights of the inverse prediction model 70b are continuously updated by comparing the predicted sequence of joint velocities (in the case of the linear velocity and angular velocity at a given input) with the ground truth sequence of joint velocities from a batch of training data set D (which can be systematically selected or randomly selected from a data memory (not shown)). More particularly, the coefficients of the filter can be initialized with predefined values or arbitrary values. The coefficients of the filter are applied to this batch of training data set D via forward propagation and are adjusted via backpropagation to minimize any output error.

[0170] In an application, the inverse prediction model 70b infers an n-vector of the joint velocities 61b of the intervention device 30 given the linear velocity and / or angular velocity 62b of the end effector 40.

[0171] Still referring to Figure 7E, the neural architecture of an exemplary embodiment of the inverse velocity model consists of an input, a neural network library, and an output. The input is the angular velocity and the linear velocity and the output is a sequence of joint velocities provided as the output from the LSTM layer Alternatively, a single joint velocity can be regressed based on a fully connected layer consisting of m units, each unit corresponding to each connection in the robot controlled by the robot controller. The fully connected layer can have a linear or non-linear activation function.

[0172] Reference Figure 7C and Figure 7D , in stage S92d of the exemplary intervention process 90d, the TTE probe 240 (FIG. 3G) and the navigation controller 103 are used, and it includes the navigation controller 103 determining the linear velocity and / or angular velocity 62b of the TTE probe 240 to the target pose and sending the linear velocity and / or angular velocity 62b to the inverse prediction model 70b. In practice, the navigation controller 130 can implement any known guidance algorithm known in the field of the present disclosure.

[0173] Stage S94d of process 90d involves the inverse prediction model 70b and the robot controller 101. The inverse prediction model 70b infers an n-vector of the joint velocities 61b of the intervention device 30 given the linear velocity and / or angular velocity 62b, and the continuous positioning controller 50c transmits a continuous positioning command 52b to the robot controller 101, thereby controlling the positioning of the TTE probe 240 to the target pose via the robot 230 (FIG. 3G) based on the predicted positioning movement of the intervention device 40.

[0174] In practice, the forward prediction model 60a ( Figure 4A ), the inverse prediction model 70a ( Figure 5A ), the forward prediction model 60b (FIG. 64A), and the inverse prediction model 70b ( Figure 7A ) can use additional auxiliary information, such as images of anatomical structures (e.g., ultrasound images, endoscopic images, or X-ray images), forces measured at the end effector, and the shape of the robot. Depending on the application, other inputs can also be included, including information from spectral tissue sensing devices, ECG signals or EEG signals, tissue conductivity, or other physiological signals. For example, if a continuum robot operates inside the human heart, the features available in the ultrasound image and the electrophysiological signals can improve the positioning of the end effector relative to the anatomical structure, thus improving the guidance accuracy.

[0175] Reference Figure 8A and Figure 8BIn addition to being shown as being trained on the forward kinematics of the interventional device, the forward prediction model 60c is shown as being trained on the sequence Q of joint variables 61a and the end effector pose T 62a. In addition, the forward prediction model 60c is shown as being trained on the shape 35a, image 35b and force 35c of the interventional device. Therefore, in application, the forward prediction model 60c will be able to predict the navigation pose of the end effector based on the sequence Q of joint variables 61a and the shape 35a, image 35b and force 35c of the interventional device.

[0176] A person of ordinary skill in the art will know how to apply the shape 35a, image 35b and force 35c of the interventional device and any other additional auxiliary information to the inverse prediction model 70a, the forward prediction model 60b and the inverse prediction model 70b.

[0177] Figures 9A - 9E The diagram illustrates the training and application of the forward prediction model 60d of the present disclosure, which is trained on the forward kinematics of the interventional device 30 (Figure 1) that predicts the navigation pose of the end effector 40 (Figure 1) and the shape of the robot, thereby facilitating the application of the forward prediction model 60d to a sequence of successive shapes of the interventional device 30 with embedded OSS technology during the interventional process to present the predicted navigation pose and shape of the end effector 40, whereby the continuous positioning controller 50 (Figure 1) can generate continuous positioning data 51c of information about repositioning the end effector 40 to a target pose by the interventional device 30 based on the predicted navigation pose of the end effector 40.

[0178] More specifically, refer to Figure 9A In the training phase, a training controller (not shown) is configured to interpret a real-world training data set D as exemplarily taught in the description of FIGS. 16-18 . The data set consists of a 2-tuple d i =(H i , H i+1 ) represents n sequences W of i data points. This 2-tuple consists of a sequence of k consecutive shapes 61d (H i ∈(h t ,h t+1 …h t+k ), where h∈(p1…p m ) is a set of m vectors p m , which describes both the position of the OSS sensor (e.g., shape sensing guidewire) embedded in the interventional device 30 in 3D Euclidean space and auxiliary shape parameters (e.g., strain, curvature, and twist). This 2-tuple is also composed of a future time point h t+k+1 A sequence H of k consecutive shapes 62b i+1 ∈(ht+1 , h t+2 …h t+k+1 ) consists of, where h ∈ (p1...p m ) is a set of m vectors p m , which describes both the position and auxiliary shape parameters (e.g., strain, curvature, and twist) of the OSS intervention device 30 in 3D Euclidean space.

[0179] In practice, the training dataset D is a collection of expert data that reasonably covers different navigations of the OSS intervention device 30. To this end, the diverse dataset for learning, the training dataset D, should include anatomical structures with different curvatures, motion amplitudes, mechanical differences between various types of robots, wear of hardware components, and other system-related factors (e.g., temperature and humidity of the environment).

[0180] Reference Figure 9B , the application phase of the forward prediction model 60d involves the continuous positioning controller 50d using the forward prediction model 60d to execute a deep learning type algorithm, where the recurrent layer is trained on expert data that reasonably covers different situations. The diverse dataset for learning includes various working conditions (temperature, fiber bending, etc.), different operating motions of the device, and differences in hardware (fibers, interrogators, etc.).

[0181] In one embodiment as Figure 9E shown, the forward prediction model 60d uses a neural architecture that sequentially includes an input layer 163a, a sequence-to-sequence model 263a, an output layer 262a, and an extraction layer 264. This neural architecture is a deep convolutional neural network with a recurrent layer that is configured to infer a future sequence consisting of k shapes Therefore, the last shape in the predicted sequence is used to estimate the position of the OSS intervention device 30 at a future time point.

[0182] In training, by comparing the sequence of future shapes i (predicted by the model given the input sequence H ) with the ground truth future sequence H i+1 from the training dataset D (which can be systematically selected or randomly selected from a data memory (not shown)), the training weights of the forward prediction model 60d are continuously updated. More specifically, the coefficients of the filter can be initialized with predefined values or arbitrary values. The coefficients of the filter are applied to this batch of the training dataset D via forward propagation and adjusted via backpropagation to minimize any output error.

[0183] In application, the forward prediction model 60d infers a future sequence consisting of k shapes Therefore, the last shape in the prediction sequence is used to estimate the position of the OSS intervention device 30 at a future time point.

[0184] In an alternative embodiment as shown Figure 9F the forward prediction model 60d uses a many-to-one model 263b instead of the sequence-to-sequence model 263a, whereby the last layer will be the last shape

[0185] Refer to Figure 9C and Figure 9D , in stage S92e of the exemplary intervention process 90e, the OSS guidewire 332 (FIG. 3F) and the shape sensing controller 103 (FIG. 3F) are used, and it includes the shape sensing controller 103 measuring and storing the shape of the OSS guidewire 332 during the navigation of the end effector 340 to the target pose.

[0186] Stage S94e of process 90e involves the shape sensing controller 103, the forward prediction model 50d, and the display controller 104. The shape sensing controller 103 transmits a sequence 61d of k successive shapes to the forward prediction model 60e to infer the next shape sequence wherein the last shape is the predicted position of the OSS guidewire 332. The display controller 104 controls the display of the sensed position image 105a of the OSS guidewire 332 for the purpose of guiding the end effector 340 to the target pose.

[0187] Refer to Figure 11A and Figure 11B , the imaging prediction model 80a of the present disclosure is trained on expert data that reasonably covers different situations. Given an ultrasound image 81a, the neural network 80a infers the relative position 82a of the image 81a with respect to a reference anatomical structure. As a result, as will be further described herein, the movement between the previous pose of the end effector in an anatomical structure (e.g., the heart) and the current pose of the end effector in the anatomical structure (e.g., the heart) can be calculated.

[0188] Figure 11A Batch training of the imaging prediction model 80a is shown. During training, the network continuously updates the weights of the imaging prediction model 80a using tuples from a ground truth dataset consisting of ultrasound images and the relative positions of the images with respect to a reference anatomical structure.

[0189] Figure 11BShows real-time inference using the imaging prediction model 80a given an image 81a, which predicts the relative pose 82a of the end effector n with respect to a reference anatomical structure (e.g., a reference ultrasound image).

[0190] During the training phase, a data acquisition controller (not shown) is configured to receive and interpret information from the robot and the end effector (e.g., an ultrasound device) and store the data on a data storage medium (not shown) in a format defined by the following specification: The training dataset D consists of i data points represented by the 2-tuple d i =(U i , T i ). The 2-tuple consists of the ultrasound image U i 81a acquired at a certain position T∈SE(3)82a with respect to the reference position.

[0191] The training controller is configured to interpret the training dataset D stored on the data storage medium. The dataset D consists of i data points represented by the 2-tuple d i =(U i , T i ). The 2-tuple consists of: the ultrasound image U i 81a of the anatomical structure and the relative motion T i 82a between the current pose of the end effector when acquiring the ultrasound image U i and an arbitrarily selected reference position.

[0192] In one embodiment as shown in FIG. 11E, the image prediction model 80a uses a neural network library 180a, which includes an input layer, a hidden layer, and an output layer, and these layers are derived from a combination of one or more fully connected layers (FCL) 183a, one or more convolutional layers (CNL) 184a, one or more recurrent layers (RCL) 185a, and one or more long short-term memory (LSTM) layers 186a.

[0193] In practice, the combination of these layers is configured to implement the relative positioning of the image U C to the reference image and thus achieve the pose

[0194] In one embodiment for implementing the relative positioning of the image U C to the reference image and thus achieve the pose , the neural network library 180a includes a set of N fully connected layers 183a.

[0195] In one embodiment for implementing the relative positioning of the image U CRelative positioning to a reference image and thus achieving pose In a second embodiment, the neural network library 180a includes a set of N convolutional layers 184a, followed by a set of M fully connected layers 183a or a set of W recurrent layers 185a or a set of W long short-term memory layers 186a.

[0196] In the case of implementing the image U C Relative positioning to a reference image and thus achieving pose In a third embodiment, the neural network library 180a includes a set of N convolutional layers 184a, followed by a combination of a set of M fully connected layers 183a and a set of W recurrent layers 185a or a set of W long short-term memory layers 186a.

[0197] In practice, the fully connected layer 183a may include K neurons, where N, M, W, and K can be any positive integers, and the values can vary according to the embodiment. For example, N can be approximately 8, M can be approximately 2, W can be approximately 2, and K can be approximately 1000. Moreover, the convolutional layer 184a can implement a non-linear transformation, which can be a composite function of operations (such as batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 184a can also include a non-linear function (such as including a rectified non-linear ReLU operation) configured to extract a modified feature map.

[0198] Additionally, in practice, one of the layers 183a or 184a serves as an input layer for the input image U C and one of the layers 183a, 185a, and 186a can serve as an output layer for the pose 182a of the end effector 40 in Cartesian space (e.g., translation and rotation of the end effector 40 in Cartesian space). The output pose of the end effector 40 in Cartesian space can be represented as a vector parameterization and / or non-vector parameterization of the position and orientation of a rigid body. More particularly, the parameterization can be in the form of Euler angles, quaternions, matrices, exponential maps, and / or angle axes representing rotation and / or translation (e.g., including the direction and magnitude of translation).

[0199] Moreover, in practice, the output layer can be a non-linear fully connected layer 183a, which gradually reduces the high-dimensional output of the last convolutional layer 184a of the neural network library 180a to produce a set of output variables.

[0200] During training, by using the predicted relative motion of the end effector relative to a certain reference anatomical structure obtained by using the image prediction model (given the ultrasound image 161c and using it as an input) The training weights of the image prediction model 80a are continuously updated by comparing with the ground truth T of a batch of training data set D (which can be systematically or randomly selected from a data memory (not shown)). It can be systematically or randomly selected from a data memory (not shown). More specifically, the coefficients of the filter can be initialized with predefined values or arbitrary values. The coefficients of the filter are applied to this batch of training data set D via forward propagation and adjusted via backpropagation to minimize any output error.

[0201] Reference Figure 12A and Figure 12B , the imaging prediction model 80ba of the present disclosure is trained on expert data that reasonably covers different situations. Given an ultrasound image 81a, the neural network 80ba infers the relative position 82a of the image 81a with respect to a reference anatomical structure. As a result, as will be further described herein, the movement between the previous pose of the end effector in the anatomical structure (e.g., the heart) and the current pose of the end effector in the anatomical structure (e.g., the heart) can be calculated.

[0202] Figure 12A Shows the batch training of the imaging prediction model 80ba. During training, the network continuously updates the weights of the imaging prediction model 80ba using 2-tuples from the ground truth data set, which consists of an ultrasound image and the relative position of the image with respect to a reference anatomical structure.

[0203] Figure 12B Shows the real-time inference using the imaging prediction model 80ba given an image 81a, which predicts the linear and angular velocities 83a of the end effector with respect to a reference anatomical structure (e.g., a reference ultrasound image).

[0204] In the training phase, a data acquisition controller (not shown) is configured to receive and interpret information from the robot and the end effector (e.g., an ultrasound device) and store the data in a data storage medium (not shown) in a format defined by the following specification via the data acquisition controller (not shown): The training data set D consists of i data points represented by the 2-tuple d i =(U i , V i ). The 2-tuple consists of an ultrasound image U i 81a acquired at a certain position T∈SE(3)82a with respect to a reference position obtained from the vector 83a of the linear and angular velocities of the end effector.

[0205] The training controller is configured to interpret the training data set D stored on the data storage medium. The data set D consists of the 2-tuple d i =(Ui , V i ), which consists of i data points represented by: the ultrasonic image U of the anatomical structure i 81a and the linear and angular velocities of the end effector at the time of acquiring the ultrasonic image U i and the relative n vector 83a of the reference position selected arbitrarily.

[0206] In one embodiment shown in FIG. 12E, the image prediction model 80b uses a neural network library 180b, which includes an input layer, a hidden layer, and an output layer, and these layers are derived from a combination of one or more fully connected layers (FCLs) 183b, one or more convolutional layers (CNLs) 184b, one or more recurrent layers (RCLs) 185b, and one or more long short-term memory (LSTM) layers 186b.

[0207] In practice, the combination of these layers is configured to implement the relative positioning of the image U C to the reference image and thereby derive the linear velocity and / or angular velocity of the end effector 40.

[0208] In one embodiment for implementing the relative positioning of the image U C to the reference image and thereby deriving the linear velocity and / or angular velocity of the end effector 40, the neural network library 180b includes a set of N fully connected layers 183b.

[0209] In a second embodiment for implementing the relative positioning of the image U C to the reference image and thereby deriving the linear velocity and / or angular velocity of the end effector 40, the neural network library 180b includes a set of N convolutional layers 184b, followed by a set of M fully connected layers 183b or a set of W recurrent layers 185b or a set of W long short-term memory layers 186b.

[0210] In a third embodiment for implementing the relative positioning of the image U C to the reference image and thereby deriving the linear velocity and / or angular velocity of the end effector 40, the neural network library 180b includes a set of N convolutional layers 184b, followed by a combination of a set of M fully connected layers 183b and a set of W recurrent layers 185b or a set of W long short-term memory layers 186b.

[0211] In practice, the fully connected layer 183b may include K neurons, where N, M, W, and K can be any positive integers, and the values can vary according to the embodiments. For example, N can be approximately 8, M can be approximately 2, W can be approximately 2, and K can be approximately 1000. Moreover, the convolutional layer 184b can perform a non-linear transformation, which can be a composite function of operations (such as batch normalization, rectified linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 184b can also include a non-linear function (such as including a rectified non-linear ReLU operation) configured to extract a modified feature map.

[0212] Additionally, in practice, one of the layers 183b or 184b serves as the input layer for the input image U C and one of the layers 183b, 185b, and 186b can serve as the output layer, which is used to output the linear and angular velocities of the end effector regressed according to the last fully connected layer (e.g., 6 units, 3 units for linear velocity, and 3 units for angular velocity) using a linear or non-linear activation function.

[0213] During training, the training weights of the image prediction model 80b are continuously updated by comparing the predicted linear and angular velocities of the end effector relative to a certain reference anatomical structure (given the ultrasound image 161c as the input) with the true linear and angular velocities of the end effector describing the movement to a certain reference anatomical structure from a batch of training data set D (which can be systematically selected or randomly selected from a data memory (not shown)). More specifically, the coefficients of the filter can be initialized with predefined values or arbitrary values. The coefficients of the filter are applied to this batch of training data set D via forward propagation and are adjusted via backpropagation to minimize any output error.

[0214] Reference Figure 13A and Figure 13B , the continuous positioning controller 50f uses the inverse prediction model 70a, the image prediction model 80a, the subtractor 53, and the control law 54a to perform the closed-loop continuous position control method of the present disclosure as represented by the flowchart 190a.

[0215] In one TEE probe embodiment, stage S192a of process 190a includes inserting the TEE probe handle 132 (FIG. 3A) into a robot controller 100 (FIG. 3B) known in the art of the present disclosure to control the toggling of the handle 132 and the rotation of the TEE probe 130. The TEE transducer 140 (FIG. 3A) is inserted into the body through the esophagus and positioned near the anatomical structure of interest (e.g., the heart) (e.g., positioned as Figure 13C(at the mid-esophageal position shown). The ultrasound image parameters are defined at this target pose of the TEE transducer 140.

[0216] Stage S194a of process 190a includes the image prediction model 90 processing the current ultrasound image 81a (which we will call the previous ultrasound image U f ) to predict the relative position of the image plane with respect to the reference anatomical structure The sonographer uses the ultrasound image to observe the anatomical structure and determine the movement T that the transducer needs to make from its current position d . Alternatively, the movement of the required transducer can be provided from: an external tracking device, a user interface, other imaging modalities registered to the ultrasound image (e.g., an X-ray image registered to a 3D TEE image using EchoNavigator (Philips)).

[0217] Based on T d , the inverse prediction model 70a predicts the joint variables required to move the robot to the desired position 72a. The robot controller 100 receives the joint variables 72a and moves the TEE probe 130 accordingly.

[0218] Stage S196a of process 190a includes the ultrasound transducer reaching another position when acquiring the ultrasound image U c . The image prediction model 90g uses the processing of the current ultrasound image U c to predict the relative positions of the two current image planes with respect to the reference anatomical structure As a result, the movement between the previous position and the current position in the anatomical structure (e.g., the heart) can be calculated as follows:

[0219]

[0220] In the second TEE probe embodiment, as Figure 13D shown, path selection on the image 201 generated by an external imaging modality registered to the ultrasound image can be utilized to determine the desired position T d (e.g., by using methods known in the art (Philips EchoNavigator), an X-ray image and a cone beam CT image can be registered to the ultrasound image).

[0221] Reference Figure 14A and Figure 14B , the continuous positioning controller 50g uses the inverse prediction model 70b, the image prediction model 80b, the subtractor 53 and the control law 54b to perform the closed-loop continuous position control method of the present disclosure as represented by the flowchart 190b.

[0222] In one embodiment of a TEE probe, stage S192b of process 190b includes inserting the TEE probe handle 142 (FIG. 3A) into a robotic controller 100 (FIG. 3B) known in the art of the present disclosure to control the toggling of the handle 142 and the rotation of the TEE probe 140. The TEE transducer 140 (FIG. 3A) is inserted into the body through the esophagus and positioned near the anatomical structure of interest (e.g., the heart) (e.g., positioned at the mid-esophageal location as shown in Figure 14C ). The ultrasound image parameters are defined at this target pose of the TEE transducer 140.

[0223] Stage S194b of process 190b includes the user desiring to move the transducer in the image space, for example, by selecting a path from point A to point B in the ultrasound image 203 or a transformation between image planes A and B. In this embodiment, the 末端执行器 J 图像 Jacobian 204 transforms the first linear velocity and the first angular velocity 202 defined by the path on the image to the end effector coordinate system. 末端执行器 J 图像 The Jacobian 204 is calculated by using methods known in the art of the present disclosure to know the spatial relationship between the end effector and the image coordinate system.

[0224] Based on V d , the inverse prediction model 70b ( Figure 7B ) predicts the joint velocities 72b required to move the robot to the desired position. The robotic controller 100 receives the joint velocities 72b and moves the TEE probe 130 accordingly.

[0225] Stage S196b of process 190a includes another position when the ultrasound transducer reaches the U c for acquiring the ultrasound image. The image prediction model 80b processes the current ultrasound image U c 81a to predict the velocity vector 83a of the end effector between points A and B. The image prediction model 80b estimates the function between the Cartesian velocity of the end effector and the velocity in the joint space, i.e., the neural network models the manipulator Jacobian given a 6-vector consisting of the linear velocity and the angular velocity of the end effector 71c to predict the n-vector 72c of the joint velocities.

[0226] As understood by those skilled in the art of the present disclosure, a neural network that models the spatial relationship between images of an anatomical structure (e.g., the heart) requires a large training data set specific to a given organ.

[0227] In an alternative embodiment, features are directly extracted from the image to verify the position of the transducer. In such asFigure 14C In the illustrated embodiment, the user selects an object on the image 205, e.g., the apical wall, and the system extracts certain salient features around the object via feature extraction 206. These features (which may include the edges, shape, and size of the object) are first detected using methods known in the art of the present disclosure (e.g., Canny edge detector, morphological operations, etc.), and second, the scale-invariant feature transform (SIFT) known in the art of the present disclosure is used to track these features. Finally, the system defines a path between the selected object and the center of the field of view, which path indicates the desired movement on the image. By using SIFT to track the salient features, the continuous position controller 50g( Figure 14A ) is able to correct predictions from the network within a closed-loop control.

[0228] More particularly, a velocity-based control system for a continuum robot. Once the user selects an object on the image (e.g., the apical wall (see the red dot on the ultrasound image)), the desired movement on the image is identified. The movement is defined by the path between the center of the field of view and the selected object, and the Jacobian 204 can be used to transform this movement into linear and angular velocities in the end-effector space. This Cartesian velocity is then sent to a neural network, which infers the joint velocities. The achieved position is iteratively verified against the path defined by the continuously tracked object and the center of the field of view.

[0229] In practice, the closed control loop of FIGS. 13 and 14 can be closed using other modalities, e.g., optical shape sensing (OSS), electromagnetic tracking, or an X-ray image registered to the ultrasound image using, e.g., a Philips EchoNavigator.

[0230] Moreover, in practice, the prediction accuracy of the neural network is affected by the configuration of the flexible endoscope. Therefore, first, the position of the transducer relative to the heart is defined using, e.g., neural network g or a Philips HeartModel (which will implicitly define one of the possible configurations). Second, a specific set of network weights is loaded into the model according to the detected configuration, thereby improving the prediction accuracy.

[0231] A similar method can be used to guide the user to a position that can provide optimal imaging and guidance.

[0232] Moreover, one of the most difficult problems in machine / deep learning is being able to access big data in the right format to train a prediction model. More specifically, collecting and constructing training sets and validation sets is very time-consuming and expensive because it requires domain-specific knowledge. For example, to train a prediction model to accurately distinguish between benign and malignant breast tumors, such training requires thousands of ultrasound images annotated by radiology experts and transforming them into a numerical representation that the training algorithm can understand. Additionally, the image dataset may be inaccurate, corrupted, or have noisy labels, all of which factors can lead to inaccurate detection, and collecting large medical datasets may raise ethical and privacy issues as well as many other problems.

[0233] Reference Figure 15 , the training data collection system of the present disclosure uses a shape sensing controller 103 that provides a 3D position vector 233 for each point on an optical shape sensing fiber 332 that is mounted to an endoscope 131 of a robot and a tether-driven manipulator 231. The distal portion of the optical fiber 332 is embedded in a plastic housing that rigidly couples the optical fiber to the end effector and introduces a certain curvature within the shape.

[0234] For example, Figure 16A and Figure 16B shows the distal end 332d of the optical shape sensing fiber 332 embedded in a plastic housing 350 that is rigidly attached to an ultrasonic transducer 352 of a manipulator 231 as shown in Figure 16C and Figure 16D . The plastic housing 350 imposes a certain curvature on the shape, thereby enabling the use of template matching algorithms known in the art of the present disclosure to estimate the end effector pose (e.g., a data acquisition sequence 370 as shown in Figure 17 , where α and β on each axis correspond to Figure 15 the knob positions on a TEE handle 132 as shown in

[0235] Returning to FIG. 16, generally, by detecting such a pattern, it is possible to estimate the end effector pose T ∈ SE(3) using methods known in the art. The robot controller 100 sends motion commands to the robot, and the robot control is responsible for actuating the knobs that tighten or slacken the tethers. By changing the state of the tethers, the position of the end effector changes, as shown in FIGS. 3D and 3E. The data storage controller 190 receives the shape h ∈ (p1…p n ) of the optical fiber, the end effector pose T, and the motion commands (i.e., joint positions) q t from the shape sensing controller and the robot controller, respectively. The data is stored as a triple on a storage device and later used to train the deep convolutional neural network of the present disclosure.

[0236] More particularly, the shape sensing guide wire 332 is embedded or attached to the continuum robot, and the shape sensing guide wire 332 uses optical shape sensing (OSS) technology known in the art. OSS uses light along a multi-core optical fiber for device positioning and navigation during surgical intervention. The principle involved utilizes distributed strain measurement in the optical fiber using characteristic Rayleigh backscattering or a controlled grating pattern.

[0237] The shape sensing controller 103 is configured to acquire the shape of the shape sensing guide wire 322 and estimate the pose T∈SE(3) of the end effector that is rigidly attached to the plastic housing 350. The plastic housing 350 applies a certain curvature to the guide wire, as described above. The method for estimating the pose T is based on a well-defined curvature and a template matching algorithm known in the field of the present disclosure.

[0238] The data acquisition controller 191 is configured to generate a sequence of motor commands according to a predefined acquisition pattern (e.g., spiral, radial, or square motion, etc.) and send movement commands to the robot controller 100.

[0239] The robot controller 100 is configured to receive the robot position and send a movement signal to the robot. The electric knob robot will tighten / relax the tether, which will cause the movement of the end effector. The robot controller is also configured to receive and interpret information from the data acquisition controller 191 and change the robot position based on the information from the data acquisition controller 191.

[0240] The data storage controller 190 is configured to receive and interpret information from the robot controller 100 and the shape sensing controller 103 and store the data in a data storage medium (not shown) in a format defined by the following specifications:

[0241] The first specification is to acquire a training data set D for all configurations predefined by the data acquisition controller 191. The data set D consists of a set of n sequences W: D = {W1, W2,..., W n}, where each sequence W n is composed of i data points d i : W n = {d1, d2,..., d i}; and each data point d i from the sequence is defined by a 3-tuple: d i = (T i , H i , Q i ).

[0242] This 3-tuple consists of the end effector pose T∈SE(3), a sequence of k successive shapes (e.g., H∈(ht , h t+ 1...h t+k )), where h ∈ (p1...p m ) is a set of m vectors p m , which describe the position of the shape-sensing guidewire in 3D Euclidean space, auxiliary shape parameters (e.g., strain, curvature, and twist), and a sequence of j successive joint variables (Q ∈ (q t , q t+1 ...q t+j )) collected at sequential time points from t to t + j. For example, the entry q t can be the angles α, β on the control knob collected at time point t.

[0243] Reference Figure 19 , the training data collection method 360 of the present disclosure is performed by the Figure 15 training data collection system.

[0244] Reference Figure 15 and Figure 19 , stage S362 of method 360 includes moving the robot to the origin position by the robot controller 100 using, for example, limit switches or proximity sensors. The distal portion of the shape-sensing guidewire 332 is inserted into the notch 353 provided in the plastic housing 350. The notch 353 will impose a certain curvature on the shape.

[0245] The housing 350 is rigidly attached to the end effector of the continuum robot.

[0246] By using a template matching algorithm known in the art during stage S364 of method 360, the shape-sensing controller 103 can now estimate the pose T ∈ SE(3) of the end effector. Preferably, the coordinate system of the end effector is defined by the template. However, an additional calibration matrix can also be used. When the robot system is still in the origin position, the pose of the end effector in the OSS coordinate system is acquired. Each subsequent pose acquired during the experiment is estimated relative to this initial position.

[0247] The data acquisition controller 191 generates a motion sequence (i.e., a set of joint variables) according to a predefined acquisition pattern (e.g., Figure 18 pattern 370). The data acquisition controller 191 iteratively sends the motion sequence to the robot controller 100, and the robot controller 100 moves the robot according to the generated joint variables.

[0248] Stage S366 of method 300 includes the data storage controller 190 acquiring and storing the data tuple d i = (Ti , H i , Q i ). Importantly, since H i and Q i are sequential, all previous time points are stored in the memory by the data storage controller 190.

[0249] To facilitate further understanding of the various inventions of the present disclosure, the following description of Figure 19 teaches an exemplary embodiment of the continuous positioning controller of the present disclosure. From this description, one of ordinary skill in the art will understand how to apply various aspects of the present disclosure to make and use additional embodiments of the continuous positioning controller of the present disclosure.

[0250] Refer to Figure 19 , the continuous positioning controller 400 includes one or more processors 401, a memory 402, a user interface 403, a network interface 404, and a storage device 405 interconnected via one or more system buses 406.

[0251] Each processor 401 can be any hardware device capable of executing instructions stored in the memory 402 or the storage device or otherwise processing data, as known in the art of the present disclosure or contemplated hereinafter. In a non-limiting example, the (one or more) processors 401 can include a microprocessor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other similar devices.

[0252] The memory 402 can include various memories, e.g., non-transitory memory and / or static memory as known in the art of the present disclosure or contemplated hereinafter, including but not limited to L1, L2, or L3 cache memories or system memory. In a non-limiting example, the memory 402 can include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read only memory (ROM), or other similar memory devices.

[0253] The user interface 403 can include one or more devices for enabling communication with a user such as an administrator, as known in the art of the present disclosure or contemplated hereinafter. In a non-limiting example, the user interface can include a command line interface or a graphical user interface that can be presented to a remote terminal via the network interface 404.

[0254] The network interface 404 may include one or more devices for implementing communication with other hardware devices as known in the art of the present disclosure or contemplated hereinafter. In a non-limiting example, the network interface 404 may include a network interface card (NIC) configured to communicate according to the Ethernet protocol. Additionally, the network interface 404 may implement a TCP / IP stack for communication according to the TCP / IP protocol. Various alternative or additional hardware or configurations for the network interface 404 will be apparent.

[0255] The storage device 405 may include one or more machine-readable storage media as known in the art of the present disclosure or contemplated hereinafter, including but not limited to read-only memory (ROM), random access memory (RAM), disk storage media, optical storage media, flash memory devices, or similar storage media. In various non-limiting embodiments, the storage device 405 may store instructions for execution by the processor(s) 401 or data on which the processor(s) 401 may operate. For example, the storage device 405 may store a basic operating system for controlling various basic operations of the hardware. The storage device 405 also stores application program modules in the form of executable software / firmware for implementing the various functions of the controller 400a as previously described in the present disclosure, including but not limited to the forward prediction model(s) 60, the inverse prediction model(s) 70, and the imaging prediction model(s) 80 as previously described in the present disclosure.

[0256] In practice, the controller 400 may be installed within an X-ray imaging system 500, an interventional system 501 (e.g., an interventional robotic system), or a stand-alone workstation 502 (e.g., a client workstation or a mobile device such as a tablet) that communicates with the X-ray imaging system 500 and / or the interventional system 501. Alternatively, the components of the controller 400 may be distributed among the X-ray imaging system 500, the interventional system 501, and / or the stand-alone workstation 502.

[0257] Moreover, in practice, additional controllers of the present disclosure, including a shape sensing controller, a data storage controller, and a data acquisition controller, may each also include one or more processors, a memory, a user interface, a network interface, and a storage device interconnected via one or more system buses as arranged in Figure 19 such that the storage device contains applicable application program modules of the controller as previously described herein. Alternatively, two or more controllers of the present disclosure may be integrated into a single controller, where the storage device contains applicable application program modules of the two or more controllers as previously described herein.

[0258] Referring to FIGS. 1-19, those of ordinary skill in the art of the present disclosure will recognize many benefits of the present disclosure.

[0259] In addition, as those of ordinary skill in the art will recognize in view of the teachings provided herein, the structures, elements, components, etc. described in the present disclosure / specification and / or depicted in the drawings can be implemented in various combinations of hardware and software and provide functions that can be combined in a single element or multiple elements. For example, the functions of the various structures, elements, components, etc. shown / illustrated / depicted in the drawings can be provided by using dedicated hardware and hardware that is capable of running software associated with appropriate software for additional functions. When provided by a processor, the functions can be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors, some of which can be shared and / or multiplexed. Further, the explicit use of the terms "processor" or "controller" should not be construed to refer exclusively to hardware capable of running software and can implicitly include, without limitation, digital signal processor ("DSP") hardware, memory (e.g., read only memory ("ROM") for storing software, random access memory ("RAM"), non-volatile storage, etc.), and any unit and / or machine that is substantially capable of (and / or configurable to) execute and / or control a process (including hardware, software, firmware, combinations thereof, etc.).

[0260] Moreover, all statements herein reciting principles, aspects, and embodiments of the present invention, as well as specific examples thereof, are intended to cover both structural and functional equivalents thereof. Additionally, such equivalents are intended to include both currently known equivalents and equivalents developed in the future (i.e., any elements developed that perform the same or substantially similar function regardless of structure). Thus, for example, in view of the teachings provided herein, those of ordinary skill in the art should recognize that any block diagrams presented herein can represent a conceptual view of illustrative system components and / or circuits implementing the principles of the present invention. Similarly, in view of the teachings provided herein, those of ordinary skill in the art should recognize that any flowcharts, flow diagrams, etc. can represent various processes that can be substantially represented in a computer-readable storage medium and run by a computer, processor, or other device having processing capabilities, whether or not such a computer or processor is explicitly shown.

[0261] Having described the preferred and exemplary embodiments of the various inventions of the present disclosure, which are intended to be illustrative and not limiting, it should be noted that, in view of the teachings provided herein (including the drawings), those skilled in the art can make modifications and variations. Accordingly, it should be understood that changes within the scope of the embodiments disclosed herein can be made to the preferred and exemplary embodiments of the present disclosure.

[0262] In addition, it should be anticipated that corresponding and / or relevant systems that include and / or implement the device / system according to the present disclosure, or corresponding and / or relevant systems that can be used / implemented in the device, for example, are also anticipated and considered to be within the scope of the present disclosure. Moreover, corresponding and / or relevant methods for manufacturing and / or using the device and / or system according to the present disclosure are also anticipated and considered to be within the scope of the present disclosure.

Claims

1. A positioning controller (50) for an interventional device (30), the interventional device including a device portion (40), the positioning controller (50) comprising: A memory including at least one of the following: A forward prediction model (60) configured to receive a commanded positioning movement of the interventional device (30) and output data related to a prediction of a navigation pose of the device portion (40) using the embedded kinematics of the interventional device, the positioning movement meaning the movement of the interventional device, and A control prediction model (70) configured to receive target pose data of the interventional device and output data related to a prediction of a positioning movement of the interventional device (30) using the kinematics of the interventional device, the positioning movement meaning the movement of the interventional device; and At least one processor in communication with the memory, wherein the at least one processor is configured to perform at least one of the following operations: (i) Apply the forward prediction model (60) to the commanded positioning movement of the interventional device (30) to present a predicted navigation pose of the device portion (40), and generate positioning data regarding information for positioning the device portion (40) to a target pose by the interventional device (30) based on the predicted navigation pose of the device portion (40); and (ii) Apply the control prediction model (70) to the target pose of the device portion (40) to present a predicted positioning movement of the interventional device (30), and generate a positioning command for controlling the interventional device (30) to position the device portion (40) to the target pose based on the predicted positioning movement of the interventional device (30).

2. The positioning controller according to claim 1, wherein The forward prediction model (60) is trained or has been trained on the forward kinematics of the interventional device (30), and / or the control prediction model (70) is an inverse prediction model that is trained or has been trained on the inverse kinematics of the interventional device (30).

3. The positioning controller according to claim 1, wherein, The device portion is an end effector of the interventional device (30).

4. The positioning controller according to claim 1, further configured to continuously generate the positioning data and / or continuously generate the positioning command, such that the positioning controller is considered a continuous positioning controller.

5. The positioning controller (50) according to claim 1 or 2, wherein, The forward prediction model (60) includes: A neural network library having an input layer and an output layer, the input layer being configured to input joint variables of the interventional device, the joint variables representing a commanded positioning movement of the interventional device (30), the output layer being configured to output at least one of translation, rotation, and pivoting of the device portion (40) derived by regression according to the joint variables of the interventional device, wherein the at least one of the translation, the rotation, and the pivoting of the device portion (40) infers the predicted navigation pose of the device portion (40).

6. The positioning controller (50) according to claim 1 or 2, wherein, The control prediction model (70) includes: A neural network library having an input layer and an output layer, wherein the input layer is configured to input at least one of translation, rotation, and pivoting of the device part (40), and the output layer is configured to output joint variables of the intervention device derived from regression based on at least one of the translation, the rotation, and the pivoting of the device part (40). Wherein the joint variables of the intervention device infer the predicted positioning movement of the intervention device (30).

7. The positioning controller (50) according to claim 1 or 2, wherein, The forward prediction model (60) includes: A neural network library having an input layer and an output layer, wherein the input layer is configured to input joint velocities of the intervention device (30), the joint velocities representing the commanded positioning movement of the intervention device (30), and the output layer is configured to output at least one of a linear velocity and an angular velocity of the device part (40) obtained by regression based on the joint velocities of the intervention device (30). Wherein at least one of the linear velocity and the angular velocity of the device part (40) infers the predicted navigation attitude of the intervention device (30).

8. The positioning controller (50) according to claim 1 or 2, wherein, The control prediction model (70) includes: A neural network library having an input layer and an output layer, wherein the input layer is configured to input at least one of a linear velocity and an angular velocity of the device part (40) to the target attitude, and the output layer is configured to output joint velocities of the intervention device (30) obtained by regression based on at least one of the linear velocity and the angular velocity of the device part (40) to the target attitude. Wherein the joint velocities of the intervention device (30) infer the predicted positioning movement of the intervention device (30).

9. The positioning controller (50) according to claim 1, wherein, The forward prediction model (60) includes: A neural network library having an input layer and an output layer, wherein the input layer is configured to input a previous shape sequence of the intervention device (30), the previous shape sequence representing the commanded positioning movement of the intervention device (30), and the output layer is configured to output a subsequent shape sequence of the intervention device (30) derived from a time series prediction result based on the previous shape sequence of the intervention device (30). Wherein the subsequent shape sequence of the intervention device (30) infers the predicted navigation attitude of the device part (40).

10. The positioning controller (50) according to claim 1 or 2, wherein, The forward prediction model (60) includes: A neural network library having an input layer and an output layer, wherein the input layer is configured to input a previous shape sequence of the intervention device (30), the previous shape sequence representing the commanded positioning movement of the intervention device (30), and the output layer is configured to output a subsequent shape sequence of the intervention device (30) derived from a time series prediction result based on the previous shape sequence of the intervention device (30). Wherein the subsequent shape of the intervention device (30) infers the predicted navigation attitude of the device part (40).

11. The positioning controller (50) according to claim 2 Among them, Performs at least one of the following: The forward prediction model (60) is also trained on at least one navigation data of the intervention device (30), and the at least one navigation data assists in predicting the attitude of the device part (40) for the forward kinematics of the intervention device (30), and The control prediction model (70) is also trained on the at least one navigation data of the intervention device (30), and the at least one navigation data assists in predicting the positioning movement of the intervention device (30) for the inverse kinematics of the intervention device (30), wherein the at least one navigation data that assists in the forward kinematics or the inverse kinematics of the intervention device is in the following form: at least one image of the intervention device and / or the end effector; operating characteristics of the intervention device; and / or operating characteristics of the end effector; and wherein the at least one processor is configured to perform at least one of the following operations: (i') Apply the forward prediction model (60) to both the commanded positioning movement of the intervention device (30) and the at least one navigation data that assists in the forward kinematics of the intervention device (30) to present the predicted navigation attitude of the device part (40); and (ii') Apply the control prediction model (70) to both the target attitude of the device part (40) and the at least one navigation data that assists in the inverse kinematics of the intervention device (30) to present the predicted positioning movement of the intervention device (30).

12. The positioning controller (50) according to claim 1, Among them, performing at least one of the following: The forward prediction model (60) is configured to: further receive at least one auxiliary navigation data that assists in the navigation data of the intervention device (30), and further process the at least one auxiliary navigation data to output a prediction result of the navigation attitude of the device part (40), and the navigation data is in the following form: at least one navigation command (31); at least one actuation signal (32); and / or at least one navigation force (33) transmitted to / imposed on the intervention device, and The control prediction model (70) is configured to: further receive at least one auxiliary navigation data that assists in the navigation data of the intervention device (30), and further process the at least one auxiliary navigation data to output a prediction result of the positioning movement of the intervention device (30), and the navigation data is in the following form: at least one navigation command (31); at least one actuation signal (32); and / or at least one navigation force (33) transmitted to / imposed on the intervention device, wherein the at least one auxiliary navigation data is in the following form: at least one image of the intervention device and / or the end effector; operating characteristics of the intervention device; and / or operating characteristics of the end effector; and wherein the at least one processor is configured to perform at least one of the following operations: (i') Apply the forward prediction model (60) to both the commanded positioning motion of the interventional device (30) and the at least one auxiliary navigation data to present the predicted navigation pose of the device portion (40); and (ii') Apply the control prediction model (70) to the target pose of the device portion (40) and the at least one auxiliary navigation data to present the predicted positioning motion of the interventional device (30).

13. A machine-readable storage medium encoded with instructions that, when executed by at least one processor, cause an interventional device including a device portion to be instructed with the instructions, the machine-readable storage medium storing: At least one of the following: A forward prediction model (60) configured, using the kinematics of the interventional device, to receive the commanded positioning motion of the interventional device (30) and output data related to a prediction of the navigation pose of the device portion (40), the positioning motion meaning the movement of the interventional device, and A control prediction model (70) configured, using the kinematics of the interventional device, to receive the target pose data of the interventional device and output data related to a prediction of the positioning motion of the interventional device (30), the positioning motion meaning the movement of the interventional device; and Instructions for at least one of the following: (i) Apply the forward prediction model (60) to the commanded positioning motion of the interventional device (30) to present the predicted navigation pose of the device portion (40), and generate positioning data regarding information on positioning the device portion (40) to a target pose by the interventional device (30) based on the predicted navigation pose of the device portion (40); and (ii) Apply the control prediction model (70) to the target pose of the device portion (40) to present the predicted positioning motion of the interventional device (30), and generate a positioning command for controlling the interventional device (30) to position the device portion (40) to the target pose based on the predicted positioning motion of the interventional device (30).

14. A positioning method executable by a positioning controller (50) for an interventional device (30), the interventional device including a device portion (40), The positioning controller (50) stores at least one of the following: A forward prediction model (60) configured, using the embedded kinematics of the interventional device, to receive the commanded positioning motion of the interventional device (30) and output data related to a prediction of the navigation pose of the device portion (40), the positioning motion meaning the movement of the interventional device, and A control prediction model (70) configured, using the kinematics of the interventional device, to receive the target pose data of the interventional device and output data related to a prediction of the positioning motion of the interventional device (30), the positioning motion meaning the movement of the interventional device; Among them, The positioning method includes the positioning controller (50) performing at least one of the following operations: (i) Apply the forward prediction model (60) to the commanded positioning movement of the interventional device (30) to present the predicted navigation pose of the device part (40), and generate positioning data regarding information on positioning the device part (40) to the target pose by the interventional device (30) based on the predicted navigation pose of the device part (40); and (ii) Apply the control prediction model (70) to the target pose of the device part (40) to present the predicted positioning movement of the interventional device (30), and generate a positioning command for controlling the interventional device (30) to position the device part (40) to the target pose based on the predicted positioning movement of the interventional device (30).

Images