Feedforward continuous positioning control of end effectors

Abstract

A positioning controller 50 including a forward prediction model 60 and / or an inverse control prediction model 70 for positioning control of an interventional device 30 including an interventional device portion 40. In operation, the controller 50 applies the forward prediction model 60 to instructed positioning motions of the interventional device 30 to render a predicted navigated pose of the end effector 40, and generates positioning data informing the placement by the interventional device 30 of the interventional device portion 40 to a target pose based on the predicted navigated pose of the portion 40. Alternatively, prior to or subsequent to the application of the control prediction model 70 to a target pose of the interventional device portion 40 to render a predicted positioning motion of the interventional device 30, and generates positioning instructions for controlling the placement by the interventional device 30 of the device portion 40 to the target pose based on the predicted positioning motion of the interventional device 30.

Description

[Technical Field]

[0001] This disclosure generally relates to the placement control of parts of intervention devices (e.g., end effectors of intervention devices) used in interventional procedures (e.g., minimally invasive surgery, video-assisted thoracic surgery, vascular procedures, intracavitary procedures, orthopedic procedures). In particular, this disclosure relates to the incorporation of predictive models in the placement control of such parts of intervention devices used in interventional procedures. [Background technology]

[0002] Continuous (or discontinuous) control—the placement of device components (e.g., end effectors) within a specific workspace—is one of the most commonly attempted forms of control in conventional rigid link robots. By utilizing the robot's separate rigid link structures, precise placement of intervention device components (e.g., end effectors) can be achieved, as desired in systematic applications such as manufacturing. However, due to the delicate nature of human organs, which are made of deformable soft tissue, and patient safety concerns, the use of rigid link robots is not very desirable in clinical settings.

[0003] Furthermore, biologically inspired robots, in particular, can exhibit movements similar to those of snakes, elephants, and octopuses, which can be highly effective in manipulating soft, anatomical objects. Nevertheless, effective control of robotic structures in clinical settings, and especially effective continuous control, has proven to be extremely difficult to achieve in terms of the complexity of continuum (or quasi-continuum) structures, given the difficulty in mathematically modeling or providing sufficient actuator inputs to enable stable control for desired degrees of freedom.

[0004] For example, one approach to the continuous placement control of end effectors supported by a continuum robot involves modeling and controlling the configuration of the continuum robot, in which case a static model formulated as a set of nonlinear differential equations attempts to account for lance deformation of the continuum robot due to bending, twisting, and stretching. However, the accuracy of the mathematical model is susceptible to changes in the robot's environmental conditions (e.g., temperature and humidity) that affect the mechanical properties of the robot components, and is also susceptible to arbitrary manufacturing inaccuracies or the presence of various workloads.

[0005] Further examples suggest that another approach to controlling the position of robot-supported end effectors involves manipulating the robot by projecting the set of permissible motions and permissible forces into the joint space corresponding to the manipulator control. For example, after inserting the robot into the nasal cavity, the control device adjusts the position of each segment of the robot to increase the difference between the measured generalized force and the assumed generalized force at each end disk. However, increasing the robot's degrees of freedom to make it easier to manipulate has the detrimental effect of complicating the robot's kinematics. This is particularly problematic in the case of continuous control of a continuum robot.

[0006] However, even when effective positional control of robot-supported end effectors is achieved, improper calibration or use of the robot system, or general wear and tear of mechanical components, can negatively impact the predictive accuracy of the kinematics of the robotic structure. Similarly, this is particularly problematic in the case of continuous control of continuous robots (or continuous robotic structures). [Overview of the project] [Problems that the invention aims to solve]

[0007] Known techniques devised for positional control of parts of intervention devices (e.g., end effectors) have offered limited advantages. Therefore, there remains a need for improved techniques to provide effective positional control of these parts of intervention devices. To achieve this objective, this disclosure teaches feedforward positional control, feedback positional control, and data acquisition. This control is preferably performed sequentially. [Means for solving the problem]

[0008] For this purpose, the present invention, as a first embodiment, proposes a placement control device for an intervention device including an imaging device, as described in any one of claims 1 to 12.

[0009] As second and third embodiments, the present invention proposes a non-temporary (or non-temporary) machine-readable storage medium encoded with instructions, as described in claim 13, and a method of placement that can be performed by a placement control device for an intervention device, as described in claim 14.

[0010] Feedforward (preferably continuous) placement control This disclosure further teaches a predictive model approach for feedforward (preferably continuous) positional control of manually or automatically navigated positioning of a device portion (e.g., an end effector) supported by an intervention device, based on a predictive model configured using the kinematics of the intervention device, which is optionally trained on these kinematics.

[0011] Another embodiment of the present disclosure for feedforward (preferably continuous) placement control of a device portion (e.g., an end effector) is a (continuous) placement control device comprising a forward predictive model configured using (optionally forward) kinematics of the intervention device to predict the navigated pose of the end effector (optionally trained on these forward kinematics of the intervention device), and / or, optionally, a control (optionally inverse) predictive model configured using the kinematics of the intervention device to predict the placement motion of the intervention device, trained on the inverse kinematics of the intervention device.

[0012] For the purposes of this description and the disclosure, the term “navigated pose” broadly encompasses the pose of a portion of the intervention device (e.g., the end effector of the intervention device) as it is navigated to a spatial position via the intervention device during the intervention procedure, and the term “positioning movement” broadly encompasses any movement of the intervention device that navigates this portion of the device to a spatial position during the intervention procedure.

[0013] During operation, the continuous placement control device applies a forward prediction model to the instructed placement motion of the intervention device to render the predicted navigated pose of the end effector, and generates continuous placement data that provides information about the intervention device's placement of the device portion to the target pose based on the predicted navigated pose of the device portion.

[0014] Alternatively, preemptively, or simultaneously, a (preferably continuous) placement control device applies an inverse prediction model to a target pose of a portion of the intervention device (e.g., an end effector) to render the predicted placement motion of the intervention device, and generates (continuous) placement instructions that control the placement of the device portion by the intervention device to the target pose based on the predicted placement motion of the intervention device.

[0015] Feedback (preferably continuous) placement control This disclosure further teaches a predictive model approach for feedback (preferably continuous) positioning control of a manual or (semi)automatic positioning of an imaging device, related to a part of an intervention device (e.g., the end effector of an intervention device), or a mounted imaging device, based on an imaging predictive model configured using the kinematics of the intervention device (optionally correlated with image data) for receiving imaging data from the imaging device as feedback to the manual or automatic positioning of the imaging device (or a part of an intervention device linked to the imaging device—e.g., the end effector) to a target pose. The predictive model is optionally trained or has been trained based on images generated by the device part (e.g., the end effector).

[0016] Embodiments of the present disclosure for feedback (preferably continuous) control placement of an imaging device (or a portion of an intervention device linked to an imaging device—e.g., an end effector) include a continuous placement control device comprising an imaging predictive model trained on correlation between relative imaging by the end effector and the forward kinematics of the intervention device predicting the navigated pose of the end effector. The (continuous) placement control device further comprises a control predictive model configured using the kinematics of the intervention device to predict the correct placement motion of the intervention device. Optionally, this control predictive model is trained or will be trained on the inverse kinematics of the intervention device to output what predicts the correct placement motion.

[0017] For the purposes of this description and disclosure, the term “relative imaging” broadly encompasses the generation of an image of the intervention by the end effector in a given pose relative to a reference image of the intervention.

[0018] During operation, after the navigation of a device portion (e.g., end effector) to a target pose, the continuous placement control device (preferably) applies an imaging prediction model to imaging data generated by the end effector to render a predicted navigated pose of the device portion (e.g., end effector), applies a control (or particularly inverse) prediction model to error placement data derived in terms of the difference between the end effector's target pose and the end effector's predicted navigated pose to render a predicted correct placement motion of the intervention device, and generates continuous placement instructions to control the correct placement of the imaging device (40) to the target pose by the intervention device or a portion of the intervention device (e.g., end effector) associated with the imaging device.

[0019] Training data collection Furthermore, in order to facilitate (optionally sequential) placement control of manually or automatically navigating the placement of parts of an intervention device (e.g., end effectors) via a predictive model invariant to environmental differences (e.g., anatomical differences between patients, e.g., patient size, heart position, etc.), the disclosure further teaches (optionally trained) data acquisition techniques that presuppose the navigated placement of parts of the intervention device via a predetermined data point pattern, and the spatial placement of the intervention device, and the recording of the pose of the intervention device part at each data acquisition point, for which (training) data for a predictive model to infer the forward or inverse kinematics of the intervention device are consequently collected.

[0020] Embodiments of the present disclosure for collecting (optionally training) data for predictive models include a portion of the intervention device (e.g., an end effector) and a (optionally training) data collection system for an intervention device, comprising sensors adapted to provide positional and / or orientational and / or shape information, wherein at least a portion of the sensors (332) is attached (optionally by fixed shape) to the portion of the intervention device. Such sensors include markings visible from imaging systems (e.g., X-ray, MRI systems), electromagnetic tracking sensors, transducer sensors, and / or optical shape detection provided by / in optical fibers.

[0021] One particular embodiment of this embodiment is a data acquisition system (optionally for training) for the intervention device, which includes a part of the intervention device (e.g., an end effector) and an optical shape sensor, wherein a segment of the optical shape sensor is attached to the end effector (optionally by fixed shape).

[0022] The (training) data acquisition system uses a robot control device, a data acquisition control device, a placement identification module (or a shape detection control device in the specific embodiment described above), and a data storage control device.

[0023] During operation, the data acquisition control device instructs the robot control device to control the motion variables of the intervention device according to a predetermined data point pattern, and the placement identification module (or shape detection control device) is configured to identify positional information based on position and / or orientation and / or shape information received from the sensor in order to output the estimated pose of a portion of the intervention device and / or the estimated placement motion of the intervention device at each data point of the predetermined data point pattern. Thus, the identified positional information is acquired, derived, extracted, or received for the purpose of the estimation results, optionally based on the kinematics or the behavior of the intervention device constituting the placement identification module. In certain embodiments, the placement identification module derives or receives derived shape data from the position and / or orientation and / or shape information in order to identify the estimation results. If this placement identification module is a shape detection control device (as in the particular embodiments described above), the placement identification module controls the shape detection of the optical shape sensor, including the estimated pose of the end effector and the estimated placement motion of the intervention device at each data point of the predetermined data point pattern.

[0024] The "derivation of shape data from position and / or orientation and / or shape information" is carried out according to known techniques for deriving shape from data provided by "sensors." For example, the positions tracked by sensors provide a good indication of the overall shape of a segment of the intervention device equipped with these sensors, and algorithms (developed according to the distance between sensors and the possible shapes of the intervention device along this segment) are developed to derive or reconstruct this shape. Dynamic tracking of this arrangement further provides indication of the orientation of deformation. Sensors may further provide strain information that can indicate local placement and orientation of the intervention device (e.g., Rayleigh or Bragg diffraction grating sensors), from which shape can be derived and reconstructed (another known technique).

[0025] The estimated pose of the intervention device portion (e.g., end effector) is derived from at least some of the sensors attached to the end effector (optionally by fixed shape).

[0026] While the robot control unit controls the motion variables of the intervention device according to a predetermined data point pattern, the data storage control unit receives communications from the shape detection control unit for the estimated pose of the end effector for each data point, communications from the placement identification module (or shape detection control unit) for the estimated placement motion of the intervention device for each data point, and communications from the robot control unit for at least one motion variable of the intervention device for each data point.

[0027] In response to the communication, the data storage control device stores a temporal data sequence for the intervention device derived from the estimated pose of the end effector at each data point, the estimated spatial placement of the intervention device at each data point, and the motion variables of the intervention device at each data point. The temporal data sequence serves as training data for any type of machine learning model, in particular for the predictive model of this disclosure.

[0028] Furthermore, the data acquisition control device further instructs the robot control device to control the motion variables of the intervention device according to further predetermined data point patterns, in which case the data storage control device generates further temporal data sequences for any type of machine learning model, in particular for the predictive model of this disclosure.

[0029] Furthermore, in the description of this disclosure and the purpose of the claims, the following applies:

[0030] (1) Terms in the Art, including but not limited to “end effector,” “kinematics,” “position,” “arrangement,” “pose,” “posing,” “movement,” and “navigation,” are to be interpreted as known in the Art of the Disclosure and as illustrated herein.

[0031] (2) Examples of end effectors include, but are not limited to, intraoperative imaging devices, interventional tools / surgical instruments, and surgical sutures that are known in the art of the present disclosure and are discussed below.

[0032] (3) The term “intraoperative imaging device” broadly encompasses all imaging devices known in the art of this disclosure and discussed below for indicating anatomical objects / regions. Examples of intraoperative imaging devices include, but are not limited to, transesophageal echocardiography transducers (e.g., X7-2t transducer, Philips), laparoscopic ultrasound transducers (e.g., L10-4lap transducer, Philips), optical cameras, and detection devices (e.g., tissue spectrum detection sensors, ECG electrodes, probes for electrophysiological mapping).

[0033] (4) Examples of intervention tools / surgical instruments include, but are not limited to, surgical scalpels, cauterizers, ablation devices, needles, forceps, k-wires and associated drivers, endoscopes, awls, screwdrivers, bone knives, chisels, hammers, curettes, clamps, forceps, periodonteomes, and j-needles, which are known in the art of the present disclosure and are discussed below.

[0034] (5) The term “intervention device” broadly encompasses all devices known in the art of this disclosure and those discussed below for supporting the placement of end effectors during application. Examples of intervention devices include, but are not limited to, continuum flexible robots, flexible intervention scopes and guidewires.

[0035] (6) Examples of flexible robots include, but are not limited to, highly redundant robots (e.g., multiple independent links, meandering links, or concentric tubes), continuous backbone section robots (e.g., cable-actuated), tendon-actuated robots, gel-like flexible robots, and fluid-filled tube-actuated robots, which are known in the art of the present disclosure and are discussed below.

[0036] (7) Examples of flexible intervention scopes include, but are not limited to, transesophageal echocardiography (TEE) probe endoscopes, intracardiac echocardiography (ICE) probe endoscopes, laparoscopes, and bronchoscopes, which are known in the art of the present disclosure and are discussed below.

[0037] (8) The term “predictive model” broadly encompasses all types of models known in the art of this disclosure and those discussed below that predict navigation variables related to the sequential placement control of a part of the intervention device (e.g., an end effector) as described exemplarily in this disclosure. Optionally, these predictive models are constructed using kinematics to output this prediction based on a set of placement data. Optionally, predictive models are trainable or trainable based on the kinematics of the intervention device. Examples of predictive models include, but are not limited to, artificial neural networks (e.g., feedforward convolutional neural networks, recurrent neural networks, long-term short-term memory networks, autoencoder networks, generative adversarial networks, and many other deep learning neural networks).

[0038] (9) The term “control device” broadly encompasses all structural configurations of a main circuit board or integrated circuit for controlling the application of the principles of various inventions of the Disclosure as illustrated herein, as understood in the art of the Disclosure and illustrated herein. The structural configuration of a control device may include, but is not limited to, a processor, computer-available / computer-readable storage media, an operating system, application modules, peripheral device controllers, slots and ports, and instructions for control. The control device is housed in a workstation or is communicably linked to a workstation.

[0039] (10) The term “application module” broadly encompasses applications that are incorporated into or accessible by a control device, comprising electronic circuits (e.g., electronic components and / or hardware) and / or executable programs (e.g., executable software stored on a non-temporary (or non-temporary) computer-readable medium and / or firmware) for the execution of a particular application.

[0040] (11) The terms “signal transmission,” “data,” and “instruction” broadly encompass all forms of detectable physical quantities or impulses (e.g., voltage, current, or magnetic field strength) understood in the art of the Disclosure and illustrated in the Disclosure, for transmitting information and / or instructions that help apply the principles of the various inventions of the Disclosure, which are described later in the Disclosure. The signal transmission / data / command communications of the various components of the Disclosure involve any communication methods known in the art of the Disclosure, including but not limited to the transmission / reception of signal transmission / data / commands over any type of wired or wireless data link, and readings of signal transmission / data / instruction uploaded to a computer-available / computer-readable storage medium.

[0041] The aforementioned embodiments and other embodiments of this disclosure, as well as various structures and advantages of this disclosure, will become even clearer from the following detailed description of various embodiments of this disclosure, which should be read together with the accompanying drawings. The detailed description and drawings are illustrative and not limiting of this disclosure, and the scope of this disclosure is defined by the accompanying claims and their equivalents. [Brief explanation of the drawing]

[0042] [Figure 1] This figure shows an exemplary embodiment of an intervention device, including an end effector, known in the art of this disclosure. [Figure 2A] This figure shows exemplary embodiments of a continuous position control device according to various aspects of the present disclosure. [Figure 2B] This figure shows exemplary embodiments of a sequentially arranged state machine according to various aspects of the present disclosure. [Figure 3A] This figure shows an exemplary embodiment of a transesophageal echocardiography (TEE) probe known in the art of this disclosure. [Figure 3B] This figure shows an exemplary embodiment of a handle for a transesophageal echocardiography (TEE) probe known in the art of this disclosure. [Figure 3C-3F] This figure shows an exemplary motion of the TEE probe shown in Figure 2A, which is known in the art of this disclosure. [Figure 3G] This figure shows an exemplary embodiment of a robot-operated transthoracic echocardiography (TTE) probe known in the art of the present disclosure. [Figure 3H] This figure shows an exemplary embodiment of an optically shape-detected continuum robot known in the art of the present disclosure. [Figure 4A-4E] This figure shows a first exemplary embodiment of the forward prediction model and forward prediction method of the present disclosure. [Figures 5A-5E] This figure shows a first exemplary embodiment of the inverse prediction model and inverse prediction method of the present disclosure. [Figures 6A-6E] This figure shows a second exemplary embodiment of the forward prediction model and forward prediction method of the present disclosure. [Figures 7A-7E] This figure shows a second exemplary embodiment of the inverse prediction model and inverse prediction method of the present disclosure. [Figures 8A-8E] This figure shows a third exemplary embodiment of the forward prediction model and forward prediction method of the present disclosure. [Figures 9A-9F] This figure shows a fourth exemplary embodiment of the forward prediction model and forward prediction method of the present disclosure. [Figure 11A-11E] This figure shows a first exemplary embodiment of the image prediction model and image prediction method of the present disclosure. [Figures 12A-12E] This figure shows a second exemplary embodiment of the forward prediction model and forward prediction method of the present disclosure. [Figures 13A-13D] This figure shows the closed-loop pause control of the present disclosure. [Figure 14A-14D] This figure shows closed-loop vector velocity control according to the present disclosure. [Figure 15-18] This figure shows the training data collection system and method of this disclosure. [Figure 19] This figure shows an exemplary embodiment of a continuous arrangement control device of the present disclosure. [Modes for carrying out the invention]

[0043] This disclosure is applicable to many and diverse applications requiring continuous positional control of an end effector. Examples of such applications include, but are not limited to, minimally invasive procedures (e.g., endoscopic hepatectomy, necrotic tissue resection, prostatectomy, etc.), video-assisted thoracic surgery (e.g., vebetectomy, etc.), minimally vascular procedures (e.g., via catheters, sheaths, deployment systems, etc.), minimally invasive medical diagnostic procedures (e.g., intracavitary procedures via endoscopes or bronchoscopes), orthopedic procedures (e.g., via k-wires, screwdrivers, etc.), and non-medical applications.

[0044] This disclosure improves continuous position control of an end effector in such applications by providing predictions of the positional motion of the end effector and / or intervention device, which are used to control and / or ensure the manual or automatic navigated positioning of the end effector.

[0045] To facilitate understanding of this disclosure, the following description of Figure 1 teaches exemplary embodiments of intervention devices including end-effectors known in the art of this disclosure. From the description of Figure 1, those skilled in the art of this disclosure will understand how to apply this disclosure to manufacture and use further embodiments of intervention devices including end-effectors known in the art of this disclosure and those discussed below.

[0046] Referring to Figure 1, the intervention device 30 actually supports manual or automatic navigation of the end effector 40 during application, as indicated by the arrow extending from the end effector 40. Examples of the intervention device 30 include, but are not limited to, continuum flexible robots, flexible intervention scopes, and guidewires, and examples of the end effector 40 include, but are not limited to, intraoperative imaging devices, intervention tools / surgical instruments, and surgical sutures.

[0047] During operation, navigation instructions 31, activation signals 32, and / or navigation forces 33 are communicated to and provided to the intervention device 30, in which case the intervention device 30 is translated, rotated, and / or swirled according to the navigation instructions 31, activation signals 32, and / or navigation forces 33, thereby navigating the end effector 40 to a target pose (i.e., position and orientation in the application space).

[0048] For example, Figures 3A and 3B show a transesophageal echocardiography (TEE) probe 130 as an embodiment of an intervention device 30 that includes an imaging end effector 140 that can be inserted into the esophagus through the mouth of patient P to capture an image of the patient P's heart, and a physician (not shown) or a robotic control device 100 (Figure 2B) operates the handle 132 of the TEE probe 130 to reposition the TEE probe 130 within patient P, thereby navigating the imaging end effector 140 to a target pose.

[0049] Furthermore, the TEE probe 130 includes a flexible elongated member 131, a handle 132, and an imaging end effector 140. The flexible elongated member 131 is dimensioned and / or shaped, structurally positioned, and / or otherwise configured to be positioned within a patient's body lumen, such as the esophagus. The imaging end effector 140 is mounted on the distal end of the member 131 and includes one or more ultrasound transducer elements, in which case the imaging end effector 140 is configured to emit ultrasound energy toward an anatomical structure of the patient P (e.g., the heart). The ultrasound energy is reflected by the patient's vascular and / or tissue structures, in which case the ultrasound transducer elements in the imaging end effector 140 receive the reflected ultrasound echo signals. In some embodiments, the TEE probe 130 includes internal or integrated processing components that can locally process the ultrasound echo signals to generate image signals representing the anatomical structures of the patient P being imaged. In practice, the ultrasound transducer element is configured to provide two-dimensional (2D) or three-dimensional (3D) images of the anatomical structures of patient P. The images obtained by the TEE probe 130 depend on the insertion depth, rotation, and / or tilt of the imaging end effector 140, which are described in more detail herein.

[0050] The handle 132 is coupled to the proximal end of member 131. The handle 132 includes control elements for navigating the imaging end effector 140 to a target pose. As shown, the handle 132 includes knobs 133 and 134, and a switch 135. Knob 133 bends member 131 and imaging end effector 140 along the anterior-posterior plane of patient P (e.g., heart). Knob 134 bends member 131 and imaging end effector 140 along the lateral plane of patient P. Switch 135 controls beamforming in the imaging end effector 140 (e.g., adjusting the angle of the imaging plane).

[0051] In the manually navigated embodiment, the physician manually turns knobs 133 and 134 and / or switches 135 on and / or off when necessary to navigate the imaging end effector 140 to a target pose. The physician views the display of the image produced by the imaging end effector 140, thereby providing navigation force 33 (Figure 1) to control knobs 133 and 134 and / or switches 135 on the handle 132.

[0052] In an automated navigation embodiment, the robot system (not shown) includes electrical and / or mechanical components (e.g., motors, rollers, and gears) configured to turn knobs 133 and 134 and / or switch 135 on and / or off, in which case the robot control device 100 receives motion control instructions 31 (Figure 1) from a navigation control device (not shown) or an input device (not shown) to control knobs 133 and 134 and / or switch 135 on the handle 132. Alternatively, the robot control device 100 is configured to directly operate the TEE probe 130 via an actuation signal 32 (Figure 1) based on a guidance method performed by the robot control device 100.

[0053] The TEE probe 130 can be operated with various degrees of freedom. Figures 3C to 3F show various mechanisms for operating the TEE probe 130.

[0054] Figure 3C is a schematic diagram showing a TEE probe 130 being manually advanced into the patient's esophagus, as indicated by arrow 131b, or withdrawn from the patient's esophagus, as indicated by arrow 131c. The TEE probe 130 can be rotated manually or robotically to the left (e.g., counterclockwise) or to the right (e.g., clockwise) with respect to its long axis 131a, as indicated by arrows 139a and 139b. The rotation of member 131 can be described by a parameter denoted as γ.

[0055] Figure 3D is a schematic diagram showing a TEE probe 130 that is electronically rotated from 0 to 180 degrees (for example, for beamforming) by manual or robotic control of a switch 135 on a handle 132, as indicated by arrows 136a and 136b. The rotation of the imaging plane can be described by a parameter denoted as ω.

[0056] Figure 3E is a schematic diagram showing a TEE probe 130 that is bent along the anterior-posterior plane relative to the patient's heart, for example, by manually or robotically turning a knob 134 on a handle 132, as indicated by arrows 137a and 137b. The bending along the anterior-posterior plane can be described by a parameter denoted as α.

[0057] Figure 3F is a schematic diagram showing a TEE probe 130 that can be bent along the left-right plane relative to the patient's heart, for example, by manually or robotically turning a knob 133 on a handle 132, as indicated by arrows 138a and 138b. The bending along the left-right plane can be described by a parameter denoted as β.

[0058] In a further example of an exemplary embodiment of the intervention device 30 (Figure 1), Figure 3G shows a transthoracic echocardiography (TTE) probe 240 configured to capture ultrasound images of the anatomical structures of patient P from outside the patient P's body, and a robot 230, as shown, for handling the TTE probe 240 manually or robotically to reposition the TTE probe 240 to a target pose (i.e., the position and / or orientation of the TTE probe 240 relative to patient P). More specifically, the robot 230 includes a plurality of links 231 coupled to a plurality of joints 232 configured to hold the TTE probe 240 on the outer surface of patient P (e.g., around the chest area to image the heart) and to manipulate the TTE probe 240.

[0059] In the manually navigated embodiment, the physician manually applies a navigation force 33 (Figure 1) to link 231, thereby causing link 231 of the robot 230 to translate, rotate, and / or swivel in order to navigate the imaging TTE probe 240 to a target pose. The physician views a display of images generated by the TTE probe 240 to use as the basis for controlling link 231 of the robot 230.

[0060] In an automated navigation embodiment, the robot system (not shown) includes electrical and / or mechanical components (e.g., motors, rollers, and gears) configured to operate the link 231 of the robot 230, in which case the robot control device 101 receives motion control instructions 32 (Figure 1) from a navigation control device (not shown) or input device (not shown) in the form of Cartesian velocity parameters or joint velocity parameters, thereby operating the link 231 of the robot 230. Alternatively, the robot control device 101 is configured to directly operate the TTE probe 240 via actuation signals 32 (Figure 1) based on a guidance method performed by the robot control device 101.

[0061] In a further example of an exemplary embodiment of the intervention device 30 (Figure 1A), Figure 3H shows a shape-sensing guidewire 332 incorporated into or attached to a continuum robot 331 including an end effector 340. The shape-sensing guidewire 232 incorporates optical shape detection (OSS) techniques known in the art of this disclosure. More specifically, the shape detection control device 103 uses light along the multicore optical fiber 333 of the guidewire 332 for device localization and navigation during surgical intervention. The principle involved utilizes distributed distortion measurement in the optical fiber using characteristic Rayleigh backscatter or a controlled diffraction grating pattern (e.g., fiber Bragg diffraction grating). In practice, as the robot control device 102 or a physician (not shown) navigates the continuum robot 331 within patient P to position the end effector 340 in a target pose, the shape detection control device 103 acquires the superimposed shape of the continuum robot 331 via the shape-sensing guidewire 332.

[0062] To further facilitate understanding of this disclosure, the following descriptions of Figures 2A and 2B teach exemplary embodiments of the continuous position control device and the continuous position state machine of the disclosure, respectively. From the descriptions of Figures 2A and 2B, those skilled in the art will understand how to apply the disclosure to manufacture and use further embodiments of the continuous position control device and the continuous position state machine of the disclosure.

[0063] Furthermore, the TEE probe 130 (Figure 3A), robot 230 / TTE probe 240 (Figure 3G), and continuous robot 331 (Figure 3H) are used herein as non-limiting examples of the intervention device 30 including the end effector 40 (Figure 1) to support the description of various embodiments of the continuous position control device of the present disclosure. Nevertheless, those skilled in the art will understand how the present disclosure applies to various and many further embodiments of the intervention device 30 including the end effector 40.

[0064] Referring to Figures 1, 2A, and 2B, the intervention device 30, including the end effector 40, and the continuous placement control device 50 of this disclosure represent a continuous placement state machine 90.

[0065] In particular, as shown in Figure 2B, state S92 of the continuous placement state machine 90 includes the navigation of an intervention device 30, including an end effector 40, according to a specific application (e.g., minimally invasive procedure, video-assisted thoracic surgery, minimally vascular procedure, minimally medical diagnostic procedure, or orthopedic procedure). In practice, this application involves manual or automated navigation placement of the end effector 40 to a target pose, in which case the application incorporates imaging guidance (e.g., image segmentation, image overlay, path planning, etc.), intervention device tracking (e.g., electromagnetic, optical, or shape detection), and / or any other navigation techniques for positioning the end effector 40 to the target pose.

[0066] The execution of state S92 of the sequentially arranged state machine 90 results in the generation of navigation data 34 and the optional generation of auxiliary data 35. Generally, in practice, the navigation data 34 takes the form of navigation instructions 31, actuation signals 32, and / or navigation forces 33 communicated to / given to the intervention device 30, while the auxiliary data 35 takes the form of images of the intervention device 30 and / or end effector 40, operating characteristics of the intervention device 30 (e.g., shape, strain, twist, temperature, etc.), and operating characteristics of the end effector 40 (e.g., pose, force, etc.).

[0067] In response, state S94 of the continuous positioning state machine 90 includes continuous positioning control by the continuous position control device 50 of the navigation of the intervention tool 30 and the end effector 40 according to state S92. To achieve this objective, the continuous position control device 50 uses the forward prediction model 60, the inverse prediction model 70, and / or the imaging prediction model 80 of the Disclosure.

[0068] In practice, as further described herein, the forward prediction model 60 is any kind of machine learning model or equivalent (e.g., a neural network) for the regression of the positional motion 30 of the intervention device 30 to the navigated pose of the end effector 40, suitable for a particular type of intervention device 30 used in a particular type of application being performed, in which case the forward prediction model 60 is trained on the forward kinematics of the intervention device 30 to predict the navigated pose of the end effector 40.

[0069] During operation, the forward prediction model 60 receives navigation data 34 (and, if communicated, auxiliary data 35) related to the manual or automatic navigation of the intervention device 30, thereby predicting the navigated pose of the end effector 40 corresponding to the navigation of the intervention device 30, and outputs sequential placement data 51 that provides information about the placement of the end effector 40 to the target pose by the intervention device 30 based on the predicted navigated pose of the end effector 40. The sequential placement data 51 is used in state S92 as a control to determine the accuracy of the manual or automatic navigation of the intervention device 30 for placing the end effector 40 to the target pose, and / or to perform recalibration.

[0070] In practice, as further described herein, the inverse prediction model 70 is any kind of machine learning model or equivalent (e.g., a neural network) for the regression of the target pose of the end effector 40 to the positional motion of the intervention device 30, suitable for a particular type of intervention device 30 used in a particular type of application being carried out, in which case the inverse prediction model 60 is trained on the inverse kinematics of the intervention device 30 to predict the positional motion of the intervention device 30.

[0071] During operation, the inverse prediction model 70 receives navigation data 34 (and, if communicated, auxiliary data 35) related to the target pose of the end effector 40, thereby predicting the placement motion of the intervention device 30 to position the end effector 40 in the target pose, and outputs a continuous placement instruction 52 to control the placement of the end effector 40 by the intervention device 30 to the target pose based on the predicted placement motion of the intervention device 30. The continuous placement instruction 52 is used in state S92 as control for performing manual or automatic navigation of the intervention device 30 to position the end effector 40 in the target pose.

[0072] In practice, as further described herein, the imaging prediction model 80 is any kind of machine learning model or equivalent (e.g., a neural network or a scale-invariant feature transformation network) suitable for a particular type of end effector 40 used in a particular type of application being performed, in which case the inverse prediction model 60 is trained on the correlation of the relative imaging by the end effector 40 and the forward kinematics of the intervention device 30 predicting the navigated pose of the end effector 40.

[0073] During operation, the imaging prediction model 60 receives auxiliary data 35 in the form of images generated by the end effector 40 in one or more poses, thereby predicting the navigated pose of the end effector 40 as feedback data that provides information about the correct placement of the end effector 40 by the intervention device 30 to the target pose. The feedback data is used in a closed loop of state S94 to generate a difference between the target pose of the end effector 40 and the predicted navigated pose of the end effector 40, in which case the inverse prediction model 70 receives the difference to predict the correct placement movement of the intervention device 30 in order to reposition the end effector 30 to the target pose.

[0074] In practice, embodiments of the continuous placement control device 50 use a forward prediction model 60, an inverse prediction model 70, and / or an imaging prediction model 80.

[0075] For example, an embodiment of the continuous placement control device 50 uses only the forward prediction model 60 to facilitate the display of the accuracy of manual or automatic navigation of the intervention device 30 for positioning the end effector 40 into a target pose.

[0076] Furthermore, a user interface is provided to display an image of the attempted navigation of the end effector 40 to a target pose, and an image of the predicted navigated pose of the end effector 40 by the forward prediction model 60. The confidence level of the prediction is shown to the user. To evaluate the uncertainty of the prediction, multiple feedforward iterations of the forward prediction model 60 are performed with dropouts that are stochastically enabled as known in the art of this disclosure.

[0077] In a further example, an embodiment of the continuous placement control device 50 uses only the inverse prediction model 70 to instruct the intervention device 30 to manually or automatically navigate in order to position the end effector 40 in a target pose.

[0078] In a further example, an embodiment of the continuous placement control device 50 uses only an imaging predictive model 80 to provide feedback data that informs about the correct placement of the end effector 40 by the intervention device 30 to the target pose.

[0079] In a further example, an embodiment of the continuous placement control device 50 uses a forward prediction model 60 and a reverse prediction model 70 to instruct the intervention device 30 to manually or automatically navigate in order to position the end effector 40 in a target pose, and to display the accuracy of the manual or automatic navigation of the intervention device 30 in order to position the end effector 40 in a target pose.

[0080] In a further example, an embodiment of the continuous placement control device 50 uses an inverse prediction model 70 and an imaging prediction model 80 to instruct the intervention device 30 to manually or automatically navigate in order to position the end effector 40 in a target pose, and to provide feedback data that informs the intervention device 30 about the correct positioning of the end effector 40 in the target pose.

[0081] To further facilitate understanding of this disclosure, the following description of Figures 4–14 teaches exemplary embodiments of the forward predictive model, inverse predictive model, and imaging predictive model of this disclosure. From the description of Figures 4–14, those skilled in the art will understand how to apply this disclosure to manufacture and use further embodiments of the forward predictive model, inverse predictive model, and imaging predictive model of this disclosure.

[0082] Figures 4A to 4E illustrate the training and application of the forward prediction model 60a of this disclosure, which is trained on the forward kinematics of the intervention device 30 (Figure 1), in order to predict the navigated pose of the end effector 40 (Figure 1) and thereby facilitate the application of the forward prediction model 60a to the directed placement motion of the intervention device 30 to render the predicted navigated pose of the end effector 40 during the intervention procedure. In this case, the continuous placement control device 50 (Figure 1) generates continuous placement data 51 that provides information about the placement of the end effector 40 by the intervention device 30 to a target pose, based on the predicted navigated pose of the end effector 40.

[0083] Furthermore, referring in particular to Figure 4A, the training phase of the forward prediction model 60a involves the involvement of a training control device (not shown) configured to interpret the ground truth training data set D, as illustrated in the subsequent description of Figures 16-18 of this disclosure. The data set D consists of two-element tuples, i.e., d i =( T i Q iThe 2-element tuple consists of n sequences W containing i data points represented by ). The 2-element tuple consists of an end-effector pause (T∈SE(3))62a and a sequence of j consecutive joint variables (Q∈(q) acquired at consecutive time points from t to t+j. t , q t+1 ...q t+j ))61a and consists of. Element q t This represents all joint variables controlled by the robot control device (not shown).

[0084] In practice, training data set D is a collection of expert data with reasonable coverage of different navigations of the intervention device 30. To achieve this objective, the diverse data set for learning, training data set D, must incorporate manufacturing differences, performance characteristics, wear and tear of hardware components, and other system-dependent and system-independent factors between different types of robots, such as the temperature or humidity of the environment in which the robot operates.

[0085] Referring to Figure 4B, the application step of the forward prediction model 60a involves the continuous placement control device 50 executing a deep learning algorithm using the feedforward prediction model 60a for the regression of the directed placement motion of the intervention device 30 to the navigated pose of the end effector 40. Given a sequence Q of j consecutive joint variables 61b of the intervention device 30 (e.g., parameters α, β shown in Figure 3B), the forward prediction model 60a predicts the pose of the end effector 40.

number

[0086] In one embodiment shown in Figure 4E, the forward prediction model 60a uses a neural network base 160a which includes input layers, hidden layers, and output layers derived from a combination of one or more fully connected layers (FCLs) 163a, one or more convolutional layers (CNLs) 164a, one or more recurrent layers (RCLs) 165a, and one or more long-term short-term memory (LSTM) layers 166a.

[0087] In reality, the combination of layers,

number

[0088] Pose

number

[0089] Pose

number

[0090] Pose

number

[0091] In practice, the fully connected layer 163a contains K neurons, where N, M, W, and K are arbitrary positive integers, the values ​​of which vary depending on the embodiment. For example, N may be approximately 8, M approximately 2, W approximately 2, and K approximately 1000. Furthermore, the convolutional layer 164a performs a nonlinear transformation which is a composite function of operations (e.g., batch normalization, normalized linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 164a may further include a nonlinearity function (e.g., including a normalized nonlinear ReLU operation) configured to extract a normalized feature map.

[0092] In practice, one of layers 163a or 164a functions as an input layer for inputting a sequence 161a of articulated variables Q, in which case the size of the sequence of articulated variables Q is ≥ 1, and one of layers 163a, 165a, or 166a functions as an output layer for outputting a pose 162a of the end effector 40 in Cartesian space (e.g., translation and rotation of the end effector 40 in Cartesian space). The output pose of the end effector 40 in Cartesian space is expressed as a vector-parameterized and / or non-vector-parameterized rigid body position and orientation. More specifically, the parameterization is in the form of Euler angles, quaternions, matrices, exponential maps, and / or rotations and / or angular axes representing translation (e.g., direction and magnitude for translation).

[0093] Furthermore, the output layer is actually a nonlinear fully connected layer 163a that gradually reduces the high-dimensional output of the last convolutional layer 164a of the neural network base 160a to generate a set of output variables.

[0094] In training, the training weights of the forward prediction model 60a are given the input sequence Q and the output prediction model.

number

[0095] When applied, the forward prediction model 60a, given a sequence Q of j consecutive joint variables 62a, pauses the end effector 40.

number

[0096] Furthermore, referring to Figure 4E, the neural architecture of the exemplary embodiment of the forward prediction model 60a shown is such that, given a sequence Q of j consecutive joint variables 61a, the end effector 40 pauses

number

number

[0097] Referring to Figures 4C and 4D, stage S92a of an exemplary intervention procedure 90a using a TEE probe 130 (Figure 3A) and a robotic control device 100 (Figure 3B) includes the robotic control device 100 receiving the position of the TEE probe 130 as a joint variable and transmitting a movement signal to the handle 132 (Figure 3A) of the TEE probe 130. By using a motor-driven knob, the handle 132 pulls / releasing the tendon of the TEE probe 130, which results in the movement of the end effector 140 (Figures 3C-3F) to a target pose. In practice, the position of the TEE probe 130 as a joint variable is indicated by the user or an external tracking device or guidance system.

[0098] Stage S94a of process 90a involves the robot control device 100, the forward prediction model 50a, and the display control device 104. The robot control device 100 stores a sequence of consecutive joint variables (Q) 61a and the navigated pose of the end effector 140.

number

number

[0099] Figures 5A-5E predict the placement movement of the intervention device 30, thereby predicting the joint variable movement (q) 72 of the intervention device 30 t To render t , the training and application of the inverse prediction model 70a of the present disclosure trained based on the inverse kinematics of the intervention device 30 (FIG. 1) are shown, which smooths the application of the inverse prediction model 70a to the target pose (T) 71a of the end effector 40 during the intervention procedure. In this case, the continuous placement control device 50 (FIG. 1) generates a continuous placement instruction for controlling the rearrangement of the end effector 40 to the target pose by the intervention device 30 based on the predicted placement movement of the intervention device 40.

[0100] More particularly, referring to FIG. 5A, the training phase of the inverse prediction model 70a involves the participation of a training control device (not shown) configured to interpret the ground truth training data set D as exemplified in the description of FIGS. 17-18. The data set D consists of n sequences W containing i data points represented by the two-element tuple, i.e., d i =(T i ,Q i ). The two-element tuple consists of an end effector pose (T∈SE(3)) 71a and a sequence of j consecutive joint variables (Q∈(q t ,q t+1 …q t+j )) 72a acquired at consecutive time points starting from t and ending at t + j. The element q t represents all joint variables controlled by a robot control device (not shown). The variable j may further be equal to 1, which means that the prediction model is trained to infer a set of joint variables.

[0101] In practice, the training data set D is an aggregate of expert data with reasonable coverage of different navigations of the intervention device 30. To achieve this purpose, the diverse data set training data set D for learning must incorporate mechanical differences between various types of robots, wear and tear of hardware components, and other system-dependent factors.

[0102] Referring to Figure 5B, the application step of the inverse prediction model 70a involves the continuous placement control device 50 executing a deep learning algorithm using the inverse prediction model 70a for motion regression. The inverse prediction model 70a uses j consecutive joint variables 72 to reach the pose (T∈SE(3)) 71t of the end effector 40. t The sequence of parameters α and β shown in Figure 3B (for example)

number

[0103] In one embodiment shown in Figure 5E, the inverse prediction model 70a uses a neural network base 170a which includes input, hidden, and output layers derived from a combination of one or more fully connected layers (FCLs) 173a, one or more convolutional layers (CNLs) 174a, one or more recurrent layers (RCLs) 175a, and one or more long-term short-term memory (LSTM) layers 176a.

[0104] In reality, the combination of layers is an indirect variable.

number

[0105] Joint variables

number

[0106] Joint variables

number

[0107] Joint variables

number

[0108] In practice, the fully connected layer 173a contains K neurons, where N, M, W, and K are arbitrary positive integers, the values ​​of which vary depending on the embodiment. For example, N may be approximately 8, M approximately 2, W approximately 2, and K approximately 1000. Furthermore, the convolutional layer 174a performs a nonlinear transformation which is a composite function of operations (e.g., batch normalization, normalized linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 174a may further include a nonlinearity function (e.g., including a normalized nonlinear ReLU operation) configured to extract a normalized feature map.

[0109] In practice, one of layers 173a or 174a functions as an input layer for inputting the pose 171a of the end effector 40 in Cartesian space (e.g., translation and rotation of the end effector 40 in Cartesian space), and one of layers 173a, 175a, and 176a functions as an output layer for outputting a sequence 172a of joint variables Q, in which case the size of the sequence of joint variables Q is ≥ 1. The input pose of the end effector 40 in Cartesian space is expressed as a vector parameterization and / or non-vector parameterization of the rigid body position and orientation. More specifically, the parameterization is in the form of Euler angles, quaternions, matrices, exponential maps, and / or angular axes representing rotations, and / or translations, including direction and magnitude for translations.

[0110] Furthermore, the output layer is actually a nonlinear fully connected layer 173a that gradually reduces the high-dimensional output of the last convolutional layer 174a of the neural network base 170a to generate a set of output variables.

[0111] In training, the training weights of the inverse predictive model 70a are given the input - the ground truth end effector pause T - the inverse predictive model

number

[0112] When applied, the inverse prediction model 70a uses a sequence of j consecutive joint variables 72b (for example, parameters α, β shown in Figure 3B) to reach the provided pose (T∈SE(3)) 71b of the end effector 40.

number

[0113] Referring further to Figure 5E, the neural architecture of the exemplary embodiment of the inverse prediction model 70a shown is a sequence of j consecutive joint variables 72b (for example, parameters α, β shown in Figure 3B) to reach the pose (T∈SE(3)) 71b of the end effector 40.

number

[0114] Referring to Figures 5C and 5D, stage S92b of an exemplary intervention procedure 90b using the TEE probe 130 (Figure 3A) and the navigation control device 103 comprises the navigation control device 103 determining the target pose of the end effector 140 and transmitting the target pose 71a to the inverse prediction model 70a. In practice, the navigation control device 130 implements any known guidance algorithm known in the art of this disclosure.

[0115] Stage S94A of process 90b involves the involvement of the inverse prediction model 70a and the robot control device 100. The inverse prediction model 70a uses a sequence of j consecutive joint variables 72a (for example, parameters α, β shown in Figure 3B) to reach the pose (T∈SE(3)) 71a of the end effector 40.

number

[0116] Figures 6A to 6E illustrate the training and application of the forward prediction model 60b of this disclosure, which is trained based on the forward kinematics of the intervention device 30 (Figure 1), in order to predict the navigated pose of the end effector 40 (Figure 1) and thereby facilitate the application of the forward prediction model 60b to an indicated n-dimensional vector of joint velocities of the intervention device 30 in order to render the predicted linear velocity and / or angular velocity of the end effector 40 during the intervention procedure. In this case, the continuous placement control device 50 (Figure 1) generates continuous placement data 51 that provides information about the repositioning of the end effector 40 by the intervention device 30 to a target pose, based on the predicted linear velocity and / or angular velocity of the end effector 40.

[0117] Furthermore, referring in particular to Figure 6A, the training phase of the forward prediction model 60b involves the involvement of a training control device (not shown) configured to interpret the ground truth training data set D, as illustrated in the descriptions of Figures 16-18. The data set D consists of two-element tuples, i.e.

number

number

number

number

[0118] In practice, the training data set D is a collection of expert data with reasonable coverage of different navigations of the intervention device 30. To achieve this objective, the diverse data set for learning, training data set D, must incorporate mechanical differences between various types of robots, wear and tear of hardware components, and other system-dependent factors.

[0119] Referring to Figure 6B, the application step of the forward prediction model 60b involves the continuous placement control device 50 executing a deep learning algorithm using the feedforward prediction model 60b for end-effect motion regression. The forward prediction model 60b is configured to estimate the linear velocity and / or angular velocity 62b of the end-effector 40 given an n-dimensional vector of joint velocities 61b of the intervention device 30.

[0120] In one embodiment shown in Figure 6E, the forward prediction model 60b uses a neural network base 160b that includes input, hidden, and output layers derived from a combination of one or more fully connected layers (FCLs) 163b, one or more convolutional layers (CNLs) 164b, one or more recurrent layers (RCLs) 165b, and one or more long-term short-term memory (LSTM) layers 166b.

[0121] In practice, the combination of layers is configured to perform a regression of the joint velocity of the intervention device 30 to the linear velocity and / or angular velocity of the end effector 40.

[0122] In one embodiment for performing regression of the joint velocity of the intervention device 30 to the linear velocity and / or angular velocity of the end effector 40, the neural network base 160b includes a set of N fully connected layers 163b.

[0123] In a second embodiment for performing the regression of the joint velocity of the intervention device 30 to the linear velocity and / or angular velocity of the end effector 40, the neural network base 160b includes a set of N convolutional layers 164b, and thereafter a set of M fully connected layers 163b, or a set of W recurrent layers 165b, or a set of W long-term short-term memory layers 166b.

[0124] In a third embodiment for performing the regression of the joint velocity of the intervention device 30 to the linear velocity and / or angular velocity of the end effector 40, the neural network base 160b includes a set of N convolutional layers 164b, followed by a set of M fully connected layers 163b, and a set of W recurrent layers 165b, or a set of W long-term short-term memory layers 166b.

[0125] In practice, the fully connected layer 163b contains K neurons, where N, M, W, and K are arbitrary positive integers, the values ​​of which vary depending on the embodiment. For example, N may be approximately 8, M approximately 2, W approximately 2, and K approximately 1000. Furthermore, the convolutional layer 164b performs a nonlinear transformation which is a composite function of operations (e.g., batch normalization, normalized linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 164b may further include a nonlinearity function (e.g., including a normalized nonlinear ReLU operation) configured to extract a normalized feature map.

[0126] Furthermore, in practice, one of layers 163b or 164b is a sequence of j consecutive joint velocities.

number

number

number

[0127] In training, the training weights of the forward prediction model 60b are - given a sequence of joint velocities - predicted linear velocity and angular velocity via the forward prediction model

number

[0128] When applied, the forward prediction model 60b estimates the linear velocity and / or angular velocity 62b of the end effector 40, given a sequence of joint velocities 61b of the intervention device 30.

[0129] Referring further to Figure 6E, the neural architecture consists of an input, the aforementioned neural network base, and an output. The input is a sequence of joint velocities, and the output is linear and angular velocities that can be regressed from a fully connected layer containing 6 units. The loss function is:

number

[0130] Referring to Figures 6C and 6D, stage S92c of an exemplary intervention procedure 90c using robot 230 (Figure 3A), TTE probe 240 (Figure 3A), and robot control device 101 (Figure 3G) comprises the robot control device 101 receiving the position of the TTE probe 240 as an n-dimensional vector of the joint velocity 61b of the intervention device 30, and transmitting the n-dimensional vector of the joint velocity 61b of the intervention device 30 to robot 230. In practice, the position of the TTE probe 240 as an n-dimensional vector of the joint velocity 61b of the intervention device 30 is indicated by the user or an external tracking device or guidance system.

[0131] Stage S94c of process 90c involves the involvement of a robot control device 101, a forward prediction model 50a, and a display control device 104. The robot control device 101 stores an n-dimensional vector of the joint velocity 61b of the intervention device 30 and communicates it to the forward prediction model 60b, which predicts the linear velocity and / or angular velocity 62b of the TTE probe 240. The continuous placement control device 50c generates a confidence level of the prediction derived from a number of uncertain feedforward iterations of the forward prediction model 60b, which are performed with stochastically enabled dropout as known in the art of this disclosure. The forward prediction model 60b then derives the predicted navigated pose of the TTE probe 240 from the predicted linear velocity and / or angular velocity 62b of the TTE probe 240.

number

[0132] Figures 7A to 7E illustrate the training and application of the inverse prediction model 70b of this disclosure, which is trained based on the inverse kinematics of the intervention device 30 (Figure 1), in order to predict the positional motion of the intervention device 30 and thereby facilitate the application of the inverse prediction model 70b to the target pose of the end effector 40 for rendering the predicted positional motion of the intervention device 30 during the intervention procedure. In this case, the continuous positioning control device 50 (Figure 1) generates continuous positioning instructions that control the repositioning of the end effector 40 by the intervention device 30 to the target pose based on the predicted positional motion of the intervention device 40.

[0133] Furthermore, referring in particular to Figure 7A, the training phase of the inverse prediction model 70b involves the involvement of a training control device (not shown) configured to interpret the ground truth training data set D, as illustrated in the description of Figures 17-18. The data set D consists of two-element tuples, i.e.

number

number

number

[0134] In practice, the training data set D is a collection of expert data with reasonable coverage of different navigations of the intervention device 30. To achieve this objective, the diverse data set for learning, training data set D, must incorporate mechanical differences between various types of robots, wear and tear of hardware components, and other system-dependent factors.

[0135] Referring to Figure 7B, the application step of the inverse prediction model 70b involves the continuous placement control device 50 executing a deep learning algorithm using the inverse prediction model 70b for joint velocity regression. The inverse prediction model 70b is configured to estimate an n-dimensional vector of joint velocities 61b of the intervention device 30, given the linear velocity and / or angular velocity 62b of the end effector 40.

[0136] In one embodiment shown in Figure 7E, the inverse prediction model 70b uses a neural network base 170b which includes input, hidden, and output layers derived from a combination of one or more fully connected layers (FCLs) 173b, one or more convolutional layers (CNLs) 174b, one or more recurrent layers (RCLs) 175b, and one or more long-term short-term memory (LSTM) layers 176b.

[0137] In practice, the combination of layers is configured to perform regression of the linear velocity and / or angular velocity of the end effector 40 to the joint velocity of the intervention device 30.

[0138] In one embodiment for performing regression of the linear velocity and / or angular velocity of an end effector 40 to the joint velocity of an intervention device 30, the neural network base 170b includes a set of N fully connected layers 173b.

[0139] In a second embodiment for performing regression of the linear velocity and / or angular velocity of an end effector 40 to the joint velocity of an intervention device 30, the neural network base 170b includes a set of N convolutional layers 174b, and thereafter a set of M fully connected layers 173b, or a set of W recurrent layers 175b, or a set of W long-term short-term memory layers 176b.

[0140] In a third embodiment for performing regression of the linear velocity and / or angular velocity of the end effector 40 to the joint velocity of the intervention device 30, the neural network base 170b includes a set of N convolutional layers 174b, and a combination of a set of M fully connected layers 173b and a set of W recurrent layers 175b or a set of W long-term short-term memory layers 176b.

[0141] In practice, the fully connected layer 173b contains K neurons, where N, M, W, and K are arbitrary positive integers, the values ​​of which vary depending on the embodiment. For example, N may be approximately 8, M approximately 2, W approximately 2, and K approximately 1000. Furthermore, the convolutional layer 174b performs a nonlinear transformation which is a composite function of operations (e.g., batch normalization, normalized linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 174b further includes a nonlinearity function (e.g., including a normalized nonlinear ReLU operation) configured to extract a normalized feature map.

[0142] Furthermore, in practice, one of layers 173b or 174b is the angle and linear velocity

number

number

[0143] In training, the training weights of the inverse prediction model 70b are the predicted sequence of joint velocities.

number

number

number

[0144] When applied, the inverse prediction model 70b estimates the n-dimensional vector of the joint velocity 61b of the intervention device 30, given the linear velocity and / or angular velocity 62b of the end effector 40.

[0145] Referring further to Figure 7E, the neural architecture of an exemplary embodiment of the inverse velocity model consists of an input, a neural network base, and an output. The input is angular velocity and linear velocity.

number

number

[0146] Referring to Figures 7C and 7D, stage S92d of an exemplary intervention procedure 90d using the TTE probe 240 (Figure 3G) and the navigation control device 103 comprises the navigation control device 103 determining the linear velocity and / or angular velocity 62b of the TTE probe 240 to the target pose and transmitting the linear velocity and / or angular velocity 62b to the inverse prediction model 70b. In practice, the navigation control device 130 implements any known guidance algorithm known in the art of this disclosure.

[0147] Stage S94d of process 90d involves the involvement of the inverse prediction model 70b and the robot control device 101. Given linear velocity and / or angular velocity 62b, the inverse prediction model 70b estimates the n-dimensional vector of joint velocity 61b of the intervention device 30, and the continuous placement control device 50c communicates a continuous placement instruction 52b to the robot control device 101, thereby controlling the placement of the TTE probe 240 via the robot 230 (Figure 3G) to the target pose based on the predicted placement motion of the intervention device 40.

[0148] In practice, the forward prediction model 60a (Figure 4A), the inverse prediction model 70a (Figure 5A), the forward prediction model 60b (Figure 64A), and the inverse prediction model 70b (Figure 7A) utilize further auxiliary information, such as images of anatomical structures (e.g., ultrasound, endoscopy, or X-ray images), forces measured at the end effector, and the shape of the robot. Depending on the application, other inputs may further include information from spectral tissue detection devices, ECG or EEG signals, tissue conductivity, or other physiological signals. For example, when a continuum-like robot operates inside a human heart, features available in ultrasound images and electrophysiological signals can improve the positioning of the end effector relative to anatomical structures, and thus improve the accuracy of guidance.

[0149] Referring to Figures 8A and 8B, the forward prediction model 60c is shown to be trained based on the sequence Q of the joint variable 61a and the end effector pose T62a, as well as the forward kinematics of the intervention device, in addition to the shape 35a, image 35b, and force 35c of the intervention device. Therefore, when applied, the forward prediction model 60c is capable of predicting the navigated pose of the end effector from the sequence Q of the joint variable 61a, and the shape 35a, image 35b, and force 35c of the intervention device.

[0150] Those skilled in the art will understand how to apply the shape 35a, image 35b, and force 35c of the intervention device, and any other further auxiliary information, to the inverse prediction model 70a, the forward prediction model 60b, and the inverse prediction model 70b.

[0151] Figures 9A to 9E illustrate the training and application of the forward prediction model 60d of this disclosure, which is trained based on the forward kinematics of the intervention device 30 (Figure 1), in order to predict the navigated pose and robot shape of the end effector 40 (Figure 1) and thereby facilitate the application of the forward prediction model 60d to a sequence of continuous shapes of the intervention device 30 with built-in OSS technology for rendering the predicted navigated pose and shape of the end effector 40 during the intervention procedure. In this case, the continuous placement control device 50 (Figure 1) generates continuous placement data 51c that provides information about the repositioning of the end effector 40 by the intervention device 30 to a target pose, based on the predicted navigated pose of the end effector 40.

[0152] Furthermore, referring specifically to Figure 9A, the training phase involves the involvement of a training control device (not shown) configured to interpret the ground truth training data set D, as illustrated in the descriptions of Figures 16-18. This data set consists of two-element tuples, i.e., d i =( H i ,H i+1 It consists of n sequences W containing i data points represented by ). This 2-element tuple consists of k consecutive sequences of shape 61d (Hi ∈(h t h t+1 ...h t+k )) consists of, where h∈(p1…p m ) represents m vectors p that describe both the position of the OSS sensor incorporated into the intervention device 30 (e.g., a shape-detecting guidewire) in 3D Euclidean space, and auxiliary shape parameters such as strain, curvature, and twist. m This is a set of . This two-element tuple is further H i+1 ∈(h t+1 h t+2 ...h t+k+1 ) and other future points in time h t+k+1 It consists of a sequence of k consecutive shapes 62b, where h∈(p1…p m ) represents m vectors p that describe both the position of the OSS intervention device 30 in 3D Euclidean space and auxiliary shape parameters such as strain, curvature, and twist. m It is a set of.

[0153] In practice, training data set D is a collection of expert data with reasonable coverage of different navigations of the OSS intervention device 30. To achieve this objective, the diverse data set for learning, training data set D, must incorporate anatomical structures with different curvatures, magnitude of movement, mechanical differences between different types of robots, wear and tear of hardware components, and other system-independent factors, such as ambient temperature and humidity.

[0154] Referring to Figure 9B, the application stage of the forward prediction model 60d involves the continuous placement control device 50d running a deep learning algorithm with a forward layer that includes a recurrent layer trained on expert data with reasonable coverage of different examples. The diverse data set for training incorporates differences in various working conditions (temperature, fiber bending, etc.), different operating motions of the devices, and hardware (fibers, interrogators, etc.).

[0155] In one embodiment shown in Figure 9E, the forward prediction model 60d uses a neural architecture comprising, in sequence, an input layer 163a, an inter-sequence model 263a, an output layer 262a, and an extraction layer 264. The neural architecture consists of k shapes representing future sequences.

number

number

[0156] During training, the training weights of the forward prediction model 60d are -input sequence H i The sequence of future shapes predicted by the given model.

number

[0157] When applied, the forward prediction model 60d predicts the future sequence consisting of k shapes.

number

number

[0158] In an alternative embodiment shown in Figure 9F, the forward prediction model 60d uses a many-to-one model 263b instead of the inter-sequence model 263a, in which case the final layer is the last shape.

number

[0159] Referring to Figures 9C and 9D, stage S92e of an exemplary intervention procedure 90e using the OSS guidewire 332 (Figure 3F) and the shape detection control device 103 (Figure 3F) comprises the shape detection control device 103 measuring and storing the shape of the OSS guidewire 332 while the end effector 340 is navigating to a target pose.

[0160] Stage S94e of process 90e involves the shape detection control device 103, the forward prediction model 50d, and the display control device 104. The shape detection control device 103 communicates a sequence 61d of k consecutive shapes to the forward prediction model 60e, thereby predicting the subsequent sequence of shapes.

number

number

[0161] Referring to Figures 11A and 11B, the imaging prediction model 80a of this disclosure is trained on expert data with reasonable coverage of different examples. Given an ultrasound image 81a, this neural network 80a infers the relative position 82a of this image 81a to a reference anatomical structure. As a result, the motion of the end effector in the anatomical structure (e.g., the heart) between the previous pose and the current pose of the end effector is calculated, as will be described in more detail herein.

[0162] Figure 11A shows batch-wise training of the imaging prediction model 80a. During training, the network continuous imaging prediction model 80a updates its weights using two-element tuples from a ground truth data set consisting of ultrasound images and the relative positions of these images to reference anatomical structures.

[0163] Figure 11B shows real-time inference using an imaging prediction model 80a that predicts the relative pose 82a of the end effector to a reference anatomical structure (e.g., a reference ultrasound image) given an image 81a.

[0164] In the training phase, the data acquisition control device (not shown) interacts with the robot and end effector (e.g., an ultrasonic device) and has the following specifications: the training data set D is a 2-element tuple, i.e., d i =(U i ,T i The data acquisition control device (not shown) is configured to receive and interpret information from both the data and the data stored in the data storage medium (not shown) in a format defined by consisting of i data points represented by ). This two-element tuple represents the ultrasound image U acquired at a specific position T∈SE(3)82a relative to the reference position. i It consists of 81a.

[0165] The training control device is configured to interpret a training data set D stored in a data storage medium. This data set D consists of i data points represented by a two-element tuple, i.e., d i =(U i ,T i ). This two-element tuple consists of an ultrasonic image U i 81a of an anatomical structure, and a relative movement T i 82a between the current pose of the end effector from which the ultrasonic image U i was acquired and some arbitrarily selected reference position.

[0166] In one embodiment shown in FIG. 11E, the image prediction model 80a uses a neural network-based 180a including an input layer, a hidden layer, and an output layer derived from a combination of one or more fully connected layers (FCL) 183a, one or more convolutional layers (CNL) 184a, one or more recurrent layers (RCL) 185a, and one or more long short-term memory (LSTM) layers 186a.

[0167] In practice, the combination of layers is configured to realize the relative arrangement of the image U C with respect to the reference image, and thus the pose

number

[0168] In one embodiment for realizing the relative arrangement of the image U C with respect to the reference image, and thus the pose

number

[0169] In one embodiment for realizing the relative arrangement of the image U C with respect to the reference image, and thus the pose

number

[0170] Image U relative to the reference image C The relative arrangement, therefore the pose

number

[0171] In practice, the fully connected layer 183a contains K neurons, where N, M, W, and K are arbitrary positive integers, the values ​​of which vary depending on the embodiment. For example, N may be approximately 8, M approximately 2, W approximately 2, and K approximately 1000. Furthermore, the convolutional layer 184a performs a nonlinear transformation which is a composite function of operations (e.g., batch normalization, normalized linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 184a further includes a nonlinearity function (e.g., including a normalized nonlinear ReLU operation) configured to extract a normalized feature map.

[0172] In practice, one of layers 183a or 184a functions as an input layer for inputting an image Uc, and one of layers 183a, 185a, and 186a functions as an output layer for outputting the pose 182a of the end effector 40 in Cartesian space (e.g., the translation and rotation of the end effector 40 in Cartesian space). The output pose of the end effector 40 in Cartesian space is expressed as a vector-parameterized and / or non-vector-parameterized rigid body position and orientation. More specifically, the parameterization is in the form of Euler angles, quaternions, matrices, exponential maps, and / or angular axes representing rotations, and / or translational translations (including direction and magnitude for translational translations).

[0173] Furthermore, the output layer is actually a nonlinear fully connected layer 183a that gradually reduces the high-dimensional output of the last convolutional layer 184a of the neural network base 180a in order to generate a set of output variables.

[0174] In training, the training weights of the image prediction model 80a are used to predict the relative motion of an end effector relative to some criterion anatomical structure, given an ultrasound image 161c as input.

number

[0175] Referring to Figures 12A and 12B, the imaging prediction model 80ba of this disclosure is trained on expert data with reasonable coverage of different examples. Given an ultrasound image 81a, this neural network 80ba infers the relative position 82a of this image 81a to a reference anatomical structure. As a result, the motion of an end effector in an anatomical structure (e.g., the heart) between its previous pose and its current pose is calculated, as will be described in more detail herein.

[0176] Figure 12A shows batch-wise training of the imaging prediction model 80ba. During training, the network continuous imaging prediction model 80ba updates its weights using two-element tuples from a ground truth data set consisting of ultrasound images and the relative positions of these images to reference anatomical structures.

[0177] Figure 12B shows real-time inference using an imaging prediction model 80ba that predicts 83a the linear and angular velocities of an end effector relative to a reference anatomical structure (e.g., a reference ultrasound image) given an image 81a.

[0178] In the training phase, the data acquisition control device (not shown) interacts with the robot and end effector (e.g., an ultrasonic device) and has the following specifications: the training data set D is a 2-element tuple, i.e., d i =(U i ,V i The data acquisition control device (not shown) is configured to receive and interpret information from both the data and the data stored in the data storage medium (not shown) in a format defined by i data points represented by ). This two-element tuple is used to obtain an ultrasonic image U acquired at a specific position T∈SE(3)82a relative to the reference position via the linear velocity and angular velocity vectors 83a of the end effector. i It consists of 81a.

[0179] The training control device is configured to interpret a training data set D stored in a data storage medium. This data set D consists of i data points represented by a two-element tuple, that is, d i =(U i ,V i ). This two-element tuple consists of an ultrasonic image U i 81a of an anatomical structure, and the linear and angular velocities of the end effector from which the ultrasonic image U i was acquired, and a relative n-dimensional vector 83a of some arbitrarily selected reference position.

[0180] In one embodiment shown in FIG. 12E, the image prediction model 80b uses a neural network-based 180b including an input layer, a hidden layer, and an output layer derived from a combination of one or more fully connected layers (FCL) 183b, one or more convolutional layers (CNL) 184b, one or more recurrent layers (RCL) 185b, and one or more long short-term memory (LSTM) layers 186b.

[0181] In practice, the combination of layers is configured to realize the relative arrangement of the image U C with respect to the reference image, thereby deriving the linear velocity and / or angular velocity of the end effector 40.

[0182] In one embodiment for implementing the relative arrangement of the image U C with respect to the reference image, thereby deriving the linear velocity and / or angular velocity of the end effector 40, the neural network-based 180b includes a set of N fully connected layers 183b.

[0183] In a second embodiment for implementing the relative arrangement of the image U C with respect to the reference image, thereby deriving the linear velocity and / or angular velocity of the end effector 40, the neural network-based 180b includes a set of N convolutional layers 184b, and then a set of M fully connected layers 183b, or a set of W recurrent layers 185b, or a set of W long short-term memory layers 186b.

[0184] Image U relative to the reference image C In a third embodiment for performing a relative arrangement of the elements and thereby deriving the linear and / or angular velocity of the end effector 40, the neural network base 180b includes a set of N convolutional layers 184b, followed by a set of M fully connected layers 183b, and a set of W recurrent layers 185b, or a set of W long-term short-term memory layers 186b.

[0185] In practice, the fully connected layer 183b contains K neurons, where N, M, W, and K are arbitrary positive integers, the values ​​of which vary depending on the embodiment. For example, N may be approximately 8, M approximately 2, W approximately 2, and K approximately 1000. Furthermore, the convolutional layer 184b performs a nonlinear transformation which is a composite function of operations (e.g., batch normalization, normalized linear unit (ReLU), pooling, dropout, and / or convolution), and the convolutional layer 184b further includes a nonlinearity function (e.g., including a normalized nonlinear ReLU operation) configured to extract a normalized feature map.

[0186] Furthermore, in reality, one of layers 183b or 184b is image U C It functions as an input layer for inputting the linear and angular velocities of the end effector, with one of layers 183b, 185b, and 186b regressing from the last fully connected layer (e.g., 6 units, 3 units for linear velocity and 3 units for angular velocity) using a linear or nonlinear activation function.

number

[0187] During training, the training weights of the image prediction model 80b are continuously updated by comparing the predicted linear and angular velocities of an end-effector relative to some reference anatomical structure with the ground truth end-effector linear and angular velocities describing the motion of some reference anatomical structure from batches of training data set D, which are systematically or randomly selected from data memory (not shown). Furthermore, the coefficients for the filter are initialized using default or arbitrary values. The coefficients for the filter are applied to batches of training data set D via forward propagation and adjusted via backward propagation to minimize any output error.

[0188] Referring to Figures 13A and 13B, the continuous position control device 50f uses an inverse prediction model 70a, an image prediction model 80a, a subtractor 53, and a control rule 54a to perform the closed-loop continuous position control method of the present disclosure, as shown in flowchart 190a.

[0189] In one embodiment of a TEE probe, stage S192a of step 190a includes the insertion of a TEE probe handle 132 (Figure 3A) into a robotic control device 100 (Figure 3B), as known in the art of this disclosure, to control the dial on the handle 132 and the rotation of the TEE probe 130. The TEE transducer 140 (Figure 3A) is inserted into the body through the esophagus and positioned near an anatomical structure of interest, such as the heart, for example, at the midesophageal position shown in Figure 13C. Ultrasound imaging parameters are defined at this target pose of the TEE transducer 140.

[0190] Stage S194a of process 190a involves the image prediction model 90 determining the relative position of this image plane to a reference anatomical structure.

number

[0191] T d Based on this, the inverse prediction model 70a determines the joint variables required to move the robot to the desired position.

number

[0192] Stage S196a of process 190a is an ultrasound image U c The ultrasound transducer reaches another position where the image is obtained. The image prediction model 90g determines the relative position of both current image planes to a reference anatomical structure.

number

number

[0193] In the second embodiment of the TEE probe, as shown in Figure 13D, the selection of the path in the image 201 generated by an external imaging modality superimposed on the ultrasound image is performed at the desired position T dIt is used to identify (for example, X-ray images and cone-beam CT images may be superimposed on ultrasound images using methods known in the art (Philips EchoNavigator)).

[0194] Referring to Figures 14A and 14B, the continuous position control device 50g uses an inverse prediction model 70b, an image prediction model 80b, a subtractor 53, and a control rule 54b to perform the closed-loop continuous position control method of the present disclosure, as shown in flowchart 190b.

[0195] In one embodiment of a TEE probe, stage S192b of step 190b includes the insertion of a TEE probe handle 142 (Figure 3A) into a robotic control device 100 (Figure 3B), as known in the art of this disclosure, to control the dial on the handle 142 and the rotation of the TEE probe 140. The TEE transducer 140 (Figure 3A) is inserted into the body through the esophagus and positioned in the vicinity of an anatomical structure of interest, such as the heart, in a mid-esophageal position, as shown in Figure 14C. Ultrasound imaging parameters are defined in this target pose of the TEE transducer 140.

[0196] Stage S194b of process 190b allows the user to move the transducer in image space by selecting, for example, a path from point A to point B in the ultrasound image 203, or a transformation between image plane A and image plane B. In this embodiment, the first linear velocity and angular velocity 202 defined by the path in the image are calculated by knowing the spatial relationship between the end effector and the image coordinate system using methods known in the art of this disclosure. end-effector J image The coordinate system is transformed to the end-effector coordinate system using Jacobian 204.

[0197] V d Based on this, the inverse prediction model 70b (Figure 7B) predicts the joint velocity required to move the robot to the desired position.

number

number

[0198] Stage S196b of process 190a is an ultrasonic transducer that takes an ultrasonic image U c The image prediction model 80b predicts the velocity vector 83a of the end effector between point A and point B, using the current ultrasound image U c Process 81a. The image prediction model 80b estimates a function between the Cartesian velocity of the end effector and the velocity in joint space, i.e., given a 6-dimensional vector consisting of the linear velocity and angular velocity of the end effector 71c to predict the n-dimensional vector 72c of joint velocity, the neural network models the manipulator Jacobian.

[0199] As will be understood by those skilled in the art of this disclosure, a neural network that models the spatial relationships between images of anatomical structures, such as the heart, requires a large set of training data specific to a given organ.

[0200] In an alternative embodiment, features are extracted directly from the image to verify the position of the transducer. In this embodiment shown in Figure 14C, the user selects an object in image 205, for example, an apical wall, and the system extracts specific prominent features around the object via feature extraction 206. These features, encompassing the edges, shape, and size of the object, are first detected using methods known in the art of this disclosure (e.g., Canny edge detector, morphological computation, etc.) and then tracked using scale-invariant feature transformation (SIFT) known in the art of this disclosure. Finally, the system defines a path between the selected object and the center of the field of view that shows the motion required in the image. By tracking prominent features using SIFT, the continuous position control device 50g (Figure 14A) can modify predictions from the network within closed-loop control.

[0201] A more specific, velocity-based control system for a continuum-like robot. The desired motion in the image is immediately identified when the user selects a specific object in the image, e.g., the apical wall (see the red dot in the ultrasound image). The motion is defined by the path between the center of the field of view and the selected object, which can be converted into linear and angular velocities in end-effector space using Jacobian 204. This Cartesian velocity is then sent to a neural network that estimates joint velocities. The reached position is repeatedly validated against the path defined by the continuously tracked object and the center of the field of view.

[0202] In practice, the closed control loops in Figures 13 and 14 are closed using other modalities, such as optical shape detection (OSS), electromagnetic tracking, or X-ray images superimposed on ultrasound images using, for example, a Philips EchoNavigator.

[0203] Furthermore, in practice, the prediction accuracy of a neural network can be affected by the configuration of a flexible endoscope. Therefore, the position of the transducer relative to the heart is first defined implicitly using one of the possible configurations, for example, a neural network g or the Philips HeartModel. Then, a specific set of network weights is loaded into the model according to the detected configuration, thus improving the prediction accuracy.

[0204] A similar approach can be used to guide users to locations where optimal images and guidance can be provided.

[0205] Furthermore, one of the most challenging problems in machine / deep learning is the accessibility of large datasets in the correct format for training predictive models. Moreover, collecting and constructing training and validation sets is extremely time-consuming and expensive, particularly because it requires domain-specific knowledge. For example, training a predictive model to accurately distinguish between benign and malignant breast tumors requires thousands of ultrasound images annotated by expert radiologists and converted into numerical representations that the training algorithm can understand. Furthermore, image data sets are often inaccurate, error-filled, or noisy, all of which contribute to inaccuracies in detection, and acquiring large medical data sets raises ethical and privacy concerns, as well as many other concerns.

[0206] Referring to Figure 15, the training data acquisition system of this disclosure uses a shape detection control device 103 that provides a 3D position vector 233 for each point in an optical shape detection fiber 332 mounted on both the robot's endoscope 131 and tendon-driven manipulator 231. The distal end of the optical fiber 332 is embedded in a plastic case that rigidly connects the optical fiber to an end effector and introduces a specific curve into the shape.

[0207] For example, Figures 16A and 16B show the distal end 332d of an optical shape-detecting fiber 332 embedded in a plastic casing 350 that is rigidly mounted to the ultrasonic transducer 232 of the manipulator 231, as shown in Figures 16C and 16D. The plastic casing 350 reinforces a specific curvature in the shape and thus enables estimation of the end effector pose using template matching algorithms known in the art of this disclosure (for example, the data acquisition sequence 370 shown in Figure 17 with alpha and beta in each axis corresponds to the knob position on the TEE handle 132 shown in Figure 15).

[0208] Referring again to Figure 16, generally, by detecting this pattern, the pose T∈SE(3) of the end effector can be estimated using methods known in the art. The robot control device 100 transmits motion instructions to a robot that controls an actuated knob responsible for pulling or releasing a tendon. By changing the state of the tendon, the position of the end effector is changed as shown in Figures 3D and 3E. The data storage control device 190 receives the shape of the optical fiber h∈(p1…p) from the shape detection and the robot control device, respectively. n ), the pose T of the end effector, and the movement instruction, i.e., joint position q t The data is received. The data is stored in a storage device as a 3-element tuple and used later for training the deep convolutional neural network of this disclosure.

[0209] Furthermore, the shape-detected guidewire 332 is incorporated into or mounted on a continuum robot using optical shape detection (OSS) technology known in the art. OSS uses light along a multicore optical fiber for device positioning and navigation during surgical intervention. The principle involved uses distributed distortion measurement in the optical fiber with characteristic Rayleigh backscatter or controlled diffraction grating patterns.

[0210] The shape detection control device 103 is configured to acquire the shape of the shape-detected guidewire 322 and to estimate the pose T ∈ SE(3) of an end effector rigidly mounted to a plastic casing 350 that forces a specific curvature in the guidewire, as already described herein. The method for estimating the pose T is based on a well-defined curvature and a template matching algorithm known in the art of this disclosure.

[0211] The data acquisition control device 191 is configured to generate a sequence of motor instructions according to a predetermined acquisition pattern (e.g., spiral, radial, or quadrilateral motion) and to send movement commands to the robot control device 100.

[0212] The robot control device 100 is configured to receive the robot's position and to transmit motion signals to the robot. By using a motor-driven knob, the robot pulls / relaxes a tendon, which results in the movement of the end effector. The robot control device is further configured to receive and interpret information from the data acquisition control device 191 and to change the robot's position based on the information from the data acquisition control device 191.

[0213] The data storage control device 190 is configured to receive and interpret information from both the robot control device 100 and the shape detection control device 103, and to store the data in a data storage medium (not shown) in the format specified by the following specifications.

[0214] The first specification is for acquiring a training data set D for all configurations defined by the data acquisition control device 191. The data set D is a set of n sequences W, i.e., D = {W1, W2, ..., W n} consists of each sequence W n This consists of i data points d i , in other words, W n ={d1,d2,…,d i} and each data point d from the sequence i is a 3 - element tuple, i.e., d i =(T i , H i , Q i ), where.

[0215] The 3 - element tuple consists of an end - effector pose T ∈ SE(3), a sequence of k consecutive shapes, e.g., H ∈ (h t , h t+1 … h t+k ), where h ∈ (p1… p m ), a set of m vectors p m that describe both the position of a guide wire where the shape is detected in 3 - D Euclidean space and auxiliary shape parameters such as strain, curvature, and torsion, a sequence of k consecutive shapes, and a sequence of j consecutive joint variables Q ∈ (q t , q t+1 … q t+j ), obtained at times starting from t and up to t + j. For example, the element q t can be the angles α, β at a control knob obtained at time t.

[0216] Referring to FIG. 19, the training data collection method 360 of the present disclosure is executed by the training data collection system of FIG. 15.

[0217] Referring to both FIGS. 15 and 19, stage S362 of method 360 has the robot being moved to the home position by the robot control device 100 using, for example, a limit switch or proximity sensor. The distal portion of the shape - detected guide wire 232 is inserted into a recess 353 provided in the plastic casing 350. This recess 353 enforces a specific curvature in the shape.

[0218] The casing 350 is rigidly attached to the end - effector of the continuum - like robot.

[0219] By using a template matching algorithm known in the technical field of the present disclosure during stage S364 of method 360, the shape detection control device 103 can estimate the pose T∈SE(3) of the end effector at this stage. Preferably, the coordinate system of the end effector is defined by the template, but a further calibration matrix can be used. When the robot system is stationary at the home position, the pose of the end effector is acquired in the OSS coordinate system. Each subsequent pose acquired during the experiment is estimated relative to this initial position.

[0220] The data acquisition control device 191 generates a motion sequence, that is, a set of joint variables, according to a predetermined acquisition pattern (for example, pattern 370 in FIG. 18). The motion sequence is repeatedly transmitted by the data acquisition control device 191 to the robot control device 100 that moves the robot according to the generated joint variables.

[0221] Stage S366 of method 300 involves the acquisition and storage of the data tuple d i =(T i ,H i ,Q i ) at each time point. Importantly, since H i and Q i are continuous, all previous time points are maintained in memory by the data storage control device 190.

[0222] To facilitate a deeper understanding of the various inventions of the present disclosure, the following description of FIG. 19 teaches an exemplary embodiment of the continuous placement control device of the present disclosure. From this description, those skilled in the art will understand how to apply the various aspects of the present disclosure to manufacture and use further embodiments of the continuous placement control device of the present disclosure.

[0223] Referring to Figure 19, the continuous placement control device 400 includes one or more processors 401, memory 402, user interface 403, network interface 404, and storage 405 interconnected via one or more system buses 406.

[0224] Each processor 401 may be any hardware device known in the art of this disclosure or considered below that is capable of executing instructions stored in memory 402 or storage, or otherwise processing data. In non-limiting examples, processor 401 includes microprocessors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other similar devices.

[0225] Memory 402 encompasses various types of memory known in the art of this disclosure or discussed below, including but not limited to L1, L2, or L3 caches or system memory, such as non-temporary and / or static memory. In non-limiting examples, memory 402 encompasses static random-access memory (SRAM), dynamic RAM (DRAM), flash memory, read-only memory (ROM), or other similar memory devices.

[0226] The user interface 403 includes one or more devices known in the art of this disclosure or considered below for enabling communication with users, such as administrators. In non-limiting examples, the user interface includes a command-line interface or a graphical user interface presented to a remote terminal via the network interface 404.

[0227] The network interface 404 includes one or more devices known in the art of this disclosure or considered below for enabling communication with other hardware devices. In a non-limiting example, the network interface 404 includes a network interface card (NIC) configured to communicate in accordance with the Ethernet protocol. Furthermore, the network interface 404 may implement a TCP / IP stack for communication in accordance with the TCP / IP protocol. Various alternative or additional hardware or configurations for the network interface 404 will become apparent.

[0228] Storage 405 includes, but is not limited to, read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, or similar storage media, one or more machine-readable storage media known in the art of this disclosure or considered below. In various non-limiting embodiments, storage 405 stores instructions for execution by processor 401 or data for which processor 401 performs calculations. For example, storage 405 stores a basic operating system for controlling various basic operations of the hardware. Storage 405 further stores application modules in the form of executable software / firmware for implementing various functions of the control unit 400a, which have been described in this disclosure, including, but are not limited to, the forward prediction model 60, the inverse prediction model 70, and the imaging prediction model 80, which have been described in this disclosure.

[0229] In practice, the control device 400 may be installed in the X-ray imaging system 500, the intervention system 501 (e.g., an intervention robot system), or a standalone workstation 502 (e.g., a client workstation or mobile device, e.g., a tablet) that communicates with the X-ray imaging system 500 and / or the intervention system 501. Alternatively, the components of the control device 400 may be distributed among the X-ray imaging system 500, the intervention system 501, and / or the standalone workstation 502.

[0230] Furthermore, in practice, further control devices of the Disclosure, including shape detection control devices, data storage control devices, and data acquisition control devices, may each further include one or more processors, memory, user interfaces, network interfaces, and storage interconnected via one or more system buses configured in Figure 19, in which case the storage includes applicable application modules for its control device, as already described herein. Alternatively, two or more control devices of the Disclosure may be integrated into a single control device, where the storage includes applicable application modules for the two or more control devices, as already described herein.

[0231] A person skilled in the art will understand the many advantages of this disclosure by referring to Figures 1 to 19.

[0232] Furthermore, those skilled in the art will understand, in consideration of the teachings provided herein, that structures, elements, components, etc., described and / or shown in the figures may be implemented by various combinations of hardware and software, and may provide functions combined in one or more elements. For example, the functions of various structures, elements, components, etc., shown / exemplified / depicted in the figures may be provided through the use of dedicated hardware and hardware capable of running software in conjunction with appropriate software for additional functions. Where provided by a processor, the functions may be provided by one dedicated processor, one shared processor, or several individual processors, some of which may be shared and / or multiplexed. Furthermore, the explicit use of the terms “processor” or “control device” should not be interpreted as exclusively representing hardware capable of running software, but implicitly encompasses, but is not limited to, digital signal processor ("DSP") hardware, memory (e.g., read-only memory ("ROM"), random-access memory ("RAM"), non-volatile storage, etc.) for storing software, and substantially any means and / or machine (including hardware, software, firmware, combinations thereof, etc.) capable of performing and / or controlling processing.

[0233] Furthermore, all descriptions herein relating to the principles, aspects, and embodiments of the present invention and specific examples thereof are intended to encompass both structural and functional equivalents. Moreover, such equivalents are intended to encompass both currently known equivalents (e.g., any elements developed that can perform the same or substantially similar functions regardless of their structure) and equivalents to be developed in the future. Thus, it will be understood, in consideration of the teachings provided herein, that any block diagram presented herein may represent a conceptual diagram of exemplary system components and / or circuits embodying the principles of the present invention. Similarly, it will be understood, in consideration of the teachings provided herein, that any flowchart, flow chart, etc., may be substantially represented on a computer-readable storage medium and may represent various processes performed accordingly by a computer, processor, or other device with processing capabilities, whether such computer or processor is explicitly indicated or not.

[0234] While various inventions of this disclosure and preferred and exemplary embodiments of many inventions (these embodiments are intended to be illustrative and not limiting) have been described, it should be noted that modifications and variations may be made by those skilled in the art based on the teachings provided herein, including the drawings. Therefore, modifications may be made to / from the preferred and exemplary embodiments of this disclosure, and it should be understood that such modifications are within the scope of the embodiments disclosed herein.

[0235] Furthermore, it is assumed and thought that corresponding and / or related systems incorporating and / or implementing the devices / systems, or those used / implemented in / with the devices of this disclosure, are also within the scope of this disclosure. Furthermore, corresponding and / or related methods for manufacturing and / or using the devices and / or systems of this disclosure are also assumed and thought to be within the scope of this disclosure.

Claims

1. An apparatus for an intervention device, comprising a device portion used for an intervention procedure, A memory including a forward predictive model trained using the kinematics of the intervention device, which inputs data indicating the instructed positioning motion of the intervention device for navigating the device portion to a target pose in the automatic or manual navigation of the intervention device, and outputs data related to the prediction of the navigated pose of the device portion based on the instructed positioning motion of the intervention device. A processor that communicates with the memory, wherein the at least one processor At least one processor that applies the forward prediction model to the instructed positioning motion of the intervention device to predict the navigated pose of the device portion, and generates positioning data that provides information about the positioning of the device portion by the intervention device to the target pose based on the prediction of the navigated pose of the device portion, A device equipped with the following features.

2. The memory includes an inverse prediction model trained using the kinematics of the intervention device, which inputs data indicating the target pose and outputs data related to the prediction of the placement movement of the intervention device based on the target pose. The apparatus according to claim 1, wherein the at least one processor applies the inverse prediction model to the target pose to predict the positioning motion of the intervention device, and generates a positioning instruction that controls the positioning of the device portion by the intervention device to the target pose based on the prediction of the positioning motion of the intervention device.

3. The forward prediction model is trained or will be trained based on the forward kinematics of the intervention device. The apparatus according to claim 1.

4. The inverse prediction model is trained or will be trained based on the inverse kinematics of the intervention device. The apparatus according to claim 2.

5. The intervention device comprises an intervention robot that holds the device portion, The intervention robot includes multiple links connected by multiple joints, The aforementioned forward prediction model, The neural network base includes an input layer that inputs joint variables, which are parameters indicating the joint movements of the intervention robot representing the instructed positional movement of the intervention device, and an output layer that outputs parameters indicating at least one of translational movement, rotation, and pivot of the device portion derived from the regression of the joint variables of the intervention robot, At least one of the translational movement, rotation, and pivot of the device portion represents a prediction of the navigated pose of the device portion. The apparatus according to claim 1 or 3.

6. The intervention device comprises an intervention robot that holds the device portion, The intervention robot includes multiple links connected by multiple joints, The aforementioned inverse prediction model, The neural network base includes an input layer that inputs a parameter representing at least one of the translational movement, rotation, and pivot of the device portion, and an output layer that outputs joint variables, which are parameters representing the joint movement of the intervention robot derived from the regression of at least one of the translational movement, rotation, and pivot of the device portion, The joint variables of the intervention robot represent the predicted positional motion of the intervention device. The apparatus according to claim 2 or 4.

7. The intervention device comprises an intervention robot that holds the device portion, The intervention robot includes multiple links connected by multiple joints, The aforementioned forward prediction model, The neural network base includes an input layer that inputs the joint velocity of the intervention robot representing the instructed positioning motion of the intervention device, and an output layer that outputs at least one of the linear velocity and angular velocity of the device portion from the regression of the joint velocity of the intervention robot, The prediction of the navigated pose of the device portion is derived from at least one of the linear velocity and angular velocity of the device portion. The apparatus according to claim 1 or 3.

8. The intervention device comprises an intervention robot that holds the device portion, The intervention robot includes multiple links connected by multiple joints, The aforementioned inverse prediction model, The neural network base includes an input layer that inputs at least one of the linear velocity and angular velocity of the device portion to the target pose, and an output layer that outputs the joint velocity of the intervention robot from the regression of at least one of the linear velocity and angular velocity of the device portion to the target pose, The joint velocity of the intervention robot represents the predicted positional motion of the intervention device. The apparatus according to claim 2 or 4.

9. The forward prediction model is further trained on auxiliary data which includes at least one of the following: an image of the intervention device, an image of the device portion, auxiliary shape data representing the deformation of the intervention device, auxiliary shape data representing the deformation of the device portion, and data relating to environmental conditions. The aforementioned at least one processor, To render the predicted navigated pose of the device portion, both the instructed positional motion of the intervention device and the auxiliary data are input into the forward prediction model. The apparatus according to claim 3.

10. The inverse prediction model is further trained on auxiliary data, which includes at least one of the following: an image of the intervention device, an image of the device portion, auxiliary shape data representing the deformation of the intervention device, auxiliary shape data representing the deformation of the device portion, and data relating to environmental conditions. The aforementioned at least one processor, To render the intervention device performing the predicted positional movement, both the target pose and the auxiliary data are input into the inverse prediction model. The apparatus according to claim 4.

11. The forward prediction model further inputs data including at least one image of an anatomical structure and at least one physiological signal as auxiliary information, and further processes the at least one auxiliary information to output a prediction of the navigated pose of the device portion. The aforementioned at least one processor, To render a prediction of the navigated pose of the device portion, both the instructed positional motion of the intervention device and the at least one piece of auxiliary information are input into the forward prediction model. The apparatus according to claim 1.

12. The inverse prediction model further inputs data including at least one image of an anatomical structure and at least one physiological signal as at least one auxiliary information for navigation data representing at least one navigation force, and further processes the at least one auxiliary information to output a prediction of the positional movement of the intervention device. The aforementioned at least one processor, To render the intervention device performing the predicted positional movement, both the target pose and the at least one piece of auxiliary information are input into the inverse prediction model. The apparatus according to claim 2.

13. The aforementioned device portion is the end effector of the intervention device. The apparatus according to any one of claims 1 to 12.

14. The at least one processor continuously generates the placement data and / or the placement instructions. The apparatus according to claim 2, or any one of claims 3 to 13 that is directly or indirectly dependent on claim 2.

15. A machine-readable storage medium encoded with instructions for execution by at least one processor that commands an intervention device, which includes a device portion used for intervention procedures, wherein the machine-readable storage medium is A forward predictive model trained using the kinematics of the intervention device inputs data indicating the instructed positioning motion of the intervention device for navigating the device portion to a target pose in the automatic or manual navigation of the intervention device, and outputs data related to the prediction of the navigated pose of the device portion based on the instructed positioning motion of the intervention device. Instructions for applying the forward prediction model to the instructed placement movement of the intervention device to predict the navigated pose of the device portion, and for generating placement data that provides information about the placement of the device portion by the intervention device to the target pose based on the prediction of the navigated pose of the device portion, A machine-readable memory medium that stores data.

16. The machine-readable storage medium is An inverse prediction model, trained using the kinematics of the intervention device, inputs data representing the target pose and outputs data related to the prediction of the placement movement of the intervention device based on the target pose. Instructions for applying the inverse prediction model to the target pose to predict the placement movement of the intervention device, and for generating placement instructions that control the placement of the device portion by the intervention device to the target pose based on the prediction of the placement movement of the intervention device, A machine-readable storage medium according to claim 15, which stores the following:

17. A method that can be performed by an apparatus for an intervention device, which includes a device portion used for an intervention procedure, The method includes the steps of applying a forward predictive model, trained using the kinematics of an intervention device, to the instructed positioning motion of the intervention device to predict the navigated pose of the device portion, and generating positioning data that provides information about the intervention device's positioning of the device portion to the target pose, based on the prediction of the navigated pose of the device portion, by applying the forward predictive model to the instructed positioning motion of the intervention device to predict the navigated pose of the device portion, and generating positioning data that provides information about the intervention device's positioning of the device portion to the target pose, based on the prediction of the navigated pose of the device portion. method.

18. The method includes the steps of applying an inverse prediction model, trained using the kinematics of an intervention device, which inputs data indicating the target pose and outputs data related to the prediction of the placement movement of the intervention device based on the target pose, to the target pose to predict the placement movement of the intervention device, and generating placement instructions that control the placement of the device portion by the intervention device to the target pose based on the prediction of the placement movement of the intervention device. The method according to claim 17.

Images