Method for adjusting and identifying position data of an inertial measurement unit, training system, and medical device equipped with an inertial measurement unit

Abstract

The present disclosure relates to a method for calibrating and determining at least one position and / or orientation of an inertial measurement unit (2), characterized by the steps of moving the inertial measurement unit (2) along a predetermined trajectory (T) in space by a motor drive system (4), acquiring measured motion data by the inertial measurement unit (2) during the movement and providing the motion data to a control unit (8), acquiring a position and / or orientation of the inertial measurement unit (2) during the movement by a tracking system (6), linking / correlating the measured motion data with the acquired position and / or orientation to obtain a training data set, training an AI system (10) using the training data set to obtain an IMU calibration, acquiring the measured motion data of the inertial measurement unit (2) as input to the trained AI system (10), and outputting the position and / or orientation of the inertial measurement unit (2) by the trained AI system (10) based on the input motion data. Furthermore, the present disclosure relates to a training system (1), a medical instrument (18), a computer-readable storage medium and a training data set according to the independent claims.

Description

[Technical field]

[0001] The present disclosure relates to a method for training and adjusting, and predicting or determining at least one position or position data of an inertial measurement unit. Furthermore, the present disclosure relates to a training system comprising an inertial measurement unit, a medical instrument comprising an inertial measurement unit, a computer readable storage medium, a computer program, and a training data set according to the general terms of the independent claims. [Background technology]

[0002] In endoscopic surgery, navigation inside the patient's body is a major challenge. For example, optical navigation systems with optical tracking systems have a limited field of view, and if the line of sight to the tracked object is blocked, the ability to determine its position and orientation is lost. In addition, the ergonomic suitability of the instruments used is also limited, for example due to the large tracking devices. Therefore, optical navigation systems are highly unsuitable for use in endoscopic surgery.

[0003] In the field of medical technology, efforts are underway to use inertial measurement units (IMUs) for navigation, which have the decisive advantage that they have no visual problems and can be integrated into various systems and products without significantly affecting the ergonomics of the product.

[0004] However, inertial measurement units still have significant drawbacks, so they are not currently used in surgical environments, or at least it is very difficult to use them in surgical environments. Although inertial measurement units can detect acceleration, orientation and rotation rates, they cannot provide absolute position information on a global scale. According to mathematical theory, the relative position of the IMU to its initial / starting position can be calculated from the acceleration data measured by the IMU by double integration. However, the error continues to appear and even grows over time, so calculating the position by integration from the acceleration data is very susceptible to measurement errors. A small error at the start of the measurement grows larger as the path of the IMU is traveled. It has been found that double integration of acceleration data leads to large errors in the position determination over time. Furthermore, the orientation data is also subject to sensor adjustments and other interferences. Due to these effects, it is difficult to achieve high-precision orientation measurements.

[0005] In the field of medical technology, especially cost and size are decisive factors. Large and relatively precise inertial measurement units, such as those used in aeronautics, are not suitable for use in surgical procedures. Semiconductor sensors are of particular interest as inertial measurement units because they are small and inexpensive. However, position and velocity estimation based on acceleration from inexpensive sensors is usually very prone to errors. One reason for this is that the orientation of the sensor must be very accurate to be able to distinguish between the effects of gravity and the actual physical acceleration of the sensor. Even small errors in the orientation estimation lead to large errors in the measured acceleration. These errors lead to even larger errors in the position estimation due to double integration.

[0006] Without the addition of an external sensor, accurate position estimation is often not possible. Acceleration sensors measure both physical acceleration and the normal force effect of gravity. Therefore, motion data necessarily shows the effect of gravitational acceleration. Furthermore, integration is performed using a finite time step, so a precisely determined time measurement is also required.

[0007] Since integrating IMU motion data alone is not sufficient for accurate localization, the prior art proposes supplementing the IMU motion data with additional data.

[0008] It is known from the prior art to use algorithms as filters to adjust or smooth data. For example, from current research in unrelated fields, it is known to equip a vehicle with an IMU, which acquires motion data while the vehicle is moving. At the same time, the vehicle's position (on the earth) is tracked by GPS, so that redundant detection is performed. The use of Kalman filters for IMU estimation and smoothing is also known from the prior art.

[0009] However, in the prior art, it is not known how to individually calibrate an IMU and calculate, identify or predict the exact position of the IMU in space. Summary of the Invention [Problem to be solved by the invention]

[0010] The object of the present disclosure is to avoid or at least reduce the drawbacks of the prior art, and in particular to provide a method, a training system, a medical instrument, a computer readable storage medium, a computer program, and a training data set for determining as accurately as possible at least one position of an IMU in space based on measured motion data. Another object is to provide a link between the measured motion data, in particular the measured accelerations and the measured orientation, and at least the position data of the IMU, without specifying mathematical formulas. [Means for solving the problem]

[0011] These objects of the present disclosure are achieved with respect to the method by the features of claim 1 according to the present disclosure, with respect to the training system by the features of claim 9 according to the present disclosure, with respect to the generic medical instrument by the features of claim 11 according to the present disclosure, with respect to the computer readable storage medium by the features of claim 14 according to the present disclosure and with respect to the training data set by the features of claim 15 according to the present disclosure.

[0012] Thus, the basic idea of ​​the present disclosure is to establish a link between the measured (raw) motion data of an Inertial Measurement Unit (IMU), in particular the measured acceleration data and the measured rotational speed or the measured direction, and at least the position data, by means of an AI system (artificial intelligence system). For this purpose, the AI ​​system is appropriately trained. The trained AI system can finally act as a kind of converter between input and output, converting the input measured motion data into an output (at least) one precise position specification or precise position data of the IMU, thereby at least predicting the position. This achieves a kind of tuning of the IMU that is customizable and does not require special predefined formulas.

[0013] Thus, the object of the disclosure is achieved as follows, at least with respect to a method for adjusting and predicting position data of an Inertial Measurement Unit (IMU). Specifically, the method comprises the following steps: The IMU is actively moved in space along a trajectory, in particular along a predefined trajectory, by a motor drive system or an actuator system. That is to say, at least the position, in particular the position, orientation and preferably also the acceleration of the IMU changes along the trajectory. In particular, the trajectory may be determined at the start of the movement to follow a kind of target trajectory. During the movement, measured movement data is acquired by the IMU, which movement data is provided to a control unit. Thus, the IMU is used to read out internally measured (raw) movement data along the trajectory and provide it to the control unit. Furthermore, a tracking system acquires the position and / or orientation, in particular the position and orientation (i.e. location), of the IMU while it is moved along the trajectory. In a further step of the method, preferably, the measured movement data is linked or associated with the acquired position, in particular the position and orientation, in order to obtain a training data set. An AI system (artificial intelligence system) is trained with the training data set to obtain a trained AI system and thus the IMU adjustment. That is, after only one input, the trained AI system is ready for application. Finally, the method involves obtaining measured motion data of the IMU as input to the trained AI system, and based on the input motion data, the position and / or orientation, in particular the location, of the IMU is output by the trained AI system. In particular, the position and / or orientation, in particular the location, of the IMU may be provided to a navigation system, which may output the position and / or orientation, for example by an output device.

[0014] The IMU is fixed or attached to the motor drive system. The motor drive system moves in space along a trajectory, in particular on a predefined / predetermined trajectory. The IMU is therefore attached to the motor drive system and moved in space along a trajectory together with the motor drive system, changing its position, in particular its position and orientation. The IMU acquires motion data measured while the motor drive system is moving. The motion data is preferably in the form of acceleration in three directions (a_x, a_y, a_z) and orientation of the IMU around three directions, in particular in the form of rotational speed or direction around three axes (e.g. X-axis, Y-axis, Z-axis). In particular, six values ​​are acquired as motion data: three acceleration values ​​and three rotational speed values. Preferably, a magnetic field may further be detected by the IMU.

[0015] While the IMU is moved along a trajectory by the motor drive system, the tracking system also acquires the position and / or orientation of the IMU in space. The tracking system provides in a certain way the absolute position and / or orientation (also called position data) in the acquired space. The tracking system is preferably an optical tracking system or a mechanical kinematic tracking system. By acquiring the measured motion data and position data, two time-dependent data sets are available. The motion data and position data are related such that the corresponding position and / or orientation is linked to the motion data at a specific time or time period. This creates a linked (training) data set in which the motion data and the corresponding position data are linked to each other.

[0016] The linked data set is used to train or teach the AI ​​system. The linked data set is thus a training data set for the AI ​​system. The AI ​​system is preferably an artificial neural network. The motion data is the input to the AI ​​system and the position and / or orientation is the output of the AI ​​system. By training the AI ​​system, the individual weights of the neural network are adjusted such that an association or correlation between the motion data and the position data is determined without the need to explicitly state it. This completes the training phase of the AI ​​system. The trained AI system essentially performs IMU calibration, i.e. the acquired and measured raw motion data is to some extent individually corrected by the trained AI system and linked to the corresponding position data. For example, the effects of gravity and other disturbances can be removed from the raw data. In a subsequent step, the trained AI system is used to make a prediction of the IMU position and / or orientation in space relative to the measured IMU motion data. Thus, the method or the trained AI system can very accurately determine the IMU position and / or orientation based on the measured IMU motion data. In particular, a prediction of the IMU's position data is output for one or more new trajectories not yet acquired by the tracking system.

[0017] In summary, the essence of this disclosure is to obtain measured motion data for each trajectory using an IMU while simultaneously obtaining position and / or orientation by a tracking system. The obtained data is used to train an AI system. Based on the measured motion data along an unknown trajectory, the trained AI system can predict / output position and / or orientation within this trajectory without the need for IMU position tracking.

[0018] The method according to the present disclosure has the following advantages: As explained above, calculating the position and / or orientation from the acceleration data by double integration leads to large calculation errors; The effect of gravitational acceleration leads to large noise and interference in the acquired acceleration data. Each of these problems is solved by the method according to the present disclosure and the trained AI system. The trained AI system can predict the position and / or orientation of the IMU as accurately as possible from the acquired IMU motion data. The learning process and training of the AI ​​system can remove / eliminate disturbances and systematic adjustment errors caused by the effect of gravitational acceleration. Furthermore, the AI ​​system can be tailored to each individual IMU. Furthermore, the trained AI system can be provided for a specific class or a large number of IMUs. By eliminating systematic measurement errors and accurately predicting the position and orientation of the IMU, sufficient accuracy can be obtained even when small and inexpensive sensors are used. As a result, costs can be reduced and the installation space on the medical instrument equipped with the IMU can be optimally utilized. In addition, there is an advantage that sufficient training data for the AI ​​system can be generated easily, efficiently, and quickly.

[0019] The problem of the present disclosure is also solved by a training system for adjusting (and predicting) at least position data of an inertial measurement unit (IMU). For this purpose, the training system comprises an IMU to be adjusted, which acquires measured movement data and provides it to a control unit. Furthermore, the training system comprises a motor drive system adapted to move the IMU to be adjusted along a trajectory in space, in particular along a predefined trajectory. The training system also comprises a tracking system adapted to acquire a position and / or an orientation of the IMU and provide it to the control unit. The control unit is adapted to link the measured movement data with the acquired position and / or orientation to obtain a training data set. Finally, the training system comprises an AI system for training the control unit with the training data set to obtain a trained AI system for the IMU adjustment.

[0020] Thus, the IMU is moved along a trajectory by a motor drive system. The motion data measured while being moved are acquired by the IMU. In a controlled environment, the position and / or orientation of the IMU while being moved is also acquired very accurately by a tracking system. These acquired data are linked by a control unit to generate a linked training data set. An AI system is trained with the generated training data set. The AI ​​system is trained by a training system according to the present disclosure so as to adjust the IMU motion data and provide the best possible prediction of the IMU position and / or orientation based on the acquired motion data. The training system may thus be used to train an AI system dedicated to an individual IMU or to a number of (near) identical IMUs. This trained AI system may in particular be stored in a computer-readable storage medium and then be used with the IMU for applications in the medical field, in particular for surgical navigation.

[0021] Furthermore, according to the present disclosure, the problem is solved by a medical instrument, in particular a surgical instrument. The medical instrument comprises an inertial measurement unit (IMU) and a trained AI system, which is trained / adapted by a training system, in particular a training system according to the present disclosure, and the measured motion data of the IMU is used as input to the trained AI system, and based on the input motion data, the position and / or orientation of the IMU is obtained by the trained AI system. In order to determine the position and / or orientation as directly and immediately as possible, the IMU is fixed, in particular as a module, to the medical instrument, preferably to the distal tip of the medical instrument that is inserted into the body cavity during the operation. When the medical instrument is used and moved during the operation, the IMU obtains the motion data. The IMU together with the medical instrument is moved along a new trajectory, and the trained AI system can use the measured motion data of the IMU as input to determine and output the position and / or orientation of the IMU. In this way, the user can always know where and / or in what direction the IMU, and thus the medical instrument (in particular the distal tip of the medical instrument), is located in space on a global scale.

[0022] Furthermore, the problem of the present disclosure is solved by a computer-readable storage medium and a computer program, each of which comprises instructions that, when executed by a computer, cause the computer to perform the steps of the method according to the present disclosure.

[0023] Furthermore, the problem of the present disclosure is solved by a training dataset for training, i.e., tuning (and predicting), an IMU, the training dataset including motion data of the IMU measured along a trajectory with a linkage of a position and / or orientation of the MIU obtained along the trajectory. Thus, the training dataset has the measured motion data as input and the position and / or orientation as output. The training dataset may be used for training an AI system. In particular, a computer-readable storage medium may include the training dataset according to the present embodiment. In particular, the training dataset may be acquired by a training system according to the present disclosure.

[0024] By "location" we mean a geometric location in three-dimensional space, in particular one that is specified by coordinates in the Cartesian coordinate system. In particular, a location can be specified by the three coordinates X, Y, Z.

[0025] A "direction" refers to a direction in space (approximately at that location). A direction can also be said to be a direction that specifies a bearing or rotation in three-dimensional space. In particular, a direction can be specified by three angles.

[0026] "Location" includes both position and orientation. In particular, a position can be specified using six coordinates: three position coordinates X, Y, Z, and three angular coordinates for orientation.

[0027] Position or orientation herein is considered in particular with respect to a coordinate system, preferably with respect to a defined coordinate system. For example, the tracking system may acquire the position and orientation of the inertial measurement unit at start, then define this start data as the (start) coordinate system (i.e. position zero, orientation zero), and then (continuously) track the position and orientation in this defined COS as the IMU moves. Alternatively, for example, the local COS of the tracking system may also be defined as a COS, and at start, the start position and start orientation of the IMU may be specified, and the position and orientation may be tracked as the IMU moves relative to the start position and start orientation.

[0028] As used herein, "rotation rate," as it is generally used in the art with respect to inertial measurement units, refers to the measurable rotational speed or three angular velocities. In particular, based on a starting orientation (e.g., of an IMU) and a measured rotation rate (e.g., continuously over time), a direction at another time can be inferred, such that rotation rate and direction can be linked together via a functional relationship.

[0029] Further advantageous embodiments of the present disclosure are the subject matter of the subclaims and are described in detail below.

[0030] In the step of acquiring the measured motion data, at least the acceleration and the direction or rotation speed are preferably acquired. Furthermore, it is preferable that the direction relative to the magnetic field can also be acquired. The acceleration is acquired by one or more acceleration sensors. The IMU preferably has three orthogonal acceleration sensors to acquire three accelerations in three different directions (a_x, a_y, a_z). The direction and rotation speed around each axis, specifically around the X-axis, Y-axis, and Z-axis, are acquired by a gyroscope, specifically by three orthogonal gyro sensors. Thus, the IMU preferably has six sensors, three acceleration sensors and three gyro sensors. The IMU may have only one acceleration sensor and one direction sensor, such as a gyro sensor. The IMU only needs to be able to acquire acceleration in three directions and acquire directions in or around three directions. By acquiring this motion data, the movement of the IMU can be sufficiently captured. The acquired motion data is part of the training data set.

[0031] Between the step of training the AI ​​system with the training data set and the step of acquiring the measured motion data from the IMU as input to the trained AI system, the IMU is preferably attached to the medical instrument. That is, the IMU is removed from the motorized drive system and transferred to the medical instrument. Similarly, the trained AI system is transferred to the medical instrument or at least a data connection is established with the medical instrument so that the trained AI system can be accessed by the medical instrument. When the medical instrument is moved, the IMU acquires motion data of this movement. Although it was possible to acquire the position and / or the IMU while it is being moved by the motorized drive system, it is no longer desirable to acquire the position and / or the orientation of the IMU by the tracking system while the medical instrument is being moved, in order to prevent visibility issues. Thus, the trained AI system is used to calculate / estimate / predict the position and / or orientation of the IMU from the acquired motion data of the IMU.

[0032] In the step of acquiring the position and / or orientation of the IMU by the tracking system, the position and / or orientation of the IMU is preferably acquired by an optical tracking system and / or a machine kinematics-based tracking system. The optical tracking system acquires the position and / or orientation of the IMU using one or more cameras. The machine kinematics-based tracking system acquires the position and / or orientation of the IMU by sensors and / or actuators of a motorized system, preferably servo motors, for example, in a robotic system, by joint angle sensors between the robot arm segments using a kinematic model of the robot. The (global) position and / or orientation of the IMU is acquired with sufficient accuracy by the tracking system. In particular, this step is performed in a controlled environment where there is always good visual contact between the cameras of the optical tracking system and the tracked IMU, and where other rigid trackers (such as IR markers) are attached to enable, for example, accurate measurements. Once the AI ​​system has been trained, these trackers are no longer necessary, allowing the IMU to be used in challenging environments.

[0033] In the linking step, the measured movement data is preferably linked at a first time point, in particular at many discrete time points, preferably at a predefined time interval, to the position and / or orientation acquired at the first time point. Both the acquired movement data and the acquired position and / or orientation are time-dependent. Due to this time-dependency, the position and / or orientation can also be linked to the movement at any time point. This results in a linked training data set. In particular, the first position and orientation, i.e. the position at the first time point, the second position, and the measured acceleration and rotational speed of the IMU at the first time point can be linked to the data set. The AI ​​system then finds the link on its own through so-called training in order to infer the second position from the first position based on the measured movement data and to mimic the tracking system.

[0034] The moving step preferably includes moving the IMU by a robot and the acquiring step preferably includes acquiring by a robot kinematic tracking system. If the IMU is mounted on a movable robot arm, the position and / or orientation of the IMU may be acquired by sensors and / or actuators on the robot itself. This avoids the need for complex and expensive cameras as used in optical tracking systems and also avoids line of sight issues.

[0035] The steps of acquiring the measured movement data, acquiring the IMU position and / or orientation and linking them are preferably performed at different times: values ​​are acquired at specific time intervals, which makes linking the acquired data easier, since although in theory continuous measurements are possible, in practice discrete time measurements are always used even for the corresponding determinations.

[0036] The measured motion data includes acceleration values ​​in three directions and three orientation values ​​around three axes, and the AI ​​system preferably has at least six nodes in the input layer. The six values ​​are obtained when recording the measured motion data of the IMU: acceleration in three directions and orientation in three directions. Since the six values ​​are input to the AI ​​system, the AI ​​system preferably has six nodes in the input layer. This means that the motion data can be directly input to the AI ​​system without prior transformation.

[0037] As AI system, preferably an artificial neural network is used, in particular a recurrent deep neural network, particularly preferably a long short-term memory network (LSTM network) and / or a recurrent convolutional network. This allows, in particular, the depth of the neural network to be adapted and training to be carried out until a sufficiently accurate weighting is obtained. The acquired data is in any case time-dependent. The aforementioned artificial neural network is particularly suitable to be trained with time-dependent data.

[0038] Preferably, the step of moving along at least one trajectory, here a predefined trajectory, preferably at least two predefined trajectories, has a varying speed and / or a varying acceleration and / or a varying direction, wherein the trajectory or trajectories are moved with different speeds, accelerations and / or directions in order to collect sufficient training data.

[0039] In the output step, in addition to the position and / or orientation, the (corrected) velocity and / or the corrected acceleration of the IMU are preferably output. The trained AI system basically performs IMU adjustments to determine and output at least the position and / or orientation, as well as the corrected acceleration (approximate to the real acceleration) and the (corrected) velocity from the measured motion data. The raw motion values ​​are corrected by adjusting or filtering the acquired raw motion values. In particular, the trained AI system outputs corrected motion data that is adjusted for the effects of gravitational acceleration and the sensor adjustments. The corrected motion values ​​may also be output by the AI ​​system. This provides the control unit with correction values ​​that can be used continuously if necessary. The velocity of the IMU is output by the trained AI system without errors that may occur when integrated from the raw acceleration data. In particular, the AI ​​system may be trained not only on the relationship between the motion data (including acceleration and rotational velocity) and the position and / or orientation, but also on the relationship between the motion data (including acceleration and rotational velocity or rotational velocity) and the acceleration and orientation or rotational velocity.

[0040] The output position and / or direction preferably has three position coordinates and / or three direction or angle coordinates. To know the exact position of the IMU in space, at least three position coordinates need to be output by the AI ​​system. The same goes for the direction. To output these values, the AI ​​system should have at least three nodes in the output layer. This allows the position and / or direction to be output directly. There is no need to convert the values. Of course, it is also conceivable that the AI ​​system has six nodes in the output layer. In this case, the AI ​​system may output three directions and three positions.

[0041] Preferably, the movement is performed along a second trajectory (different from the first trajectory). In order to collect as much training data as possible, preferably there are multiple trajectories and the position and / or orientation within the multiple trajectories are acquired by the tracking system.

[0042] The motor drive system may preferably be a robot, in particular a robotic arm, that guides the IMU. The IMU may be moved along a trajectory by the robot.

[0043] The tracking system preferably comprises or may be an optical tracking system or a kinematics-based tracking system.

[0044] The IMU preferably obtains measured movement data, at least acceleration and direction, and preferably orientation relative to a magnetic field.

[0045] The IMU is preferably affixed to the motor drive system and moved along the track together with the motor drive system, the IMU acquiring motion data while being moved.

[0046] The measured movement data, preferably acceleration values ​​in three directions and three directional values ​​around three axes, are obtained by an IMU, and the AI ​​system preferably has at least six nodes in the input layer.

[0047] The measured movement data is linked by the control unit to positions and / or orientations obtained at a first time point, whereby a linked training data set is generated.

[0048] The AI ​​system of the training system is preferably an artificial neural network, in particular a recurrent deep neural network, particularly preferably a long short-term memory network (LSTM network) and / or a recurrent convolutional network, which are particularly suitable for processing time-dependent data.

[0049] The AI ​​system preferably has three or six nodes in the output layer.

[0050] The medical instrument is preferably an endoscope, preferably a rigid endoscope, with an IMU, particularly at the handle and / or distal tip. The IMU captures motion data while the medical instrument is moved. The captured motion data is input to the trained AI system. When the endoscope is inserted into a patient's body cavity and moved, the IMU captures motion data of its movement.

[0051] The medical device is preferably a gait analysis system comprising at least one IMU, in particular installed on the knee and / or hip and / or foot of the subject. The IMU is fixed to the gait analysis system so as to be able to acquire the patient's movements during the intervention (e.g. being actively moved by the physician) and the subject's movements while walking. During walking, the IMU acquires movement data.

[0052] The medical instrument is preferably a mobile surgical robot having an IMU at its distal tip, specifically at the distal tip of a medical end effector.

[0053] All disclosures relating to methods according to the present disclosure apply equally to the training systems and medical devices, and all disclosures relating to the training systems and medical devices of the present disclosure apply equally to the methods of the present disclosure.

[0054] The present disclosure will now be described in more detail with reference to preferred embodiments using the figures. [Brief description of the drawings]

[0055] [Figure 1] FIG. 1 is a schematic front view of a training system according to a preferred embodiment of the present disclosure, in which an IMU mounted at a distal tip is moved along a trajectory from a first position to a second position. [Diagram 2] FIG. 1 is a schematic diagram of a further preferred embodiment training system of the present disclosure, comprising an optical tracking system and an actively moving IMU. [Diagram 3] 1 is a schematic perspective view of a medical instrument according to a preferred embodiment of the present disclosure in the form of a rigid endoscope; FIG. [Figure 4] FIG. 2 is a flow diagram of a method according to a preferred embodiment of the present disclosure. [Figure 5a] 1 shows an example time-dependent curve of input values ​​to an AI system. [Figure 5b] 5b shows an exemplary time-dependent curve of output values ​​of an AI system based on input values ​​of FIG. 5a. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0056] The figures are schematic and are merely for the purpose of understanding the disclosure. Identical elements are provided with identical reference numbers. Features of the various embodiments are interchangeable.

[0057] FIG. 1 shows a schematic diagram of a training system 1 according to a preferred embodiment of the present disclosure. The training system 1 comprises an inertial measurement unit (IMU, hereafter referred to only as IMU) 2, for which IMU calibration and binding of motion data to position and orientation are performed. For this purpose, the training system 1 comprises a motor drive system 4 in the form of a robot adapted to actively move the IMU 2. The IMU 2 is a set of sensors for acquiring motion data, preferably acceleration and orientation. In this embodiment, the IMU 2 comprises an acceleration sensor and a gyro sensor (not shown) and acquires as measured motion data the acceleration (in three dimensions) (in all spatial directions) and the rotational speed around the three axes, i.e. the orientation of the IMU 2 in space.

[0058] Specifically, the IMU 2 is fixed to the distal tip of the motor drive system 4. The motor drive system 4 moves the IMU 2 along a predetermined trajectory T in space. While being moved by the motor drive system 4, the IMU 2 acquires motion data corresponding to the predetermined trajectory T. In FIG. 1, for ease of understanding, two different positions of the robot at two different times are shown with different hatchings.

[0059] Furthermore, the training system 1 of Fig. 1 comprises a tracking system 6 in the form of a machine kinematics based tracking system 6. The motorized drive system 2 comprises actuators, in this embodiment servo motors, which set the position of the IMU very precisely and may be read out with suitable knowledge of the kinematic model. In particular, the motorized drive system 4 is a robot having a movable robot arm connected to a fixed robot base 12, the robot arm having a number of robot arm segments 14 movable relative to each other and between which joint angles can be set. Thus, the position and orientation, i.e. location, of the IMU 2 can be measured by the robot's internal tracking system 6.

[0060] The training system further comprises a control unit 8 and an AI system 10. Both the measured movement data and the position and orientation acquired by the tracking system 6 are provided to the control unit 8. The control unit 8 links the movement data acquired by the IMU 2 with the position and / or orientation of the IMU 2 acquired by the tracking system 6 to generate a linked training data set. With this training data set the AI ​​system 10 is trained to obtain the IMU alignment of the movement data with respect to position and orientation. In this embodiment, the AI ​​system 10 of the training system 1 is an artificial neural network, in particular a recurrent deep neural network, in particular a long-short-term memory network and / or a recurrent convolutional network. This recurrent deep neural network is particularly suitable for time-dependent data such as the data acquired by the IMU in this embodiment.

[0061] Thus, the training system 1 trains the AI ​​system by controlling the robot such that the control unit moves the IMU in various translational motions, various orientations and rotations or various directions, with varying accelerations (variable velocities) along multiple (particularly predetermined) trajectories. By varying the accelerations, directions and orientations, a highly detailed training data set can be generated that can be used to train the AI ​​system. In particular, the IMU 2 is detachably fixed to the distal tip of the motor drive system 2 and moved along one or more trajectories until a sufficiently detailed training data set is created for the AI ​​system to be trained, and then the trained AI system is stored in the IMU, specifically in a computer-readable storage medium, to establish a specific IMU adjustment for each IMU 2. The IMU 2 is finally removed from the training system 1 and inserted into a medical instrument (see FIG. 3).

[0062] In this way, multiple IMUs 2 can be individually calibrated one after the other, and these IMUs 2, together with their associated AI system 10, can very accurately associate their individual motion data with position and orientation, which can be used in surgical applications.

[0063] Thus, the training system 1 may be used to calibrate the IMU and associate the motion data with position and orientation, regardless of the subsequent operating environment. Once the AI ​​system has been trained on the IMU 2, the IMU 2 may be used with the AI ​​system 10. Thus, a two-stage system is provided: in a first stage, the training system 1 trains the AI ​​system, i.e. calibrates the IMU, and in another stage, stage 2, the IMU is used with a position and orientation determined with sufficient accuracy based on the measured motion data to allow navigation without an external (global) tracking system. The trained AI system 10 thus mimics a tracking system, thereby allowing accurate navigation without an external tracking system.

[0064] In particular, the trained AI system 10 and IMU 2 may of course be integrated into a navigation system that includes a tracking system, and in addition to the position and orientation being determined by the tracking system, the position and orientation may also be determined by the trained AI system 10 and IMU 2. That is, for example, in the case of an optical tracking system, when line of sight is interrupted, the trained AI system 10 and IMU 2 may determine the position and orientation, and when line of sight is returned, the optical tracking system may resume the determination.

[0065] Fig. 2 shows a schematic diagram of a training system 1 according to a further second preferred embodiment of the present disclosure. In contrast to the training system 1 of Fig. 1, the motor drive system 4 and the tracking system 6, the training system 1 does not have a robot but a general motor drive system (not shown here) and an optical tracking system 6. This optical tracking system 6 comprises a number of cameras 16 arranged at intervals to enable three-dimensional spatial detection, in particular by triangulation, by image analysis of the tracking system 6, in particular by a control unit 8. The cameras 16 are arranged to track the IMU 2 along a trajectory T in space and to obtain (by determination by the tracking system 6) the position and orientation of the IMU 2 in space. This position and orientation obtained while being moved are provided to the control unit 8.

[0066] The control unit 8 links the measured movement data acquired by the IMU 2 with the position and / or orientation of the IMU 2 acquired by the tracking system 6 to generate a linked training data set. The AI ​​system 10 is trained with the training data set. In this embodiment too, the AI ​​system 10 is an artificial neural network in the form of a recurrent deep neural network, in particular a long-short-term memory network and / or a recurrent convolutional network. The AI ​​system 10 to be trained has six input nodes for inputting the three acceleration values ​​measured by the IMU 2 and the three rotational or angular velocities measured. Furthermore, the AI ​​system has six output nodes, three of which have three position coordinates and three of which have three orientation coordinates (in this example, angular coordinates). In this embodiment, the control unit 8 and the AI ​​system 10 to be trained are connected to each other via a wireless data connection, such as WLAN or Bluetooth, so that the IMU 2 and the AI ​​system 10 form a so-called unified module, which can later be separated and inserted into the medical instrument.

[0067] 3 is a perspective view of a medical instrument 18 according to a preferred embodiment of the present disclosure. In this embodiment, the medical instrument 18 is configured as a rigid endoscope with an IMU 2 at its distal tip, a control unit 8 and a trained AI system 10 in the handle.

[0068] When a medical instrument 18 in the form of an endoscope is moved inside a patient (not shown), the (global) position of the distal tip 20 cannot be obtained by the optical tracking system of the navigation system. The position of the medical instrument is therefore determined using an IMU 2 fixed to the distal tip of the medical instrument 18. The position of the IMU 2 is determined based on measured motion data of the IMU 2. The IMU 2 acquires the motion data while the medical instrument 18 is moved. The acquired motion data is provided to a control unit 8, which uses it as input to an AI system 10 (trained on the IMU 2). The trained AI system 10 outputs a determination / prediction or a highly accurate estimation of the position and / or orientation of the IMU 2, i.e. the distal tip 20 of the medical instrument 18, based on the acquired motion data. In particular, the medical instrument may be equipped with an IMU 2 having an associated AI system 10 trained by the training system 1 of the present disclosure.

[0069] In Figure 3, the medical instrument 18 is configured as an endoscope. However, it is also contemplated that the medical instrument 18 may be configured as a system for gait analysis or as a mobile surgical robot with an end effector (neither of which are shown).

[0070] FIG. 4 is a flow diagram of a method according to a preferred embodiment of the present disclosure.

[0071] In a first step S1, the IMU 2 is moved along a predetermined trajectory T by the motor drive system 4.

[0072] In step S2, motion data (in this example, acceleration and direction) is acquired by the IMU 2 while the IMU 2 is being moved.

[0073] At the same time, in step 3, the position and orientation of the IMU 2 is acquired by the (external) tracking system 6. The acquired IMU 2 motion data and the acquired position and orientation of the IMU 2 are provided to the control unit 8.

[0074] In step S4, the control unit 8 links the acquired movement data with the acquired positions and / or orientations. The movement data is time-bound to each position and orientation. This generates a training data set.

[0075] In a further step S5, the AI ​​system 10 is trained using the generated training data set. The trained AI system 10 essentially performs IMU calibration on the acquired motion data to obtain a correlation between the motion data and position and orientation. Step S5 provisionally or finally completes the training of the AI ​​system.

[0076] In particular, in an intermediate step, the IMU 2 is attached to a medical instrument 18, such as a rigid endoscope (see, for example, FIG. 3 ). Also, a data connection between the medical instrument or a control unit and the trained AI system is established. Preferably, the trained system is integrated into the medical instrument.

[0077] In a further step S6, the IMU 2 acquires motion data, preferably along a trajectory other than the (in particular predetermined) trajectory T followed during training. The acquired motion data becomes input to the trained AI system 10. In a final step S7, the trained AI system 10 outputs a position and orientation of the IMU 2 based on the acquired motion data. The trained AI system 10 mimics a tracking system.

[0078] 5a and 5b are for further illustrating the correlation between the input values ​​and the output values ​​of the AI ​​system 10. FIG. 5a shows the time curve of the measurement variable of the IMU2. In practice, three directions and three accelerations are obtained as motion data by the IMU2 and used as inputs to the AI ​​system 10. Based on these inputs, the trained AI system 10 outputs values. An example of such an output value curve is shown in FIG. 5b. For ease of explanation, only one output value is illustrated. In this embodiment, for example, FIG. 5a may show the acceleration in the X direction (a_x), and FIG. 5b may show the position coordinate in the X direction (r_x). Alternatively, FIG. 5a may show the direction around the axis, and FIG. 5b may show the corrected direction around the axis. In practice, three positions or three directions are output. The input value and output value curve are time-dependent. While the IMU2 is being moved, new data is input to the AI ​​system 10 in real time, and based thereon, the corresponding value is output by the trained AI system. Since the two acquired data sets are time dependent, it is possible to associate a position with a motion data set at a specific point in time.

[0079] Thus, the present disclosure allows for the use of (any) inertial measurement unit, which is calibrated prior to actual use through training of an AI system, the trained AI system being some sort of replacement or analogue of a tracking system, capable of determining at least one position and / or orientation with sufficient accuracy based on the internal measurements (motion data) of the IMU, in particular to pre-empt global errors and deviations of the IMU2 due, for example, to manufacturing. [Explanation of symbols]

[0080] 1: Training system 2,2´: Inertial Measurement Unit (IMU) 4: Motor drive system 6: Tracking system 8: Control unit 10: AI system 12: Robot Base 14: Limbs / Robot Arm Segments 16: Camera 18: Medical equipment 20: Distal tip T:Trajectory S1: Step to move the IMU along the trajectory S2: Step of acquiring measured movement data S3: Obtaining position and orientation by a tracking system S4: Linking step S5: Training the AI ​​system using the training data set. S6: Acquiring movement data as input to the AI ​​system S7: Step of identifying and outputting the position and direction by the AI ​​system

Claims

1. A method for adjusting and determining the position and / or orientation of at least one inertial measurement unit of a medical instrument, comprising: moving the inertial measurement unit in space by a motor drive system along a predetermined trajectory, the trajectory having a varying velocity and a varying acceleration; acquiring measured motion data by the inertial measurement unit while in motion and providing the motion data to a control unit; acquiring the position and / or orientation of the inertial measurement unit while in motion by an optical tracking system including one or more cameras; linking the measured movement data with the acquired positions and / or orientations to obtain a training data set; training an AI system of the medical instrument using the training data set to obtain IMU adjustments of motion data to position and / or orientation; obtaining measured motion data of the inertial measurement unit as input to the AI ​​system of the trained medical device; and determining and outputting the position and / or orientation of the inertial measurement unit by the trained AI system based on the input motion data.

2. 2. The method according to claim 1, characterized in that in the step of acquiring the measured movement data, at least one acceleration and one direction, in particular at least one acceleration and at least one rotation rate, preferably also a direction relative to a magnetic field, are acquired.

3. 2. The method according to claim 1, characterized in that in the linking step, the measured movement data, in particular the acceleration and rotational speed, are linked to the position and / or orientation obtained at a first point in time at the first point in time.

4. 10. The method of claim 1, wherein the moving step comprises moving the inertial measurement unit by a robot.

5. the measured movement data includes acceleration values ​​in three directions and three rotational velocity values ​​or three direction values ​​around three axes; 10. The method of claim 1, wherein the AI ​​system comprises at least six nodes in an input layer.

6. 2. The method according to claim 1, characterized in that an artificial neural network, in particular a recurrent deep neural network, particularly preferably a long-short-term memory network and / or a recurrent convolutional network, is used as the AI ​​system.

7. 2. The method of claim 1, wherein the moving step is performed along predetermined trajectories, preferably at least two predetermined trajectories, with varying speed and / or varying acceleration and / or varying rotational speed or direction.

8. 2. The method of claim 1, wherein the determining and outputting step outputs a corrected acceleration and / or a corrected velocity of the inertial measurement unit in addition to the position and / or orientation.

9. 1. A training system for adjusting and identifying at least one position and / or orientation of an inertial measurement unit, comprising: an inertial measurement unit to be calibrated, which acquires the measured motion data and provides it to the control unit; a motor drive system adapted to move the inertial measurement unit to be calibrated along a predetermined trajectory in space, the trajectory having a varying velocity and a varying acceleration; an optical tracking system adapted to acquire and provide the position and / or orientation of the inertial measurement unit to the control unit, the optical tracking system including one or more cameras; a control unit for linking the measured movement data with the acquired positions and / or orientations to obtain a training data set; an AI system that is trained by the control unit using the training data set to obtain a trained AI system having IMU adjustments of position and / or orientation movement data.

10. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method steps of the method of any one of claims 1 to 8.

Images