Endoscopic robotic camera system

Abstract

The invention provides an endoscopic robotic camera system as well as a method for operating a robotic camera system. The endoscopic robotic camera system (1000) comprises: a single-jointed or multi-jointed robotic arm (110); an endoscopic camera (120) mounted on the robotic arm (110); and a control unit (100) configured to: receive (S10) a command (71) for the robotic arm (110) indicating a trajectory to be followed by the robotic arm (110); obtain (S20) at least one mechanical movement parameter of the robotic arm (110) for the trajectory according to the command (71); determine (S30), based on the obtained at least one mechanical movement parameter, whether a mechanical power threshold value, P max , would be crossed by the trajectory according to the command (71); control (S40), in case the mechanical power threshold value, P max , would be crossed, the robotic arm (110) to move according to a modified trajectory in which the mechanical power threshold value, P max , would not be crossed; and control (S50), in case the mechanical power threshold value, P max , would not be crossed, the robotic arm (110) to move according to the command (71).

Description

[0001] Endoscopic robotic camera system

[0002] The present invention relates to an endoscopic robotic camera system and to a method for operating a camera system in a damageable environment.

[0003] Operating robotic camera systems in damageable environments is a challenging task. The task often comprises moving a camera using a multi-jointed robotic arm while following with its viewing field another robotic arm equipped with an instrument of some kind so that the operator of the instrument may have a clear view of the instrument’s use. Meanwhile, both the instrument’s operator and the camera system’s operator are independently trying to maintain the integrity of the environment. It is not uncommon that the camera, in order to continue to provide a suitable view of the instrument, has to move in a direction not shown on the image of the robotic camera system itself, which makes it more difficult to be aware of the boundaries and vulnerabilities of the damageable environment in that direction.

[0004] Known solutions for assisting in this task comprise providing the robotic camera system with a large number of sensors. However, in many application, such as in endoscopic medical applications (e.g., laparoscopy), any robotic system is desired to be as small and as agile as possible so as to cause the least disruption to the damageable environment (e.g., a patient). The provision of a large number of sensors is therefore not always feasible.

[0005] It is therefore an objective of the present invention to provide a robotic camera system, and a method for operating a robotic camera system, that are better designed for maintaining the integrity of a damageable environment.

[0006] According to a first aspect of the present invention, an endoscopic robotic camera system is provided, comprising: a single-jointed or multi-jointed robotic arm; an endoscopic camera mounted on the robotic arm; and a control unit configured to: receive a command for the robotic arm indicating a trajectory to be followed by the robotic arm; obtain at least one mechanical movement parameter of the robotic arm for the trajectory according to the command; determine, based on the obtained at least one mechanical movement parameter, whether a mechanical power threshold value, Pmax, would be crossed (or: exceeded) by the trajectory according to the command in case of a collision with a resting object; control, in case the mechanical power threshold value would be crossed (or: exceeded), the robotic arm to move according to a modified trajectory in which the mechanical power threshold value, Pmax, would not be crossed; and control, in case the mechanical power threshold value, Pmax, would not be crossed (or: exceeded), the robotic arm to move according to the command (or: according to the unmodified trajectory).

[0007] One basic idea of the invention is that the main task is not to ensure that the robotic camera system never touches the damageable environment surrounding it, but to ensure that the robotic camera system never damages the damageable environment in case of a collision. The robotic camera system of the present invention is designed with this task in mind. In the prior art, there is typically a large sensor suite, and a corresponding powerful computing device, employed for ensuring that there is absolutely no collision with the damageable environment, even if in many (or even all) of the possible circumstances a collision would not actually cause any damages.

[0008] The present invention, by contrast, provides a comparatively simple, sleek, and cost-efficient solution.

[0009] The endoscopic camera may be mounted specifically on (or as) an end effector of the robotic arm, although in some variants it may be positioned at other sections of the robotic arm.

[0010] A mechanical movement parameter may be understood to be any parameter that is used in the field of Applied Mechanics for describing the properties of a moving body, for example, any of:

[0011] - a torque value or vector;

[0012] - a speed value or velocity vector;

[0013] - an acceleration value or vector;

[0014] - a rotational speed value or rotational velocity vector;

[0015] - a rotational acceleration value or vector;

[0016] - a force value or vector;

[0017] - and / or the like. In each case or in any of the cases, obtaining the mechanical movement parameter may comprise, or consist of, measuring said mechanical movement parameter directly, or calculating it. Several examples for both will be given in the course of this description.

[0018] The robotic arm may be single-jointed or multi-jointed. In case of a multi-joint robotic arm, it is preferred that the at least one mechanical movement parameter is obtained for each of the joints. For example, a torque value, a rotational speed value, a rotational acceleration value, and so on may be obtained for each of the joints of the robotic arm, e.g. using a corresponding sensor, typically arranged at the corresponding joint.

[0019] The robotic arm may be part of a robotic system that comprises another robotic arm, for example equipped (or equippable) with an instrument, in particular a medical instrument.

[0020] The robotic camera system may be a user-operated system, wherein the user navigates the robot based on a camera image (typically the image of the endoscopic camera itself) using wired or wireless remote control device. The robotic camera system may also be an automatic robotic camera system.

[0021] The term “mechanical” in “mechanical power” is used herein to stress that said power is not an electrical power. It is evident that the mechanical power threshold value Pmaxcan be set by a user of the robotic camera system depending on the environment for which it is intended. Since the robotic camera system according to the present invention is prevented from ever moving in a way such that a power larger than the mechanical power threshold value Pmaxwould be exerted onto any resting objecting coming in contact with it, by choosing a sufficiently small mechanical power threshold value Pmax tailored to the intended deployment environment it can be ensured that the damageable environment can never be damaged by a controlled movement of the endoscopic camera.

[0022] The modified trajectory may be calculated based on the original trajectory (according to the original command) based on any known trajectory modification, or object avoidance, algorithm known in the prior art. For example, the scientific publication “Passivation of Projection-Based Null Space Compliance Control Via Energy Tanks” by A. Dietrich et al. IEEE ROBOTICS AND AUTOMATION LETTERS, VOL. 1 , NO. 1 , JANUARY 2016, presents a concept of a passivated closed-loop system, in particular for human-robot interaction and safety-relevant applications.

[0023] Ideally and preferably, the modified trajectory has the same endpoint as the originally commanded trajectory. However, in extreme cases, the modified trajectory may collapse to a point, i.e., indicate a full stop, in the - under regular circumstances presumably quite rare - event that no further movement is possible without a crossing of the mechanical power threshold value Pmax. Similarly, if movement is possible but no acceptable (i.e., no nonthreshold-crossing) trajectory is found, the modified trajectory may be a point, or full stop, as well. The control unit may be configured to output in each or any of these cases a visual, acoustic, or haptic warning signal. It is also possible that the modified trajectory is a retracting trajectory, in the sense that it retraces a previously followed trajectory.

[0024] The control unit may be realized as any device or any means for computing, in particular for executing a software, an app, or an algorithm. For example, the control unit may comprise at least one processing unit such as at least one central processing unit, CPU, and / or at least one graphics processing unit, GPU, and / or at least one field-programmable gate array, FPGA, and / or at least one application-specific integrated circuit, ASIC and / or any combination of the foregoing. The control unit may further comprise a working memory operatively connected to the at least one processing unit and / or a non-transitory memory operatively connected to the at least one processing unit and / or the working memory. The control unit may be implemented partially and / or completely in a local apparatus and / or partially and / or completely in a remote system such as by a cloud computing platform.

[0025] In some advantageous embodiments, refinements, or variants of embodiments, the control unit is configured to, for obtaining the at least one mechanical movement parameter, obtain at least one force vector of a force that would be exerted by the robotic arm during the trajectory according to the command in case of a collision with a resting object.

[0026] In some advantageous embodiments, refinements, or variants of embodiments, obtaining the at least one force vector comprises obtaining at least one torque vector and matrixmultiplying the at least one torque vector with a robot-specific matrix.

[0027] For example, mechanical movement properties of the end effector of a robotic arm can be calculated using the Jacobian Matrix J (or simply “Jacobian” for short) of that robotic arm in its current (mechanical) configuration. Specifically, a Cartesian force F exerted by the end effector of the robotic arm with the Jacobian J is given by

[0028] F = ]~tt>(eq. 1) wherein ]~Tis the inverse transposed of the Jacobian J, and T is the robotic arm joint torque vector (comprising one torque value entry for each of the joints of the robotic arm), as may be found, for example, in the standard textbook “Introduction to robotics” by John J. Craig, in any of its many editions. The Jacobian J and its transposed inverse ]~Tare easy to calculate and constant for one robotic arm in one mechanical configuration. Typically, the Jacobian J is a 6x / i-dimensional matrix, with 6 degrees of freedom of movement (3 linear translations, 3 angular rotations), and n the number of joints of the robotic arm.

[0029] The torque value entry for one (or several, or all) of the joints may be sensed or measured directly. For example, one, several, or all joints of the robotic arm may be equipped with a corresponding torque sensor. Alternatively, the torque value entry for one (or several, or all) of the joints may be computed, for example based on a measurement of a motor current of a joint motor of the corresponding joint.

[0030] Accordingly, in some advantageous embodiments, refinements, or variants of embodiments, the robotic arm comprises at least one torque sensor, and the control unit is configured to read out the at least one torque vector T at least partially from the at least one torque sensor. Preferably, all of the joints of the robotic arm comprise (or: are equipped with) torque sensors, and the control unit is preferably configured to read out the entire torque vector T from the plurality of the torque sensors. Reading out the torque vector T “at least partially” indicates that the control unit may be configured to read out the torque vector entries r, for joints / that are equipped with torque sensors from those torque sensors, while it may obtain the torque vector entries r7for other joints / using one or more different methods.

[0031] In some advantageous embodiments, refinements, or variants of embodiments, the robotic arm comprises at least one joint motor, and the control unit is configured to read out at least one joint motor current value lm,i, and to derive the at least one torque vector T at least partially therefrom. Typically, a torque value TJ for a joint / is linearly proportional to its joint motor current value lm,i, so that the torque vector entry Tj for a joint / can be calculated by multiplying the joint motor current value lmof that joint with a stored proportionality factor k, of that joint / .

[0032] Preferably, the robotic arm comprises a joint motor at each of its joints. The control unit may be configured to read out, or measure, all the joint motor currents at each joint motor to obtain the corresponding joint motor current values lm,i- As has been described in the foregoing, a combination is possible, where, for example, some entries r7of the torque vector T are obtained based on joint motor current values lmand some entries Tj of the torque vector r are obtained based on torque sensor readouts (or: measurements).

[0033] In some advantageous embodiments, refinements, or variants of embodiments, the control unit is configured to, for obtaining the at least one mechanical movement parameter, obtain at least one velocity vector with which the robotic arm (specifically the end effector of the robotic arm) is moving or would move during the trajectory according to the command. For example, the (Cartesian) velocity vector of the end effector, symbolized by X, may be derived using the current Jacobian J using a joint rotational velocity vector 9, using the known equation = / 0, (eq. 2) wherein the joint rotational velocity vector 9 consists of entries 9, that each indicate the rotational speed value of a corresponding joint / of the joints. The rotational speed values , may in each case be measured or determined otherwise (e.g., calculated).

[0034] The calculation of the Jacobian J and / or the obtaining of the at least one mechanical movement parameter (or all mechanical movement parameters) preferably is performed regularly, in particular periodically, for example with a period of 10 microseconds or smaller, 5 microseconds or smaller, or 1 microseconds or smaller. As an example, encoders are able to give joint positions of the robotic arm at a high rate. Then, differentiations can be computed very fast to produce joint velocities, and also the Jacobian Jean be computed in a time span of the order of a few microseconds (e.g. 10 microseconds or less, 5 microseconds or less, or even 1 or less microseconds). So the end effector velocity can advantageously be computed very fast and within the control rate of the robotic arm.

[0035] In some advantageous embodiments, refinements, or variants of embodiments, the control unit is configured to obtain, for each of the at least one obtained force vector, F, a contemporaneous velocity vector X (specifically of the end effector of the robotic arm), for example as has been described in the foregoing.

[0036] The control unit may also be configured to calculate for at least one point along the trajectory a scalar power value Pof a mechanical power exerted by the robotic arm during the trajectory according to the command, the calculation using the obtained force vector Fand the obtained (Cartesian) velocity vector X at that at least one point.

[0037] Moreover, the control unit may be configured to determine whether the mechanical power threshold value Pmai< would be crossed (i.e., P> Pma ?) by the trajectory according to the command in case of a collision with a resting object based on the calculated scalar power value P.

[0038] Again using known equations, the power at the end effector of the robotic arm may be calculated using the Jacobian Jas well as some auxiliary parameters that may advantageously be calculated as has been described in the foregoing

[0039] P = F * X (eq. 3). With the force vector F being a covariant vector, and the velocity vector X being a contravariant vector, the resulting power value P is a scalar, which can therefore easily be compared to the (scalar) mechanical power threshold value Pmax.

[0040] The scalar power value P may be calculated, and compared to the mechanical power threshold value Pmax, specifically for a point at, or a position of, the endoscopic camera. Preferably, this is done for a plurality of points, and in each case compared to the mechanical power threshold value Pmax, with the consequences, i.e. a modification of the originally commanded trajectory in case the mechanical power threshold value Pmaxwould be crossed. Said points may be chosen points along an outer contour of the endoscopic camera, points most likely to collide with the damageable environment and / or points of a wire frame model of the endoscopic camera or another instrument wielded by the robotic arm. In some variants, the points are only taken along the outer contour of the endoscopic camera; in others, they may also be taken along the outer contour of the robotic arm.

[0041] In some advantageous embodiments, refinements, or variants of embodiments, the control unit is configured to determine the at least one mechanical movement parameter for all points along the trajectory according to the command, and to determine whether the mechanical power threshold value Pmaxwould be crossed by the movement according to the command in case of a collision with a resting object at any point along the trajectory. This provides additional safety for the operation of the robotic camera system.

[0042] In some advantageous embodiments, refinements, or variants of embodiments, the control unit comprises a threshold modification module configured to modify the mechanical power threshold value Pmax, for example based on a current position and / or orientation of the endoscopic camera, in particular with respect to its environment. The modification of the mechanical power threshold value Pmaxmay, for example, be based on the camera image of the endoscopic camera itself, for example according to a wavelength analysis of the image, a brightness analysis of the image, and / or the like.

[0043] Although sometimes herein the function of modifying the mechanical power threshold value Pmaxis described in the context of, or as being performed by, the threshold modification module, it shall be understood that this does not necessarily mean that such a module is provided as an entity distinctly separate from the control unit itself. The threshold modification module may be implemented by program code sections or program code snippets which may be distinct from the other program code of the control unit, but which may also be interwoven. In some advantageous embodiments, refinements, or variants of embodiments, the threshold modification module is configured to modify the mechanical power threshold value at least in part based on a pre-determined table available to the threshold modification module. For example, the space accessible to the endoscopic camera (taking into account possible changes of intended trajectories) may be divided into sections, wherein each section is associated with a different mechanical power threshold value Pmax, advantageously depending on the level of vulnerability (or: damageability) of the damageable environment within, and / or bordering on, said section.

[0044] In some advantageous embodiments, refinements, or variants of embodiments, the threshold modification module is configured to modify the mechanical power threshold value Pmaxat least in part based on an output of a machine learning model, such as an artificial intelligence entity. For example, a trained artificial neural network, ANN, comprising at least one convolutional layer, may be employed, using the image output of the endoscopic camera as its input, and giving a mechanical power threshold value Pmaxas its output.

[0045] In some advantageous embodiments, refinements, or variants of embodiments, the robotic arm is further configured to handle at least one medical instrument simultaneously in addition to the endoscopic camera.

[0046] In some advantageous embodiments, refinements, or variants of embodiments, the robotic camera system comprises another (or at least one other) robotic arm configured to handle at least one medical instrument, in particular while simultaneously the endoscopic camera is handled by the (first) robotic arm. The control unit may be configured to control the movement of any or all of its additional robotic arms in the same way as has been described for the robotic arm equipped with the endoscopic camera. Typically, the mechanical power threshold value Pmaxmay be the same for the control of all robotic arms, although it could also be different, for example in case the instruments at the end effectors of the robotic arms pose different degrees of danger to the damageable environment.

[0047] According to a second aspect of the present invention, a method for operating a robotic camera system in a damageable environment is provided, the method comprising at least the steps of: receiving a command indicating a trajectory to be followed by a robotic arm wielding (or: equipped with) a camera; obtaining at least one mechanical movement parameter for the trajectory of the robotic arm according to the command; determining, based on the obtained at least one mechanical movement parameter, whether a mechanical power threshold value would be crossed by the trajectory according to the command in case of a collision with a resting object; controlling, in case the mechanical power threshold value would be crossed, the robotic arm to move according to a modified trajectory in which the mechanical power threshold value would not be crossed; and controlling, in case the mechanical power threshold value would not be crossed, the robotic arm to move according to the command (or: to the unmodified trajectory).

[0048] Operating the robotic camera system in a damageable environment may be understood, for example, to mean that at least a part of the robotic camera system, typically at least the camera of the robotic camera system, is partially or entirely arranged within the damageable environment. It does not have to be the case (and will typically not be the case) that the entire robotic camera system is arranged within the damageable environment, although that may be a possibility.

[0049] In some advantageous embodiments, refinements, or variants of embodiments, the method further comprises modifying the mechanical power threshold value, Pmax, in particular based on a current position and / or orientation of the camera, specifically with respect to the damageable environment. The modification may, alternatively or additionally, also be performed according to any of the other variants described in the foregoing or in the following.

[0050] The method according to any embodiment of the present invention may be performed, for example, with the endoscopic robotic camera system according to any embodiment of the first aspect of the present invention, but also independent from it. Thus, the method may be adapted according to any options, variants, modifications, or refinements that are described with respect to the robotic camera system, and vice versa.

[0051] The invention also provides, according to a third aspect, a computer program product comprising executable program code configured to, when executed by a computing device, perform the method according to any embodiment of the second aspect of the present invention.

[0052] The invention also provides, according to a fourth aspect, a computer-readable, nontransient data storage medium comprising executable program code configured to, when executed by a computing device, perform the method according to any embodiment of the second aspect of the present invention. The invention also provides, according to a fifth aspect, a data stream comprising, or configured to generate, executable program code configured to, when executed by a computing device, perform the method according to any embodiment of the second aspect of the present invention.

[0053] Further technical considerations, advantages as well as variants and refinements are presented in the following, in particular in the dependent claims as well as in the specification with respect to the drawings and the drawings themselves.

[0054] BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The invention will be explained in greater detail with reference to exemplary embodiments depicted in the drawings as appended.

[0056] The accompanying drawings are included to provide a further understanding of the present invention, are incorporated in, and constitute a part of this specification. The drawings illustrate the embodiments of the present invention and together with the description serve to explain the principles of the invention.

[0057] The numbering of method steps is done for the purpose of distinguishing between them and does not necessarily imply a temporal order although a temporal order according to the numbering is possible. In particular, one or more method steps may be performed at the same time, overlapping one another, continuously or repeatedly, and / or the like. Although the method is presented herein in a comprehensive manner including several steps, it shall be understood that most of the steps are not essential and only provide advantageous additional benefits.

[0058] In the figures:

[0059] Fig. 1 shows a schematic depiction of an endoscopic robotic camera system according to an embodiment of the present invention;

[0060] Fig. 2 shows a schematic flow diagram illustrating a method according to another embodiment of the present invention;

[0061] Fig. 3 shows a schematic block diagram illustrating a computer program product according to yet another embodiment of the present invention; and

[0062] Fig. 4 shows a schematic block diagram illustrating a data storage medium according to still another embodiment of the present invention. DETAILED DESCRIPTION OF THE DRAWINGS

[0063] Fig. 1 shows a schematic depiction of an endoscopic robotic camera system 1000 according to an embodiment of the present invention. As an example of a delicate, damageable environment 1 , a schematic representation of a patient is shown, specifically, of an internal cavity of that patient, such as the one used in, or created for, a minimally invasive surgical procedure, such as a laparoscopy or the like. It shall be understood that the same endoscopic robotic camera system 1000 may also be used with and for any other kind of damageable environment 1 , even if medical applications will feature prominently in the following.

[0064] The endoscopic robotic camera system 1000 comprises at least one a single-jointed or multijointed robotic arm 110; 210; 310. In Fig. 1 , three multi-jointed robotic arms 1 10; 210; 310 are shown. In this example, a first robotic arm 1 10 is equipped with an endoscopic camera 120 of the endoscopic robotic camera system 1000, which is mounted to an end effector of the first robotic arm 1 10.

[0065] In the following, most details will be presented about the first robotic arm 110; however, it shall be understood that all of these details may equally apply to any of the other robotic arms 210; 310. As has been described in the foregoing, these other robotic arms 210; 310 may be equipped with their own instrument, in particular medical instruments such as a scalpel 220 or another cutting instrument, a clamp, an electric sealing tool, and / or the like. Medical instruments may include articulated / wristed medical instruments, straight medical instruments, monopolar medical instruments, and / or bipolar medical instruments.

[0066] The endoscopic robotic camera system 1000 further comprises a control unit 100, shown in Fig. 1 only schematically. As has been described, the control unit 100 may be implemented as, or by, a computing device of any kind. It may also be advantageously integrated into another device such as into a camera control unit, a robotic arm 1 10; 210; 310, or the like.

[0067] The control unit 100 is configured to receive a command 71 (or, more precisely: a command signal indicating the command 71 ) for the robotic arm 110, indicating a trajectory to be followed by the robotic arm 1 10 within the damageable environment 1 , specifically by the end effector of the robotic arm 110.

[0068] For example, while somebody (e.g., a physician) is moving the scalpel 220 using the second robotic arm 210, the command 71 may indicate a trajectory that is designed such that the field-of-view of the endoscopic camera 120 follows that scalpel 220. The command 71 may be received from a remote controller 50 operated by a user, who may be locally present (e.g., in the same operation room as the patient), or remotely connected. The command 71 may in another embodiment, or even alternatively in the same embodiment, be received from a machine learning model 60, e.g. an artificial intelligence entity, that is trained and configured to generate the command 71 automatically. The endoscopic robotic camera system 1000 may accordingly comprise such a remote controller 50 and / or such a machine learning model 60, configured for generating the command 71 and transmitting it to the control unit 100.

[0069] As a reminder, the control unit 100 may also receive further commands indicating trajectories to be followed by the other robotic arms 210; 310 within the damageable environment 1 (specifically by their end effectors), for which everything applies analogously. In particular, the commands relating to other robotic arms 210; 310 may be received from the same source as the commands 71 for the first robotic arm 110, or from different sources. For example, it is common that a first person, e.g. an assistant or assistant physician, operates the endoscopic camera 120, while a second person, e.g. a head physician, operates at least one medical instrument (here for instance the scalpel 220). In that case, the first person may operate a first remote controller 50 for generating commands 71 for the first robotic arm 110, while the second person may operate a second remote controller 50 for generating commands for (at least) the second robotic arm 210.

[0070] In other variants, a single remote controller 50 may be configured to generate commands for multiple, or all, robotic arms 110; 210; 310. In yet another variant, one or more machine learning models 60 are trained and configured to generate the commands for all of the robotic arms 110; 210; 310.

[0071] The control unit 100 is further configured to obtain at least one mechanical movement parameter of the robotic arm 110 for the trajectory according to the command 71 .

[0072] As an example, sensor signals 73 may be received by the control unit 100 from the robotic arm 110 (and / or the other robotic arms 210; 310), and the at least one mechanical movement parameter may be based on the received sensor signals 73.

[0073] As one particular example of the various examples that have been described in the foregoing, the control unit 100 may be configured to receive sensor signals 73, indicating a torque value Ti (as a first mechanical movement parameter) of each of the joints 111-1 , 111- 2, 111-3 (sometimes collectively designated herein as 111-i), as well as a rotational velocity value 9\ (as a second mechanical movement parameter) for each of the joints 111-i at the same time, i.e., contemporaneously. Although in Fig. 1 only three joints 111-1 , 111-2, 111-3 are marked explicitly, it shall be understood that any robotic arm 110; 210; 310 may comprise any number of joints 111 -i.

[0074] The control unit 100 is able to put together a torque vector r as well as a rotational velocity vector e , wherein in each case the entries are given by the values for the individual joints 111 -i of the robotic arm 110.

[0075] It should be remembered that receiving sensor signals 73 is only one way of receiving the at least one mechanical movement parameter, other ways being receiving predicted mechanical movement parameters from a model or the like. Any or all mechanical movement parameters may also be present within the control unit 100 itself as part of the calculation the control unit 100 makes to control the robotic arm 110 based on the command 71.

[0076] Returning to the example, with the known equations (eq. 1) and (eq. 2) cited above, using a Jacobian matrix J of the robotic arm 110 available to the control unit 100 (preferably regularly, periodically, or constantly calculated, or updated, by the control unit 100 based on the current configuration of the robotic arm 110), the control unit 100 is configured to calculate the force vector Ffrom the torque vector r, and to calculate the (Cartesian) velocity vector X from the rotational velocity vector 9. Then, with the force vector Fand the velocity vector X, using equation (eq. 3), the mechanical power P is calculated, P = F * X.

[0077] The control unit 100 is further configured to determine, based on the obtained at least one mechanical movement parameter, whether a mechanical power threshold value, Pmax, would be crossed by the trajectory according to the command in case of a collision with a resting object, in particular with the damageable environment 1 surrounding it (e.g. a patient’s organs). In the present example, this is done by the control unit 100 comparing the calculated mechanical power Pto a predetermined and / or dynamically adjustable mechanical power threshold value Pmax-

[0078] The control unit 100 is also configured to control, in case the mechanical power threshold value Pmaxwould be crossed, the robotic arm 110 (using control signals 72) to move according to a modified trajectory in which the mechanical power threshold value Pmax would not be crossed, and to control, in case the mechanical power threshold value Pmax would not be crossed, the robotic arm 110 to move according to the trajectory according to the command 71 . Different variants of generating the modified trajectory have been described in the foregoing, for example with respect to the scientific publication by A. Dietrich et al.

[0079] For example, at some time point t, the control unit 100 determines that the mechanical power P currently has a value of Pi<Pmax- The control unit 100 is also able to determine that in order to follow the trajectory according to the command 71 , it would have to send control signals 72 to the robotic arm 110 that would result in a new mechanical power P2>Pmax- Thus, the control unit 100 in this example determines that the mechanical power threshold value Pmaxwould be crossed, and generates or selects a modified trajectory. In case of several choices for the modified trajectory, a modified trajectory that runs as close as possible (for example determined according to the method of least squares) may be selected preferentially.

[0080] As has also been already mentioned, variants are possible in which the mechanical power threshold value Pmai< is modifiable, for example depending on the position and / or orientation of the endoscopic camera 120 with respect to the damageable environment 1 , using the output of a machine learning model 60, or a pre-determined table, or the like. Thus, the control unit 100 may comprise a threshold modification module 101 configured to modify the mechanical power threshold value Pmai< during operation of the endoscopic robotic camera system 1000, in particular in any of the ways that have been described in the foregoing. The mechanical power threshold value Pmai<may be modified once or more, twice or more,..., regularly or even continuously.

[0081] Fig. 2 shows a schematic flow diagram illustrating a method according to an embodiment of the present invention, i.e. a method for operating a robotic camera system 1000 in a damageable environment. The method may be performed using the endoscopic robotic camera system 1000 so that in the following some of the reference signs in Fig. 1 will be mentioned. However, it shall be understood that the method is also performable independent of the endoscopic robotic camera system 1000 of Fig. 1 .

[0082] In a step S10, a command 71 indicating a trajectory to be followed by a robotic arm 110 wielding (or: equipped with, in particular at its end effector) a camera (in particular an endoscopic camera 120) is received, for example by a control unit 100 and / or from a remote controller 50 and / or from a machine learning model 60 as has been described in the foregoing.

[0083] In a step S20, at least one mechanical movement parameter for the trajectory of the robotic arm 110 according to the command 71 is obtained, e.g. a current value of a mechanical movement parameter while the robotic arm 110 is already moving along said trajectory. As has been described in the foregoing, the at least one mechanical movement parameter may be selected from a large number of possibilities, for example, torque values, joint motor currents, rotational velocity values, and so on. Obtaining S20 the at least one mechanical movement parameter may comprise measuring S21 and / or calculating S22 any or each of the at least one mechanical movement parameter. In a step S30, based on the obtained at least one mechanical movement parameter, it is determined (or: checked) whether a mechanical power threshold value Pmaxwould be crossed by the trajectory according to the command 71 in case of a collision with a resting object (e.g. the damageable environment 1), for example as has been described in the foregoing with respect to configuration of the control unit 100.

[0084] In case it is determined that the power threshold value Pmaxwould be crossed (“+” symbol in Fig. 2), in a step S40 the robotic arm 110 is controlled (e.g. using control signals 72 as shown in Fig. 1) to move according to a modified trajectory in which the mechanical power threshold value Pmaxwould not be crossed (or: exceeded). Step S40 may comprise a substep S41 of retrieving the modified trajectory, for example from a predetermined library, and / or a sub-step S42 of calculating (or: generating) the modified trajectory, for example as has been described in the foregoing with respect to the control unit 100.

[0085] In case it is determined that the power threshold value Pmaxwould not be crossed (“-” symbol in Fig. 2), in a step S50 the robotic arm 110 is controlled (e.g. using control signal 72 as shown in Fig. 1) to move according to the original trajectory according to the command 71 .

[0086] In both cases, thereafter the method may continue at step S20, so that the currently intended trajectory is always evaluated for any potential damage to the damageable environment 1 . If the trajectory has been modified in step S40, this new trajectory is then treated as the “original” trajectory in step S30, which may be further modified (step S40 again) or not (step S50 again) in the following, and so on. The modification of the original trajectory may then be treated as a new command.

[0087] In an optional step S15, the mechanical power threshold value Pmaxis, or may be, modified, based e.g. on a current position and / or orientation of the camera 120, in particular with respect to its environment, for example as has been described in the foregoing with respect to the threshold modification module 101 . As indicated by the small loop at its left-hand side in Fig. 2, step S15 may be continuously performed (or it may be continuously monitored whether step S15 should be performed, triggering step S15 accordingly) so that the mechanical power threshold value Pmaxcan immediately be modified when the underlying situation changes, e.g., when the camera 120 is moved into a more, or less, easily damageable environment 1 , when something about the damageable environment 1 changes, and / or the like.

[0088] Fig. 3 shows a schematic block diagram illustrating a computer program product 400 according to an embodiment of the third aspect of the present invention. The computer program product 400 comprises executable program code 450 configured to, when executed (by a computing device such as the control unit 100), perform the method according to any embodiment of the second aspect of the present invention, in particular as has been described with respect to the preceding figures.

[0089] Fig. 4 shows a schematic block diagram illustrating a non-transitory computer-readable data storage medium 500 according to an embodiment of the fourth aspect of the present invention. The data storage medium 500 comprises executable program code 550 configured to, when executed (by a computing device such as the control unit 100), perform the method according to any embodiment of the second aspect of the present invention, in particular as has been described with respect to the preceding figures.

[0090] The non-transient computer-readable data storage medium may comprise, or consist of, any type of computer memory, in particular semiconductor memory such as a solid-state memory. The data storage medium may also comprise, or consist of, a CD, a DVD, a Blu- Ray-Disc, an USB memory stick or the like.

[0091] Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate and / or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the present invention. Generally, this application is intended to cover any adaptations or variations of the specific embodiments discussed herein.

[0092] List Signs

[0093] 1 cavity

[0094] 50 remote controller

[0095] 60 machine learning model

[0096] 71 command

[0097] 72 control signal

[0098] 73 sensor signal

[0099] 100 control unit

[0100] 101 threshold modification module

[0101] 110 robotic arm

[0102] 120 endoscopic camera

[0103] 111 -i joint

[0104] 210 robotic arm

[0105] 220 scalpel

[0106] 310 robotic arm

[0107] 400 computer program product

[0108] 450 program code

[0109] 500 data storage medium

[0110] 550 program code

[0111] 1000 endoscopic robotic camera system

[0112] S10..S50 method steps

Claims

Claims1 . An endoscopic robotic camera system (1000), comprising: a single-jointed or multi-jointed robotic arm (110); an endoscopic camera (120) mounted on the robotic arm (110); and a control unit (100) configured to: receive (S10) a command (71 ) for the robotic arm (110) indicating a trajectory to be followed by the robotic arm (1 10); obtain (S20) at least one mechanical movement parameter of the robotic arm (110) for the trajectory according to the command (71 ); determine (S30), based on the obtained at least one mechanical movement parameter, whether a mechanical power threshold value, Pmax, would be crossed by the trajectory according to the command (71 ) in case of a collision with a resting object (1 ); control (S40), in case the mechanical power threshold value, Pmax, would be crossed, the robotic arm (1 10) to move according to a modified trajectory in which the mechanical power threshold value, Pmax, would not be crossed; and control (S50), in case the mechanical power threshold value, Pmax, would not be crossed, the robotic arm (1 10) to move according to the command (71 ).

2. The robotic camera system (1000) of claim 1 , wherein the control unit (100) is configured to, for obtaining (S20) the at least one mechanical movement parameter, obtain at least one force vector, F, of a force that would be exerted by the robotic arm (1 10) during the trajectory according to the command (71 ) in case of a collision with a resting object.

3. The robotic camera system (1000) of claim 1 , wherein obtaining the at least one force vector comprises obtaining at least one torque vector, T, and matrix-multiplying the at least one torque vector, T, with a robot-specific matrix, J~T.

4. The robotic camera system (1000) of claim 3, wherein the robotic arm comprises at least one torque sensor, and the control unit (100) is configured to read out the at least one torque vector, T, at least partially from the at least one torque sensor.

5. The robotic camera system (1000) of claim 3, wherein the robotic arm comprises (1000) at least one joint motor, and the control unit (100) is configured to read out at least one joint motor current value, lm,i, and to derive the at least one torque vector, T, at least partially therefrom.

6. The robotic camera system (1000) of any of claims 1 to 5, wherein the control unit (100) is configured to, for obtaining the at least one mechanical movement parameter, obtain at least one velocity vector with which the robotic arm (100) is moving or would move during the trajectory according to the command (71).

7. The robotic camera system (1000) of claim 6 and any of claims 2 to 5, wherein the control unit (100) is configured to obtain, for each of the at least one obtained force vector, F, a contemporaneous velocity vector, X, to calculate for at least one point along the trajectory a scalar power value, P, of a mechanical power exerted by the robotic arm (110) during the trajectory according to the command (71 ) using the obtained force vector, F, and the obtained velocity vector, X, at that at least one point, and to determine (S30) whether the mechanical power threshold value, Pmax, would be crossed by the trajectory according to the command (71) in case of a collision with a resting object (1) based on the calculated scalar power value, P.

8. The robotic camera system (1000) of any of claims 1 to 7, wherein the control unit (100) is configured to determine the at least one mechanical movement parameter for all points along the trajectory according to the command (71), and to determine (S30) whether the mechanical power threshold value, Pmax, would be crossedby the movement according to the command (71) in case of a collision with a resting object at any point along the trajectory.

9. The robotic camera system (1000) of any of claims 1 to 8, wherein the control unit (100) comprises a threshold modification module (101 ) configured to modify the mechanical power threshold value, Pmax, based on a current position and / or orientation of the endoscopic camera (120), in particular with respect to its environment (1 ).

10. The robotic camera system (1000) of claim 9, wherein the threshold modification module (101) is configured to modify the mechanical power threshold value, Pmax, at least in part based on an output of a machine learning model (60), such as an artificial intelligence entity.11 . The robotic camera system (1000) of claim 9 or claim 10, wherein the threshold modification module (101) is configured to modify the mechanical power threshold value, Pmax, at least in part based on a pre-determined table available to the threshold modification module (101).

12. The robotic camera system (1000) of any of claims 1 to 11 , wherein the robotic arm is (110) further configured to handle at least one medical instrument (220) simultaneously in addition to the endoscopic camera (120), or wherein the robotic camera system (1000) comprises another robotic arm (210; 310) configured to handle the at least one medical instrument (220).

13. A method for operating a robotic camera system in a damageable environment, comprising: receiving (S10) a command (71) indicating a trajectory to be followed by a robotic arm (110) wielding a camera (120); obtaining (S20) at least one mechanical movement parameter for the trajectory of the robotic arm according to the command;determining (S30), based on the obtained at least one mechanical movement parameter, whether a mechanical power threshold value, Pmax, would be crossed by the trajectory according to the command (71) in case of a collision with a resting object; controlling (S40), in case the mechanical power threshold value, Pmax, would be crossed, the robotic arm (110) to move according to a modified trajectory in which the mechanical power threshold value, Pmax, would not be crossed; and controlling (S50), in case the mechanical power threshold value, Pmax, would not be crossed, the robotic arm (110) to move according to the command (71).

14. The method of claim 13, further comprising modifying (S15) the mechanical power threshold value, Pmax, in particular based on a current position and / or orientation of the camera (120), specifically with respect to the damageable environment (1 ).

15. A computer program product (400) comprising executable program code (450) configured to, when executed by a computing device, perform the method according to claim 13 or claim 14.

Images