Support system, computer-implemented control method, and computer-readable recording medium.

Abstract

The present invention relates to a surgical assistance system (1), which comprises a surgical robot (2) having a movable robot arm connected to a robot base (4), a visualization system (18) configured to generate and provide at least one real-time image, a surgical effector (20) positioned on a work axis (24) and having a work point (26) that is positioned or can be positioned as intended within the field of view (30) of the visualization system (18), and an optical reference marker positioned on the effector (20) at a predefined position and orientation relative to the work point (26), wherein the optical detection of the optical reference marker determines the position of the work point (26) and / or the work axis ( The present invention also relates to a computer-readable recording medium according to a dependent claim.

Description

Technical Field

[0001] The present disclosure relates to a medical, particularly surgical assistance system, particularly a cranial nerve surgery assistance system, comprising a navigation-assisted surgical robot for use in surgical procedures on patients. For this purpose, the robot has a robot base as a local link point of the robot, particularly as a local static coordinate system or reference coordinate system. A movable or actively movable robot arm having at least one robot arm segment is connected to the robot base. Furthermore, the assistance system has, as its end effector, a visualization system particularly comprising one or more cameras, which is attached, particularly mounted, to the robot arm, particularly to the end side of the robot arm. The visualization system is configured to generate at least one real-time physical image of the patient, preferably an in-vivo image, and preferably provide it digitally. Furthermore, the present disclosure relates to a computer-implemented control method, a computer-readable recording medium, and a computer program.

Background Art

[0002] In the fields of medicine and medical technology, automation involving the integration of digitally controllable technical devices is becoming increasingly important. Robots are being increasingly utilized in surgical procedures, particularly to assist in highly precise minimally invasive procedures. Here, robots are not only provided as stand-alone types that simply perform surgeries, but are increasingly being used directly at the treatment site as collaborative robots (cobots), i.e., assisting or auxiliary robots, that interact with medical personnel, particularly surgeons.

[0003] In such robot-assisted surgical procedures, particularly cranial nerve surgical procedures, various visualization systems and measurement systems are generally used for visual, physiological, and functional evaluation of the treatment area, as well as for decision-making support. Conventional data detected during the procedure includes images from visualization systems arranged in the treatment area, such as surgical microscopes and / or endoscopes.

[0004] In conventional support systems, instruments are tracked via optical reference markers attached to the instruments, specifically by a rigid body with four traversable spheres in space. In neurosurgery, either a navigation pointer or a suction tube is used as the instrument. However, the size and weight of the reference markers can interfere with ergonomics, creating additional workload and costs, and making them complex to use. Furthermore, microscopes often obstruct the line of sight between the navigation camera and the instrument, and the rigid body with the reference markers also obstructs the view within the microscope's field of view, causing line-of-sight problems. As a result, the navigation camera needs to be manually corrected and moved, which disables the automatic tracking function, i.e., the function in which the visualization system automatically follows the movement of the instrument. An additional problem is that the optical axis of the visualization system coincides with (coaxial with) the working axis of the instrument. Consequently, the field of view of the visualization system is strongly affected by the instrument's shaft, and is partially obstructed by the shaft. [Overview of the Initiative]

[0005] In light of these circumstances, this disclosure aims to provide a surgical assistance system, a computer-implemented control method, a computer-readable recording medium, and a computer program that avoid or at least reduce the shortcomings of the prior art, and in particular enable extremely good and intuitive control of the visualization system. Part of the purpose of this disclosure is to provide a hands-free control system that allows the surgeon to keep both hands free for the procedure while also being able to control the system using instruments held in the hands. Part of the purpose of this disclosure is to use the simplest possible objects for robot control for visualization during the procedure, in particular to ensure that these objects do not obstruct the view of the visualization system. Another purpose is to realize a method of controlling the robot with the minimum possible configuration, thereby improving manufacturing costs and avoiding error factors in existing effectors such as medical instruments. Another purpose is to enable particularly high-precision control of the robot and visualization system using effectors. Ergonomics should also be improved.

[0006] These objectives are resolved by the features of claim 1 with respect to the surgical support system according to the present invention, by the features of claim 12 with respect to the computer implementation method according to the present invention, and by the features of claim 16 with respect to the computer-readable recording medium.

[0007] Accordingly, the basic idea of ​​this disclosure is to provide a surgical assistance system comprising a robotically operated visualization system, in particular a surgical microscope or endoscope, or both, the surgical assistance system being configured to control the operation of the visualization system via (medical or surgical) effectors, in particular surgical instruments. Here, the optical axis of the visualization system is controlled, in particular tracked, with respect to the working axis of the effector using (image-based) tracking, and this control or tracking may be operated continuously / permanently and / or selectively, in particular on request, by the user.

[0008] Specifically, the surgical assistance system, in particular the neurosurgical assistance system, comprises a surgical robot, in particular a navigation-assisted surgical robot, used in surgical procedures. The robot has a robot base as a local link point of the robot and a movable robot arm connected to the robot base, having at least one robot arm segment, in particular a plurality of robot arm segments, the plurality of robot arm segments being connected to each other via bearings, in particular hinges, thereby enabling the active configuration of the robot arm. A visualization system (or visualization unit) is attached, in particular to one end of the robot arm, as an end effector. The visualization system is configured to generate at least one real-time intracellular image of the patient and then provide the intracellular image digitally and, consequently, in a computer-readable manner. Furthermore, the assistance system includes a surgical effector (separate from the robot), in particular a surgical instrument (separate from the robot), which can be manually guided or guided, in particular by a medical professional, or is mechanically operated or held. The working point of the effector is located on or at the position of the effector's principal axis, longitudinal axis, or working axis. As intended, the work point is positioned within the field of view of the visualization system, or at least can be positioned within it, thereby allowing the visualization system to provide the surgeon with an internal image in which the work point is drawn, as needed. Furthermore, an optical reference marker is provided, which is positioned on the effector at a predetermined position, particularly at a predetermined distance, and particularly at a predetermined position and orientation relative to the work point. The reference marker is configured to determine the position of the work point and / or the orientation of the work axis based on its optical detection. In contrast to conventional optical reference markers in the sense of “rigid bodies,” according to this disclosure, the reference marker is formed by an optical pattern extending at least along the surface portion of the effector, preferably along the surface portion of the effector housing. In principle, the pattern can be positioned at any position on the effector, but a distal region or distal portion is preferred, and particularly preferred to be positioned near the work point, in order to enable extremely high-precision localization within the field of view of the visualization system.By positioning the pattern distally, instruments such as suction devices, suction cups, or suction tubes can be bent before use to conform to anatomical structures and ergonomics without affecting navigation accuracy. Another component of the support system is a tracking system, particularly a navigation system, which is configured, in accordance with this disclosure, to determine the position of a work point and / or the orientation of a work axis in response to or based on a detected pattern. In particular, the adaptation of the tracking system includes a machine vision algorithm capable of determining the position of a work point and / or the orientation of a work axis based on an image and the pattern contained in the image, relative to the coordinate system of the visualization system. The control unit of the support system is configured to control the position of the visualization system and / or the orientation of the optical axis of the visualization system and / or the hyperfocal distance of the visualization system in accordance with the determined position of the work point and / or the determined orientation of the work axis. The tracking system of the support system is also preferably used to track the pose, i.e., position and orientation of the visualization system, particularly as a spatial reference to the patient, thereby also tracking the pose of at least one real-time body image.

[0009] The extension of the reference marker, configured as a pattern along the surface or portion of the surface of the effector, is positioned, or is intended to be positioned, within the field of view as intended, and has the following advantages over conventional optically detectable reference markers, such as a separate fixed / rigid body protruding from the effector or effector housing: the reference marker is no longer limited in visibility by a visualization system such as a microscope, thus eliminating the need to readjust the microscope's position, and the field of view of the visualization system is no longer obstructed by a rigid body. Therefore, the visualization system does not need to adjust its position or orientation to match the interference position or pose of the reference marker, as was required in the prior art when a rigid body was in the field of view.

[0010] In this disclosure, “effector” means, for example, a device, instrument, or similar medical device that may be used in a procedure to perform a procedure. In particular, examples of effectors include instruments, medical devices such as endoscopes or suction tubes, optical devices having a visualization axis, and pointers / indicators having a distal end for surgical navigation. In this disclosure, the “working axis” of an effector means, in particular, the longitudinal axis of the effector, in particular the longitudinal axis of the distal portion of the effector. For example, this includes the longitudinal axis of the cutting portion of a scalpel, the longitudinal axis of the tip of an endoscope, or, in the case of a rigid endoscope, the longitudinal axis of the endoscope shaft. For example, in the case of a dental instrument having a non-linear tip working portion and working point, such as a probe, the working axis may be defined in the linear portion of the instrument's handle.

[0011] Here, "working point" refers to the point or area of ​​the effector used as intended during a procedure. For example, the working point of a scalpel is the tip of the scalpel, the working point of a pointer / indicator is the tip of the pointer, the working point of scissors is the tip of the sheath of the scissors when closed, the working point of tweezers is the tip of the tweezers, the working point of a drill or cutter is its tip, the working point of a syringe is its tip, the working point of a trocar is its tip, and the working point of an endoscope is its tip. On the other hand, in the case of dental probes, the working point is the tip of the instrument, which is not the most distal point (slightly offset) from the working axis, but extends slightly "backward" in the proximal direction, similar to a hook.

[0012] In this context, "robot arm segment" refers specifically to robot components of a robot arm mounted between bearings or hinges, or, in the case of end-side robot arm segments, specifically to robot components connected in series between the end effector and the preceding robot arm segment (or the corresponding robot base if there is only one robot arm segment).

[0013] The term "position" refers to a geometric location in three-dimensional space, specifically defined by coordinates in the Cartesian coordinate system. In particular, a position may be defined by three coordinates: X, Y, and Z.

[0014] On the other hand, the term "orientation" refers to alignment in space (e.g., in terms of position), particularly axial alignment. Orientation can also be said to specify alignment with a representation of direction or rotation in three-dimensional space. In particular, orientation may be specified using up to three angles.

[0015] The term "pose" encompasses both position and orientation. Specifically, a pose can be specified by six coordinates: three position coordinates (X, Y, and Z) and up to three angular coordinates representing orientation.

[0016] Advantageous embodiments are described in the dependent claims and are specifically explained below.

[0017] In a further development, the control unit is configured to move the visualization system in tracking mode, so that the position of the effector's working point coincides with the center point of the image, or another predetermined or predefined point in the image, resulting in the visualization system following the effector's working point. Furthermore, the optical axis has a predetermined angle, particularly a static relationship, with respect to the working axis, and this angle is maintained during the movement of the effector. This tracking mode can be enabled and disabled by the user, and the effector with the working axis can be used like a joystick to set the angle and perform spatial positioning. For example, the tip of the pointer as the working point may indicate the point that should be the center of the image, and the robot moves the visualization system accordingly, tracking the tip of the pointer. Furthermore, the alignment of the optical axis is specified at a predetermined angle, such as 45°, with the longitudinal axis of the pointer as the working axis. This allows the user to intuitively and quickly identify a target area, especially when using a surgical microscope as the visualization system, and to display the area without interrupting the operation and without using their hands for adjustment. Optical reference markers require only minor modifications to instruments such as pointers and do not typically require new approvals. This is because only the surface is optically marked, and an adaptable control unit can use the instrument as an input means.

[0018] In a further developmental form, the control unit is configured to determine a flat bandwidth, in particular a plane spanning the initial optical axis of the visualization system and the new working axis of the effector, define a new optical axis of the visualization system having a predetermined angle with respect to the new working axis within the flat bandwidth / plane (as the effector moves and the corresponding working axis changes), and accordingly control the robotic arm for positioning the visualization system in space.

[0019] The surface portion of the effector having a pattern is preferably curved or planar. The pattern is preferably a 3D object or a 2D object. The pattern is preferably formed by engraving, anodizing, or laser processing of the surface portion. Mixed forms of these are also possible.

[0020] Preferably, the surface portion has a shape based on a standard geometric shape, preferably cylindrical. Preferably, its cross-section is circular or polygonal, particularly quadrilateral, pentagonal, or hexagonal. In a preferred further development, the pattern is formed parallel to the surface portion. Alternatively, the pattern has a cross-section with respect to the working axis that is different from the cross-section of the surface portion, for example, a polygonal cross-section, while the surface portion has a circular cross-section. In this case, the pattern forms angled portions, which can be easily evaluated in an image using image recognition. The pattern is at least partially dotted, and / or linear, and / or curved.

[0021] In a further developmental form, the effector is composed of fixed, i.e., non-replaceable functional units. According to this disclosure, a functional unit is a tool that performs a function required during a procedure. For example, such a functional unit may be a cutter, a drill, or an imaging system such as an ultrasound probe. The working point is located at the distal end (tip) or at least the distal region of the functional unit.

[0022] Alternatively, the effector may have a replaceable holder into which one or more possible functional units or tools can be selectively inserted. The pattern is preferably positioned within the area of ​​the replaceable holder. Preferably, all functional units, when inserted into the replaceable holder, have their respective work points at the same distance and preferably in the same orientation relative to the pattern positioned in the replaceable holder. This eliminates the need to input information about the deviation in distance or orientation of the work points into the support system when replacing functional units.

[0023] According to a preferred further development, in order to always reliably perform pattern detection and the determination of the position of the working point and / or the orientation of the working axis based thereon, the pattern is configured, dimensioned and / or arranged on the surface portion such that when the working point is arranged within the field of view as intended, it is at least partially arranged within the field of view.

[0024] According to a further development, the pattern is configured such that it can still be detected as intended, i.e., to determine the position of the working point and / or the orientation of the working axis, even when it is partially occluded, for example, by the operator's hand.

[0025] According to a further development, the pattern has a variation at least in the direction of the working axis.

[0026] According to a further development, a model of the pattern is stored in the tracking system or provided to the tracking system by the control unit. Preferably, the tracking system is configured to determine the position of the working point of the effector and / or the orientation of the working axis according to the detected pattern and the stored model.

[0027] In particular, images, especially microscopic images, are used to detect the position and / or orientation of the instrument in the microscope coordinate system through machine vision, i.e., machine vision. Preferably, the microscope and the patient are detected by an external camera, especially the camera of the navigation system, thereby closing the spatial tracking chain, and the position and / or orientation of the instrument in the patient coordinate system may be calculated via the control unit.

[0028] If the detection of the rotation around the working axis of the effector is desirable or relevant for the control of the visualization system, according to a further development, the pattern also has a variation in the circumferential direction of the working axis.

[0029] The variation may in particular be of a geometric and / or color nature.

[0030] Regardless of the rotation of the effector, in order to achieve the best detectability of the pattern, the pattern in a further developed form extends around the entire working axis or at least along the circumferential direction.

[0031] However, when the rotation of the effector is not relevant, that is, for example, when the visualization system does not follow the rotation of the effector, it has been found that a further developed form in which the pattern is at least partially rotationally symmetric with respect to the working axis is advantageous. On the other hand, when the rotation of the effector is relevant, especially when the visualization system follows the rotation of the effector, the pattern is at least partially rotationally asymmetric with respect to the working axis.

[0032] In a preferred further developed form, the pattern has elements arranged at intervals in the direction of the working axis.

[0033] In particular, the effector may have, as a pattern, concentric rings arranged at intervals around its working axis and having different widths in the working axis direction.

[0034] Even when the pattern is partially unclear, according to a further developed form, in order to easily determine the position of the working point and the orientation of the working axis, the distance between the elements and / or the respective extensions of each element in the direction of the working axis have variations.

[0035] According to a further developed form, as having a similar effect, at least some of the elements are grouped together such that the distance between the elements and / or the extensions of each element in the working axis direction are the same within these groups, while the distance between the elements and / or the extensions of the elements between the groups have variations.

[0036] In a particularly preferred further development, the visualization system is configured to provide a tracking system with an image containing a pattern, at least partially, and the tracking system is configured to determine the position of the effector's working point and / or the orientation of its working axis in accordance with the image. The advantage of this is that the visualization system has a dual function: on the one hand, it provides the image required by the surgeon to visualize the surgical site; on the other hand, it provides the input necessary to control the effector's dependent position and / or orientation and / or focal length using the image.

[0037] To determine this, the tracking system preferably has an image recognition algorithm configured to determine the position of the work point and / or the orientation of the effector in accordance with the image of the visualization system. Alternatively, or additionally, the tracking system may have a machine learning algorithm for image recognition, which is configured to determine the position of the work point and / or the orientation of the effector in accordance with the image of the visualization system.

[0038] This eliminates the need for a separate tracking or navigation camera to determine the position of the work point and / or the pose of the effector. This is simpler from an equipment technology standpoint, reduces costs, and requires less installation space, thereby increasing the usable space in the operating room. Naturally, to further improve the process reliability or accuracy for determining the position of the work point and / or the orientation of the work axis, at least one separate 2D or 3D camera may be provided for optical detection of the pattern. This at least one separate 2D or 3D camera may also be used for navigating the robotic arm relative to the patient.

[0039] For similar reasons of improving process reliability and accuracy, complementary detection techniques based on alternative reference markers, such as electromagnetic, X-ray, radar, laser, and / or LiDAR-based techniques, may be provided depending on the embodiment.

[0040] In a further development, the control unit is configured to continuously or continuously control the visualization device with respect to its position and / or the orientation of its optical axis and / or its hyperfocal distance. Alternatively, the control unit may be configured to perform the corresponding control only when requested or when input is received. A third variation is that the control is performed selectively, i.e., continuously or on demand (on-demand), i.e., in two freely selectable modes. This provides both continuous tracking and input-based tracking, which the user can switch between.

[0041] In a further development, a manually operable or automated request device may be provided, which is configured to send the request. In particular, examples of request devices include a foot pedal and / or push button provided on the effector, and / or a voice recognition device and / or a gesture recognition device.

[0042] The position and / or orientation are preferably controlled by driving the robot and at least one of its robotic arms to perform the translational movements and / or rotations necessary to move the visualization system to the position and / or orientation corresponding to the effector.

[0043] Preferably, the control unit is configured to control the position of the visualization system, and / or the orientation of the optical axis of the visualization system, and / or the hyperfocal distance of the visualization system, in accordance with the position of a determined work point and / or the orientation of a determined work axis, so that the work point is located at the center or center point of the field of view or a predefined point, and / or the optical axis is set at a predetermined angle with respect to the work axis, and / or the work point coincides with the hyperfocal distance.

[0044] In a further development, the control unit is configured to set the hyperfocal distance to the working point when the position of the visualization system is fixed, i.e., unchangeable. Furthermore, or alternatively, the control unit is configured to set the position of the visualization system such that the hyperfocal distance coincides with the working point when the hyperfocal distance is fixed. In a preferred further development, the control unit is configured in both of the above-described manner.

[0045] A method relating to this disclosure is provided for controlling the position of a visualization system, particularly a surgical assistance system, configured according to one of the embodiments described above, and / or the position of its optical axis and / or its hyperfocal distance. Such control is performed in accordance with a tracked surgical effector. According to this disclosure, the method comprises the following steps: "A step of generating and providing real-time images of the treatment area in which the working point of the surgical effector is positioned, or can be positioned, as intended, within the field of view of the visualization system." This step is performed via a visualization system connected to a robotic arm; "A step of optically detecting an optical reference marker, or at least a portion thereof, wherein the optical reference marker is formed by an optical pattern extending along at least a surface portion of an effector, and is positioned on the effector at a predetermined location, particularly at a predetermined distance, and in a predetermined orientation relative to a work point, and is configured to determine the position of the work point and / or the orientation of the work axis from the optical detection thereof." "A step of determining the position of a work point and / or the orientation of a work axis, in accordance with an optically detected pattern, particularly via machine vision based on images related to the coordinate system of the visualization system." This step is performed via a tracking system; "A step of controlling the visualization system with respect to the position of the visualization system and / or the orientation of the optical axis of the visualization system and / or the hyperfocal distance of the visualization system, depending on the position of the determined work point and / or the orientation of the determined work axis." This step is performed by a control unit adapted for this purpose.

[0046] Preferably, the method further comprises the step of "tracking a visualization system, preferably a surgical microscope and / or surgical endoscope, particularly by navigation." This step is performed by a tracking system, or alternatively, a navigation system.

[0047] In a further development of this method, the step of "controlling the visualization system with respect to the position of the visualization system and / or the orientation of the optical axis of the visualization system and / or the hyperfocal distance of the visualization system, depending on the position of the determined work point and / or the orientation of the determined work axis" is performed continuously. Here, "continuously" means that the visualization system continuously follows all movements of the effector. In a further development, it is possible to attenuate or smooth the movement of the visualization system so that the visualization system moves smoothly even when the effector moves rapidly. Naturally, similar attenuation or smoothing is possible when adjusting the focal length.

[0048] Alternatively, control may be performed only when requested, preferably in response to a request signal. In this case, the visualization system remains stationary with respect to its position and / or orientation and / or focal length for a certain period until a request signal occurs, at which point the visualization system is repositioned and / or adjusted to match the effector and / or its focal length is adjusted.

[0049] The request signal may be issued manually, for example, by a surgeon. Alternatively, the request signal may be issued automatically, in particular in response to at least one piece of surgical context information.

[0050] Furthermore, further developments are possible in which the robot's movements, and consequently the visualization system's movements, are restricted to individual translational or rotational degrees of freedom X, Y, Z, RX, RY, RZ, or a selection thereof, such as rotation only or translational movement only.

[0051] In a further developmental form, the step of "optically detecting the pattern or part of the pattern" is performed by the visualization system by generating an image such that the pattern or at least part of the pattern is drawn within the image.

[0052] In a further developmental form, the step of "determining the position of the work point and / or the orientation of the work axis" is performed via an image processing algorithm stored and executed in a tracking system, particularly a navigation system, which processes the image.

[0053] In a further developed form, the step of "controlling the visualization system with respect to the position of the visualization system and / or the orientation of the optical axis of the visualization system and / or the hyperfocal distance of the visualization system, depending on the position of the determined work point and / or the orientation of the determined work axis" is performed depending on the current position of the visualization system and / or the current orientation of its optical axis and / or the deviation of the current hyperfocal distance of the visualization system from the target.

[0054] The objective includes, in particular, that the working point is visible in a predetermined area of ​​the field of view, preferably the center or center point or a predetermined point, and / or that the working point is drawn in the image, and / or that the working point coincides with the hyperfocal distance, and / or that the optical axis has a predetermined angle with respect to the working axis. If the predetermined angle is not zero, it prevents the effector, in particular its shaft, from obstructing or partially obstructing the field of view of the microscope. Preferably, the control unit is configured to allow the predetermined angle to be preset, adjusted, and / or changed, preferably according to the operator's preference. Alternatively, the predetermined angle may be 0°.

[0055] In a further development, the step of "controlling the visualization system with respect to the position of the visualization system and / or the orientation of the optical axis of the visualization system and / or the hyperfocal distance of the visualization system, depending on the position of a determined work point and / or the orientation of a determined work axis" includes the step of "driving the robot to move at least a predetermined area of ​​the field of view or image, in particular the center, center point or a predefined point, to align with the work point and to set the optical axis at a predetermined angle with respect to the work axis." With respect to the requirement that the work point coincides with the focal length, there are two possible steps. In one variation, the distance from the visualization system to the work point is fixed, and then the step of "setting the hyperfocal distance so that the work point is in focus or coincides with the hyperfocal distance" is performed. In another variation, the hyperfocal distance is fixed, and then the step of "setting the distance from the visualization system to the work point so that the work point is in focus or coincides with the hyperfocal distance" is performed.

[0056] However, if a predetermined angle is the sole target, it is not possible to define a clear pose of the optical axis in space, as the optical axis may rotate around the working axis or an axis parallel thereto. In order to clearly define the optical axis in space, it is necessary to define the plane in which the optical axis actually exists. According to this disclosure, there are two ways to achieve this.

[0057] In one modification, a plane is generated that straddles the determined working axis and the focal point of the current optical axis. The target of the new optical axis is located in this plane at a predetermined angle to the determined working axis. In this modification, the optical axis does not rotate even if the effector rotates around the working axis.

[0058] In another variation, the pattern is configured so that all translational movements and rotations of the effector, including rotation around the effector's own working axis, are determined from the detection of the effector. In this case, as the effector rotates around its own axis, the optical axis also moves.

[0059] For most possible effectors or devices, it can be assumed that the rotation of the effector itself is not relevant to the tracking of the visualization system. Therefore, the first modification is preferred.

[0060] With respect to computer-readable recording media or computer programs, the object of this disclosure is resolved by including instructions that, when executed by a computer, cause the computer to perform each step of the method provided by this disclosure.

[0061] Any disclosures relating to surgical support systems made herein shall apply to the methods described herein, and vice versa. [Brief explanation of the drawing]

[0062] The present disclosure will be described in more detail below with reference to preferred embodiments, using the drawings.

[0063] [Figure 1] Figure 1 shows a schematic side view of a surgical support system according to a preferred embodiment.

[0064] [Figure 2] Figure 2 shows a perspective view of the visualization system and effector of the support system shown in Figure 1.

[0065] [Figure 3] Figure 3 shows a detailed diagram of an effector equipped with an optical reference marker in the pattern configuration shown in Figure 2.

[0066] [Figure 4] Figure 4 schematically shows the sequence of steps using the surgical support system shown in Figures 1 to 3.

[0067] [Figure 5] Figure 5 shows a detailed view of Figure 4.

[0068] The figures are schematic and intended to aid in understanding the present invention. Identical components are denoted by the same reference numerals. Features of different embodiments are interchangeable. [Modes for carrying out the invention]

[0069] Figure 1 shows a schematic side view of a surgical assistance system 1 according to a preferred embodiment. The assistance system 1 comprises a surgical robot 2 having a robot base 4, and in the illustrated embodiment, the robot base 4 is locally fixed. Alternatively, the surgical robot 2 may be movable, for example, to be used in different locations within a hospital operating room, as needed. In either case, the robot base 4 forms a local reference point, to which a multistage robot arm having a plurality of robot arm segments 6, 8, and 10 is connected. These segments are connected to each other via hinges 12 and 14. This allows the robot arm segments 8 and 10 to move actively relative to each other, and the robot arm 6, 8, 10, 12, and 14 to be controlled as a whole.

[0070] In the illustrated embodiment, the visualization system 18 of the support system 1 is configured as a surgical microscope and is attached to one end 16 of the robot arm. In the illustrated embodiment, the support system 1 further includes a manual guide effector 20 or instrument 20 tracked by the tracking system of the support system 1. The optical axis 22 of the visualization system 18 is inclined at an angle W with respect to the working axis 24 of the effector 20. The working point 26 of the effector 20 is located at the distal end of the working axis 24.

[0071] The position of the visualization system 18 and the orientation of its optical axis 22 can be controlled and adjusted via a specially adapted central control unit 28 located on the base 4. Here, the special adaptation is performed in particular in accordance with the position of the work point 26 and / or the orientation of the work axis 24, which are determined each time by the machine vision, and this will be described in detail below.

[0072] Figure 2 is a perspective view of the visualization system 18 having an optical axis 22 and the instrument 20 having a working axis 24, as shown in Figure 1. The visualization system 18 has an expanding field of view 30, within which the instrument 20 is at least partially visible and its working point 26 is positioned to be visible. During the procedure, the visualization system 18, for example a surgical microscope, continuously generates real-time images of the treatment area of ​​patient P and provides these images digitally.

[0073] The instrument 20 has a shaft with a surface formed by a standard shape, which in a specific configuration is formed by a circular cylinder. At the distal end of the shaft adjacent to the work point 26, its surface portion has a reference marker 32 formed as an optical pattern. The reference marker 32 is configured such that its optical image detection can be evaluated using an image processing algorithm for machine vision, thereby determining the position of the work point 26 and the orientation of the work axis 24.

[0074] According to this disclosure, optical detection is performed using the images of a visualization system 18, instead of using a separate 3D or 2D navigation camera system as is common in conventional solutions.

[0075] Figure 3 shows detail A, defined based on Figure 2, which is a magnified view of the distal surface portion of the instrument 20 having the pattern 32 in the area of ​​the work point 26. The pattern 32 is divided into individual elements 34, 36, and 38 spaced apart along the work axis 24, and for clarity, in Figure 3, only some of these elements are typically represented by reference numerals. Each of the elements 34, 36, and 38 is at a predetermined distance from the work point 26, and this predetermined distance is stored in the control unit 28 in particular. According to Figure 3, the pattern 32 has variation in the direction of the work axis. In a specific configuration example, this variation is in that the width or extension portion of each of the elements 24, 36, and 38 differs in the aforementioned direction, and in some cases, the distance between them also differs. For example, element 38 is the widest, element 32 is the narrowest, and the width of element 36 is in the intermediate range. All elements 34, 36, and 38 are grouped into groups 40, 42, 44, 46, and 48, where group 40 contains only one element 34, group 42 contains two elements 34, group 44 contains three elements 34, group 46 contains four elements 36, and group 48 contains three elements 38. All elements 34, 36, and 38 extend around the entire circumference and rotationally symmetrically on the surface portion of the fixture 32. By extending the optically detectable pattern 32 along the surface of the fixture 20 in the manner described above, the pattern 32 represents a reference marker formed as an optically detectable 3D object similar to a conventional rigid body. The advantage of this is that it does not require additional installation space and does not add any additional weight. Furthermore, since the reference marker is provided on the surface of the fixture 20 which is already in the field of view or is to be placed in the field of view, it does not obstruct the line of sight.

[0076] Figures 4a to 4d illustrate each step of the method relating to this disclosure using the surgical support system shown in Figures 1, 2, and 3.

[0077] Figure 4a shows the intended initial or basic state in which the visualization system 18 satisfies predetermined targets with respect to the position of the visualization system 18 in space, the orientation of its optical axis 22, and its hyperfocal distance D, in relation to the position of the working point 26 of the instrument and the orientation of the working axis 24. In this state, the following criteria of the target are met: the working point 26 is located in the center of the field of view 30, the working point 26 coincides with the focal distance D, and is located on the image plane 27. Furthermore, the optical axis 22 has a predetermined angle W with respect to the working axis 24.

[0078] In principle, the optical detection of the optical pattern 32 described above is performed continuously through the generation of images by the visualization system 18. Based on these images, the tracking system according to this disclosure determines the position of the work point 26 and the orientation of the work axis 24 by performing image processing evaluation of the images and the patterns 32 contained in the images using a machine vision algorithm. The control unit 28 continuously checks whether the visualization system 18 satisfies the above objective, and in the basic state shown in Figure 4a, the objective is satisfied.

[0079] Figure 4b shows the instrument 20 deflected from the position and orientation described in Figure 4a. Accordingly, the tracking system according to this disclosure determines the changed position and orientation of the work point 26 and the work axis 24. As a result, the control unit 28 determines the corresponding deviations from the target of the field of view 30, the orientation of the optical axis 22, and the hyperfocal distance D.

[0080] The control unit 28 determines the necessary translational movement T and rotation R in one or more translational or rotational degrees of freedom in order to compensate for the deviation from the aforementioned target by controlling the robot accordingly.

[0081] According to Figure 4c, the robot is first controlled via the control unit 28 to move the visualization system 18 only in translation, thereby moving the work point 26 to the center of the field of view 30 and establishing a match between the work point 26 and the hyperfocal distance D.

[0082] In the illustrated configuration example, the robot is controlled sequentially with respect to translational movement T and rotation R, but alternatively, the necessary translational movement T and rotation R may be synchronized to be performed simultaneously.

[0083] Even after the translational movement T shown in Figure 4c has been performed sequentially, the optical axis 22 and the work axis 24 still have a small angle w that deviates from the target W. This angular deviation from the target W is then compensated for by the control unit 28 controlling the robot to perform the required rotation R, as shown in Figure 4d.

[0084] Compensation for deviations from the target due to the required translational movement T and rotation R may be performed automatically, constantly, or continuously, or selectively only on request, as described with reference to, for example, Figures 4a to 4d. The request signal required for this may be transmitted, for example, by the user pressing an actuation element such as a pedal or button located within the user's operating area. For example, the device 20 may be equipped with such an actuation element as an operating interface. Alternatively, the aforementioned request signal may be transmitted by gesture or voice command.

[0085] The predetermined angle W is set according to the user's preference and may be adapted or changed as needed. The angle W prevents the shaft of the instrument 20 from excessively or partially obstructing the field of view 30 of the microscope.

[0086] Figure 5 shows an enlarged view of the process described above. The basic state of the support system 1 is shown on the right side of Figure 5, and includes the visualization system 18, the optical axis 22, the image plane 27 positioned perpendicular to the optical axis 22, the working axis 24 of the effector 20, and the working point 26 in its coordinates X, Y, Z. The optical axis 22 is tilted relative to the working axis 24 at a predetermined angle W according to the target, and the working point 26 is located at the center of the field of view and within the image plane 27, as it coincides with the hyperfocal distance.

[0087] As shown in Figure 5, the position and orientation of the device 20 are changed to a new position of the work point 26' having new coordinates X', Y', Z' corresponding to the translational movement T, and further, the orientation of the work axis 24' of the device 20' is changed to a new orientation corresponding to the rotation R.

[0088] The visualization system 18 continuously generates images showing the deformation (translational movement T and rotation R) of the instrument 20 in the optically detected changes or distortions of pattern 32.

[0089] The tracking system determines a new pose or orientation of the device 20 by processing images, i.e., by using machine vision algorithms, and provides this to the control unit 28.

[0090] Based on this, the control unit 28 determines the translational movement T and rotation R of the visualization system 18 necessary to achieve the goal. As soon as the operator initiates readjustment of the visualization system 18 via the appropriate request signal, the control unit 28 controls the robot to move the support system 1' to the configurations represented by symbols 1', 18', 22', 24', and 27'. [Explanation of Symbols]

[0091] 1 Surgical support system 2 Robot 4 Robot base 6, 8, 10 Robot arm segments 12, 14 Hinge 16 Robot arm end 18 Visualization system 20 Effector 22 Optical axis 24 Working axis 26 Working point 27 Image plane 28 Control unit 30 Field of view 32 Optical reference marker S1 Step of tracking the visualization system S2 Step of generating an image S3 Step of optically detecting the reference marker S4 Steps of determining the pose of the effector S5, S5.1, S5.21, S5.22 Steps of controlling the visualization system

Claims

1. A surgical support system (1) comprising a surgical robot (2) used for surgical procedures, A robot base (4) as a local link point of the robot (2), and a movable robot arm having at least one robot arm segment (6, 8, 10) connected to the robot base (4), A visualization system (18) having a surgical microscope and / or endoscope, which is connected to the robot arm and configured to generate and provide at least one real-time image in real time, In particular a surgical instrument, and in particular a manually guided surgical effector (20), the surgical effector (20) having a working point (26) that is positioned on or near the working axis (24) and is positioned or can be positioned as intended within the field of view (30) of the visualization system (18), An optical reference marker positioned on the effector (20) at a predetermined position, particularly at a predetermined position and orientation, with respect to the work point (26), wherein the optical reference marker is configured to determine the position of the work point (26) and / or the orientation of the work axis (24) based on optical detection of the optical reference marker, and is formed by an optical pattern (32) extending along at least a surface portion of the effector (20), A tracking system configured to determine the position of the work point (26) and / or the orientation of the work axis (24) in accordance with the optically detected pattern (32), particularly via machine vision based on images related to the coordinate system of the visualization system, A surgical assistance system comprising: a control unit (28) configured to control the position of the visualization system (18) and / or the orientation of the optical axis (22) of the visualization system (18) and / or the hyperfocal distance (D) of the visualization system (18) according to the determined position of the work point (26) and / or the determined orientation of the work axis (24).

2. In the surgical support system (1) according to claim 1, The control unit (28) is configured to move the visualization system (18) in tracking mode so that the position of the work point (26) of the effector (20) coincides with the center point or a predefined point of the image, thereby the visualization system (18) tracks the work point (26) of the effector (20), and the optical axis (22) has a predetermined angle (W), particularly a static relationship, with respect to the work axis (24), and maintains this angle while the effector (20) is moving, in a surgical assistance system.

3. In the surgical support system (1) described in any one of the prior claims, The control unit (28) is configured to control the position of the visualization system (18) and / or the orientation of the optical axis (22) of the visualization system (18) and / or the hyperfocal distance (D) of the visualization system (18) according to the determined position of the work point (26) and / or the determined orientation of the work axis (24). A surgical support system characterized in that, when the position of the visualization system (18) is fixed, the hyperfocal distance (D) is set to the work point (26), or, when the hyperfocal distance (D) is fixed, the position of the visualization system (18) is set so that the hyperfocal distance (D) coincides with the work point (26).

4. In the surgical support system (1) described in any one of the prior claims, The control unit (28) is The visualization system (18) is configured to be controlled continuously / constantly with respect to its position and / or the orientation of its optical axis (22) and / or its hyperfocal distance (D), or A surgical support system characterized by being configured to control the visualization system (18) in response to a request / input, particularly in response to at least one request signal.

5. In the surgical support system (1) described in any one of the prior claims, The control unit (28) is A flat bandwidth is determined, in particular a plane that spans the (initial) optical axis (22) and the new working axis (24'). A surgical support system characterized in that it is configured to define a new optical axis (22') having a predetermined angle (W) with respect to the new working axis (24') within the aforementioned bandwidth.

6. In the surgical support system (1) described in any one of the prior claims, The effector (20) is composed of a fixed, non-replaceable functional unit on which the work point (26) is located, or The surgical support system is characterized in that the effector (20) has a replaceable holder, one of a plurality of possible functional units can be selectively inserted into the holder, each of the functional units has the work point (26), and the replaceable holder is equipped with the pattern (32).

7. In the surgical support system (1) according to claim 6, A surgical support system characterized in that the pattern (32) and / or a part of the pattern are each at a predetermined distance from the work point (26), and the predetermined distance is stored in a memory device.

8. In the surgical support system (1) described in any one of the prior claims, The pattern (32) has variations at least in the direction of the work axis (24), and in particular the pattern (32) has elements (34, 36, 38) that are spaced apart in the direction of the work axis (24), A surgical support system characterized in that the distance between the elements (34, 36, 38) and / or the extension portions of each of the elements (34, 36, 38) in the direction of the work axis (24) change.

9. In the surgical support system (1) according to claim 7 or 8, A surgical support system characterized in that at least a portion of the elements (34, 36, 38) are grouped into groups (44, 46, 48), and within each group, the distance between the elements (34, 36, 38) and / or the extension portions of the elements (34, 36, 38) in the direction of the work axis (24) are the same, and in particular, the distance between the elements (34, 36, 38) and / or the extension portions of the elements (34, 36, 38) between each group (44, 46, 48) varies.

10. In the surgical support system (1) described in any one of the prior claims, The pattern (32) extends around the entire circumference of the work axis (24), and / or A surgical support system characterized in that the pattern (32) is at least partially rotationally symmetric with respect to the work axis (24).

11. In the surgical support system (1) described in any one of the prior claims, The visualization system (18) is configured to provide the images to the tracking system, A surgical support system characterized in that the tracking system is configured to determine the position of the work point (26) of the effector (20) and / or the orientation of the work axis (24) according to the image.

12. A computer implementation method for a surgical assistance system (1) according to any one of the prior claims, in particular for controlling the position of a visualization system (18) and / or the orientation of the optical axis (22) of the visualization system (18) and / or the hyperfocal distance (D) of the visualization system (18) in accordance with a tracked surgical effector (20) during surgery, particularly in neurosurgical procedures, Step (S2) of generating and providing real-time images of the treatment area by a visualization system (18) connected to a robot arm (6, 8, 10), wherein the working point (26) of the surgical effector (20) is positioned, or can be positioned, as intended within the field of view (30) of the visualization system (18), Step (S3) of optically detecting an optical reference marker, or at least a portion of the optical reference marker, wherein the optical reference marker is formed by an optical pattern (32) extending along at least a surface portion of the effector (20), is positioned on the effector (20) at a predetermined position, particularly at a predetermined position and orientation, with respect to the work point (26), and is configured to determine the position of the work point (26) and / or the orientation of the work axis (24) from the optical detection of the optical reference marker. Step (S4) of determining the position of the work point (26) and / or the orientation of the work axis (24) by a tracking system, in accordance with the optically detected pattern (32), particularly via machine vision based on images related to the coordinate system of the visualization system, A computer implementation method characterized by the step (S5) of controlling the visualization system (18) via the control unit (28) with respect to the position of the visualization system (18) and / or the orientation of the optical axis (22) of the visualization system (18) and / or the hyperfocal distance (D) of the visualization system (18), in accordance with the determined position of the work point (26) and / or the determined orientation of the work axis (24).

13. In the computer implementation method described in claim 12, Step (S5) of controlling the visualization system (18) via the control unit (28) with respect to the position of the visualization system (18) and / or the orientation of the optical axis (22) of the visualization system (18) and / or the hyperfocal distance (D) of the visualization system (18), in accordance with the determined position of the work point (26) and / or the determined orientation of the work axis (24), Step (S5.1) of driving the robot such that the field of view (30) or the center or center point of the image, or a predetermined point in the field of view (30) or the image, is moved to the work point (26), and the optical axis (22) is set to a predetermined angle (W) with respect to the work axis (24), Selectively, Step (S5.21): Set the hyperfocal distance (D) to a predetermined distance from the visualization system (18) to the work point (26) such that the work point (26) falls within the focal point or coincides with the hyperfocal distance (D), A computer implementation method characterized by comprising either step (S5.22) setting the distance from the visualization system (18) to the work point (26) to a predetermined fixed hyperfocal distance (D) so that the work point (26) falls within the focal point or coincides with the hyperfocal distance (D).

14. In the computer implementation method according to claim 12 or 13, A computer implementation method characterized in that the step (S5) of controlling the visualization system (18) with respect to the position of the visualization system (18) and / or the orientation of the optical axis (22) of the visualization system and / or the hyperfocal distance (D) of the visualization system is performed continuously or only when requested, preferably in response to a request signal, according to the determined position of the work point (26) and / or the determined orientation of the work axis (24).

15. In the computer implementation method according to any one of claims 12 to 14, Step (S5) of controlling the visualization system (18) with respect to the position of the visualization system (18) and / or the orientation of the optical axis (22) of the visualization system and / or the hyperfocal distance (D) of the visualization system, in accordance with the determined position of the work point (26) and / or the determined orientation of the work axis (24), is performed in accordance with the deviation of the current position of the visualization system (18) and / or the current orientation of the optical axis (22) and / or the current hyperfocal distance (D) of the visualization system (18) from the target. A computer implementation method characterized in that the objective includes the working point (26) being visible in a predetermined area of ​​the field of view (30), and / or the working point (26) being drawn in the image, and / or the working point (26) being coincident with the hyperfocal distance (D), and / or the optical axis (22) having a predetermined angle (W) with respect to the working axis (24).

16. A computer-readable recording medium, A computer-readable recording medium, which, when executed by a computer, includes instructions causing the computer to perform each method step of a method for controlling a visualization system (18) according to any one of claims 12 to 15.

Images