Optical axis calibration of a robotic camera system

Abstract

A method in which instructions are executed from a computer-readable medium calibrates a robotic camera system having a digital camera connected to an end effector of a serial robot. The end effector and camera move within a robot motion coordinate system ("robot coordinate system"). The method includes acquiring a reference image of a target object on an image plane having an optical coordinate system using the camera and receiving input signals including depth measurements and joint position signals. While moving the robot, separate roll and pitch offsets of a target point in the reference image relative to the robot coordinate system are determined. Offsets are also determined relative to the x-, y-, and z-axes of the robot coordinate system while moving the robot in separate motion sequences. The offsets are stored in a transformation matrix, which is used to control the robot during subsequent operations of the camera system.

Description

[Technical Field]

[0001] This disclosure relates to an automated method and system for calibrating the optical axis of a robotic camera system. [Background technology]

[0002] Surgeons are often assisted by real-time digital imaging of the patient's target anatomical structure. Ophthalmologists performing retinal and vitreous surgery, for example, view high-magnification images of the retina or other intraocular anatomical structures in real time, either using a high-resolution medical display screen positioned within the surgeon's field of view, or through the optical components of a microscope. The camera can be securely mounted on an end effector located at the distal end of a multi-joint serial robot. The collective motion of the serial robot's various joints and interconnected linkage mechanisms is controlled using an electronic control unit to appropriately orient and position the camera relative to the target anatomical structure.

[0003] For this purpose, a multi-axis serial robot having multiple interconnected arm segments may be used in an operating room to allow an attached digital camera to rotate or translate as needed. An example of such a serial robot is disclosed in U.S. Patent No. 10,917,543B2 to Alcon, Inc., entitled “Stereoscopic Visualization Camera and Integrated Robotics Platform,” which is incorporated herein by reference in its entirety. Robot motion occurs within a robot motion reference coordinate system (the “Robot Coordinate System”), which has at least the nominal x, y, and z axes of a typical Cartesian coordinate system.

[0004] A robotic camera system used to assist in the performance of automated machine vision assistance tasks is defined by operating parameters, including the required minimum resolution, field of view, depth of field, and optical working distance. In particular, as used herein, optical working distance refers to the linear distance along the optical axis extending between the projection center (CoP) of the digital camera and the imaged target located in the image plane, as opposed to the distance between the bottom of the camera or its housing and the target, e.g., a patient. As described above, coupled to the distal end of a serial robot via a suitable end-effector, the digital camera digitally images the target within the camera's own reference coordinate system ("camera coordinate system"). In most mathematical models of the type commonly used to control the motion of serial robots, the camera coordinate system is assumed to be orthogonal to the robot coordinate system. Therefore, in order to ensure that the digital camera is properly focused on the intended target point with respect to the robot's understanding of its relative position within the robot coordinate system, the various electronic motion control commands and feedback signals used to position the end-effector and digital camera within the workspace must first be converted to the robot coordinate system. [Overview of the project] [Means for solving the problem]

[0005] This specification discloses an automated method and associated system for calibrating the optical axis of a digital camera in a robotic camera system. The method proceeds without the need to foresee or model the relevant parameters of the camera's optical system. Rather, the method uses the parameters to generate a homogeneous transformation matrix used for subsequent control of the robotic camera system in accordance with this teaching.

[0006] As is understood in this art, in machine vision applications requiring relatively low levels of positional accuracy, potential differences between different robot and camera coordinate systems tend to be ignored. In contrast, in machine vision applications requiring relatively high levels of positional accuracy, such as precision microsurgery, attempts may be made to fully model the behavior of the optical system and then map the resulting optical model to a kinematic model of the robot's motion behavior. However, implementing such methods involves cumbersome programming work, which is inherently prone to potential positional errors because deriving accurate and reliable optical models is extremely difficult.

[0007] Therefore, in applications that do not rely on the availability of a complete optical model, significant positional errors can occur when calculating the position of a target point on a reference image. This problem worsens in precision applications with long optical distances. Under such conditions, even a slight rotation or tilt of the camera's line-of-sight vector can result in relatively large positional errors. For example, ophthalmic microscopes may have a constant or variable optical distance of approximately 250mm to 350mm. In such an exemplary configuration, a mere 0.1-degree optical axis tilt angle can result in a positional error of 5mm to 10mm on the image plane.

[0008] As an example of the actual effects that such positional errors can have, consider the exemplary case where an ophthalmic surgeon expects to see a specific target point on the displayed optical image, for example, the exact center of the dilated pupil during cataract surgery. However, due to the positional errors described above, the surgeon will instead see a completely different target point, perhaps located on the surrounding surface of the iris. The surgeon will then need further control and adjustment to properly locate the desired target point, which will prolong the surgery and result in a suboptimal outcome.

[0009] To that end, the method described in detail herein enables a simple automated calibration process when connecting a digital camera to a robot end effector. As expected by the robot's underlying target acquisition and tracking logic, which itself is referred herein as target lock or LTT function, such connection does not necessarily result in perfect alignment of the camera's optical coordinate system ("camera coordinate system") with the robot's motion coordinate system ("robot coordinate system"). That is, the camera's line-of-sight vector may be slightly tilted due to adjustments made by the surgeon or due to imperfections in the mechanical coupling mechanism used to fix the camera to the serial robot's end effector. This can result in unacceptably high levels of positional error, especially in applications utilizing longer optical distances. To minimize the resulting positional error, a transformation matrix is ​​generated during the calibration stage of the method, and the generated transformation matrix is ​​used to control the subsequent motion control stage of the robot camera system.

[0010] More specifically, the robotic camera system envisioned herein includes a digital camera coupled to an end-effector, which is positioned at the distal end of a serial robot. Thus, the end-effector and the connected digital camera move within the robotic coordinate system by the movement of the serial robot. Generally, the method proceeds by acquiring a reference image of a target object (e.g., the surface of a patient's eye or another target anatomical structure in a non-limiting ophthalmic surgical use case). The image is collected in the camera coordinate system, as opposed to the robotic coordinate system described above.

[0011] The method also involves receiving input signals using an electronic control unit (ECU) that communicates with a serial robot via wired or wireless means, the ECU being composed of a model of robotic kinematics. The ECU is characterized by the absence of a model of camera optics, as described above. The input signals include depth measurements indicating the linear distance to the target object / image plane and joint position signals indicating the position of the end effector in the robot coordinate system. For simplicity, the robot coordinate system may be described as having the nominal x, y, and z axes of a typical Cartesian reference system.

[0012] The method may include using the ECU to determine roll angle offset and pitch angle offset as angular offsets of a target point located in a reference image, such offsets obtained relative to the robot coordinate system. The method also includes determining separate x-axis offset, y-axis offset, and z-axis offset of the target point, and then recording or storing the roll, pitch, x-axis, y-axis, and z-axis offsets in the ECU's memory or in a memory accessible by the ECU. The transformation matrix is ​​then used by the ECU, along with the robot kinematics described above, to control the motion sequence of the serial robot during the subsequent operation of the robot camera system. Therefore, once the digital camera has been properly calibrated according to this method, the robot camera system does not need to be recalibrated after each subsequent use, as long as the camera remains connected to the end effector.

[0013] In some configurations, the camera may have a variable optical distance, for example, to allow a surgeon to change the optical distance during surgery. The variable optical distance may be adjusted using a focus motor. In such embodiments, the method may include recording a plurality of z-axis offsets in a lookup table while adjusting the variable optical distance over an optical distance or focal range using the focus motor. In such embodiments, determining the z-axis offsets may include retrieving the z-axis offsets from a pre-prepared lookup table during a subsequent motion sequence.

[0014] The method may optionally include using an ECU to process the autofocus settings of the camera system and determine the depth measurement described above. Alternatively, the ECU may measure the depth measurement using a depth sensor, such as a laser rangefinder or an optical sensor.

[0015] Acquiring a reference image of a target object in an optical coordinate system may involve using a digital camera to collect a digital image of a two-dimensional checkerboard pattern or another pixelated target.

[0016] The serial robot may optionally be embodied as a six-axis ophthalmic surgical robot equipped with a digital camera connected to or integrated with an ophthalmic microscope coupled to the robot's end effector. Subsequent operation of the robotic camera system may include performing three-dimensional visualization of the eye under examination during ophthalmic surgery, for example, during vitrectomy or lens replacement / cataract surgery.

[0017] Another aspect of this disclosure includes a camera system having a digital camera and an ECU that communicates with the digital camera. The digital camera, for example, a stereo camera connected to or integrated with a microscope, can be connected to an end effector of a serial robot. The end effector and the digital camera move within the robot's motion coordinate system. The ECU that communicates with the digital camera is configured to carry out the methods outlined above.

[0018] This specification also discloses a computer-readable medium on which instructions are recorded. When used with a robotic camera system having a digital camera connected to the end effector of a serial robot, for example, the processor of the ECU described above executes instructions, and the processor performs the method outlined above.

[0019] The foregoing summary is not intended to represent all possible embodiments or aspects of the present disclosure. Rather, this summary is intended to illustrate some of the novel aspects and features disclosed herein. The above and other possible features and advantages of the present disclosure will become readily apparent when read in conjunction with the accompanying drawings and the appended claims, which set forth representative embodiments and aspects for carrying out the present disclosure.

Brief Description of the Drawings

[0020] [Figure 1] Shows a diagram of a robotic camera system in which a digital camera is connected to an end effector of a serial robot, and the optical axis of the digital camera can be calibrated according to the present disclosure. [Figure 2] Is a schematic diagram of a position error that can occur from a slight angular inclination of the optical axis or line-of-sight vector in the robotic camera system of FIG. 1. [Figure 3] Is a flowchart illustrating an exemplary embodiment of a method for calibrating the optical axis of the robotic camera system shown in FIG. 1. [Figure 4] Is a diagram of a digital camera attached to a microscope, arranged with respect to a checkerboard target when implementing the embodiment of the method shown in FIG. 3. [Figure 5A] Is a schematic diagram of a representative movement of a digital camera during the process of calibrating the optical axis of the digital camera according to an aspect of the present disclosure. [Figure 5B] Is a schematic diagram of a representative movement of a digital camera during the process of calibrating the optical axis of the digital camera according to an aspect of the present disclosure. [Figure 5C] Is a schematic diagram of a representative movement of a digital camera during the process of calibrating the optical axis of the digital camera according to an aspect of the present disclosure. [Figure 6] Is a graph of optical operating distance versus focus motor position, where the optical operating distance is represented in meters and shown on the vertical axis, and the focus motor position is represented in encoder counts and shown on the horizontal axis.

Mode for Carrying Out the Invention

[0021] The aforementioned and other features of this disclosure will become more readily apparent when the following description and the attached claims are read in conjunction with the accompanying drawings.

[0022] Embodiments of the present disclosure are described herein. However, it will be understood that the embodiments disclosed are merely illustrative and that other embodiments may take various alternative forms. The figures are not necessarily drawn to scale. Some features may be exaggerated or minimized to illustrate the details of certain components. Accordingly, certain structural and functional details disclosed herein should not be constrained, but rather should be interpreted merely as representative grounds to teach those skilled in the art how to employ the present disclosure in various ways.

[0023] As those skilled in the art will understand, various features shown and described with reference to any one of the drawings can be combined with features shown in one or more other drawings to create embodiments not expressly shown or described. The illustrated combinations of features provide representative embodiments for typical uses. However, various combinations and modifications of features consistent with the teachings of this disclosure may be desired for specific uses or practices.

[0024] In the following explanation, certain terms may be used for reference purposes only and are therefore not intended to be limiting. For example, terms such as “up” and “down” refer to directions within the referenced drawings. Terms such as “front,” “rear,” “forward,” “backward,” “left,” “right,” “rear,” and “side” describe the orientation and / or position of a component or element within an arbitrary reference frame, which will become clear by referring to the text and related drawings describing the component or element being discussed. Furthermore, terms such as “first,” “second,” and “third” may be used to describe separate components. Such terms may include the terms specifically mentioned above, their derivatives, and terms with similar meanings.

[0025] Referring to the drawings, similar reference numerals refer to similar components, and in Figure 1, the operating room 10 is shown as seen during a typical ophthalmic surgery, with a patient 11 on the operating table 12. The operating room 10 as contemplated herein includes a robotic camera system 14, the robotic camera system 14 itself including an articulated serial robot 16 and a digital camera 18. Within the scope of this disclosure, the robotic camera system 14 is controlled by the operation of an electronic control unit (ECU) 50, which is programmed in software and equipped in hardware, i.e., configured to execute computer-readable instructions that embody a calibration method 70, an example of which is described below with particular reference to Figure 3. By performing the method 70, the robotic camera system 14 can be calibrated in the manner described below.

[0026] The robot camera system 14 enables a user, in this non-limiting and exemplary example, a surgeon (not shown), to view a magnified image of a target object 19 at high magnification, and high-resolution visualization is facilitated by displaying the image on one or more high-resolution display screens 20. To that end, the method 70 described in detail below enables a simple automatic calibration process to be performed by the ECU 50 when connecting the digital camera 18 to a robot end effector 26 located at the distal end E1 of the serial robot 16, for example, a mounting plate, bracket, clamp, or other suitable mounting hardware.

[0027] The connection of the digital camera 18 to the end effector 26 does not necessarily result in perfect orthogonal alignment of the camera's optical axis and corresponding line-of-sight vector with respect to the robot motion coordinate system 25, which will be hereafter referred to as the robot coordinate system 25 for simplicity, as shown in Figure 1 as the nominal x, y, and z axes of a typical Cartesian reference coordinate system. In other words, the line-of-sight vector of the digital camera 18 along the optical axis may be slightly tilted due to adjustments made by the surgeon to the line-of-sight vector, variations in the camera's lenses, or alignment errors due to installation.

[0028] In applications utilizing an extended optical working distance (WD) between the digital camera 18 and the target object 19, unacceptably high levels of positional error can ultimately occur. To minimize such positional errors, the ECU 50 generates a transformation matrix (TF) 75 during the calibration phase of the robot camera system 14, and then uses the transformation matrix 75 to control the subsequent motion of the robot camera system 14. Such a transformation matrix 75 is derived without foreseeing or modeling the relevant parameters of the camera's optics. The corrected position is used by the ECU 50, either together with or in conjunction with the distributed motor control processor, when imaging the target object 19 during the subsequent operation of the robot camera system 14, and subsequently when controlling the motion of the serial robot 16.

[0029] As understood in the art, the digital camera 18 includes internally a set of optical image sensors (not shown) collectively configured to acquire and / or record incident light when forming a pixel image. Such image sensors in a possible stereoscopic embodiment may include a right optical image sensor and a left optical image sensor, respectively, for the right and left optical paths, and may include complementary metal-oxide-semiconductor ("CMOS") sensing elements, N-type metal-oxide-semiconductor ("NMOS"), semiconductor charge-coupled device ("CCD") sensing elements, or other devices suitable for various applications.

[0030] The digital camera 18 is located within the adjustable head unit 22 and captures digital image data of the target object 19 (arrow CC). IMG The head unit 22 may be configured to collect digital image data, which may be processed and filtered by the ECU 50 to generate a stereoscopic view of the live image of the target object 19. The toggle knob 23 may be on or mounted on the head unit 22 to allow the user to manually position the head unit 22, as well as to allow the user to adjust specific features of the digital camera 18, such as level magnification or focus.

[0031] The digital camera 18 is configured to acquire two-dimensional or three-dimensional images of the target object 19 in real time for presentation in various forms, including but not limited to captured still images, real-time images, and / or digital video signals. As used herein, “real-time” means updating information at the same or approximately the same rate as acquiring the data. More specifically, “real-time” means image data (arrow CC). IMG This means that when the image constructed from the image is finally displayed on the display screen 20, the displayed image appears to move smoothly, i.e., without any sudden vibrations or delays that the user would easily notice, because the image data is acquired, processed, and transmitted at a sufficiently high data transfer rate and with a sufficiently slow delay. For reference, a suitable typical data transfer rate is approximately 30 frames per second (30fps) or higher, displayed at approximately 60fps, with a delay of approximately 1 / 30th of a second or less.

[0032] A digital camera 18 whose optical axis is calibrated in accordance with this disclosure includes a lens assembly (not shown) having the optical working distance (WD) described above. If the optical working distance (WD) is variable within a set range, the focus motor 21 selectively moves one or more lenses of the lens assembly to adjust the working distance, which is the linear distance between the digital camera 18 and a reference plane in which the target object 19 is in focus, as understood in the art. In some embodiments, the optical working distance (WD) is adjustable by using the focus motor 21 to move the rear working distance lens relative to the front working distance lens, where “front” and “rear” refer to relative positions closer to the target object 19 and farther from the target object 19, respectively. The focus motor 21 can be variously embodied as an electric motor or another suitable rotary actuator, or as a linear actuator such as a stepping motor, a shape memory alloy actuator, or another actuator suitable for the application.

[0033] Referring further to Figure 1, the serial robot 16 includes a multi-joint robotic arm 24 operably connected to a head unit 22 and configured to selectively move the head unit 22, which is mechanically coupled to an end effector 26. The operator can position and orient the digital camera 18 by automatic position control of the robotic arm 24. In a typical embodiment, the robotic arm 24, as shown in Figure 1, includes a number of rotary joints 30 collectively configured to provide six degrees of freedom ("6-DoF") when positioning and / or orienting the head unit 22.

[0034] Sensory data from force sensors can be used by the ECU 50 to determine the angular position and adjustment speed of various joints 30 when assisting the movement of the digital camera 18. Each joint 30 may comprise one or more corresponding joint motors 31 and a corresponding joint position sensor 33. Each joint motor 31 is configured to rotate one of the corresponding joints 30 around its respective axis in the robot coordinate system 25, while the joint position sensor 33 transmits the measured angular position of each joint 30 to the ECU 50.

[0035] The robotic arm 24 is selectively operable to widen the field of view of the digital camera 18 along the x, y, and / or z axes of the robotic coordinate system 25. For example, the robotic arm 24 and the digital camera 18 coupled to the robotic arm 24 may be connected to a mobile carriage 34, which may be physically or remotely connected to a display screen 20 via an adjustable arm 40. The carriage 34 is constructed of a lightweight, easily disinfectable medical-grade material, such as painted aluminum or stainless steel, and may optionally be used to house the ECU 50 for the purpose of protecting the component hardware from the ingress of possible dust, debris, and moisture. In Figure 1, the display screen 20 supported by the adjustable arm 40 is shown in the form of a high-resolution / 4K or higher medical-grade monitor, but other embodiments may, non-limitingly, include, for example, a wall-mounted high-definition or ultra-high-definition television, smart glasses or another wearable monitor, a projector, or a computer screen.

[0036] Digital image data of the target object 19 collected by the operation of the digital camera 18 (arrow CC) IMG The signals are transmitted to the ECU 50 wirelessly or via a physical high-speed transmission conductor. The ECU 50 then performs digital image processing steps essential for constructing and displaying a high-resolution digital image. For example, the ECU 50 may combine or alternate video signals from the digital camera 18 to create a stereoscopic image. The ECU 50 may be configured to store video signals and / or stereoscopic video signals in video files within an associated computer-readable medium schematically represented in Figure 1 as memory (M) 54.

[0037] Furthermore, with respect to the ECU 50, this computer device is schematically shown in Figure 1 as a single box simply for the sake of clarity and brevity in the illustration. Actual embodiments of the ECU 50 may include one or more networked computer devices, each having one or more corresponding processors (P) 52 and a sufficient amount of the above-described memory 54, which includes a non-temporary (e.g., tangible) medium on which a set of computer-readable instructions readable and executable by the processors (52) is recorded or stored. The memory 54 can take many forms, including but not limited to non-volatile and volatile media. Instructions embodying the method 70 in Figure 3 are stored in the memory 54 and can be selectively executed by the processors 52 to perform various calibration functions as described below.

[0038] As those skilled in the art will understand, non-volatile media may include optical disks and / or magnetic disks or other persistent memory, while volatile media may include dynamic random access memory (DRAM), static RAM (SRAM), etc., and any or all of these may constitute the main memory of the ECU 50. Input / output ("I / O") circuits 56 may be used to facilitate connection to and communication with various peripheral devices, including a digital camera 18, a light source (not shown), and a high-resolution display screen 20. A graphical user interface (GUI) 29 allows a surgeon or clinician to issue control commands (arrow CC). 14 ) is input to enable movement of the serial robot 16, and measured joint angle signals (arrow CC) indicating the position of the serial robot 16 in free space are used. 30 It may be connected to the ECU 50 to receive signals from the digital camera 18, control the operation of the digital camera 18, and, in other ways, to interface with the ECU 50 and its various functions. Other hardware commonly used in the art, not shown, including but not limited to a local oscillator or fast clock, signal buffer, filter, and amplifier, may be included as part of the ECU 50.

[0039] According to this disclosure, the execution of Method 70 may require the ECU 50 in Figure 1 to implement a target lock mode, which is referred to herein as “Target Lock” or LTT mode, which allows the digital camera 18 to be positioned at any location within a defined workspace of the robot’s range of motion via the serial robot 16, and the position of the target object 19 is locked for motion tracking purposes, while the LTT mode allows changes in the optical working distance (WD) while maintaining a state of proper focus on this position. As is understood in the art, it can be difficult to maintain a state of focus on the image when moving the serial robot 16 to change the orientation of the digital camera 18 connected to the serial robot 16, i.e., when the direction of the line of sight vector changes. Therefore, in embodiments of the disclosure, the target lock mode / LTT capability of the ECU 50 allows the robot arm 24 to effectively operate as an extension of the surgeon by enabling the orientation of the digital camera 18 to be re-oriented while locked onto a specific target point.

[0040] Within this defined context, the ECU 50 is programmed with computer-readable instructions that embody method 70 in Figure 3, which are executed when calibrating the robot camera system 14 to precisely correlate the motion of the serial robot 16 with the resulting positional changes on the pixel image of the target object 19. The execution of method 70 ultimately results in the generation and recording of a transformation matrix 75, which is constructed itself as a homogeneous 4x4 matrix from five parameters, namely N1, N2, N3, N4, and N5.

[0041] According to this approach, parameters N1 and N2 correspond to the calculated roll offset and pitch offset, respectively, while parameters N3, N4, and N5 correspond to the x-axis offset, y-axis offset, and z-axis offset, respectively. Thus, the transformation matrix 75 can be realized as a 4×4 (16-element) homogeneous matrix having a linear term p=[x,y,z] and a rotation term R=R(approximately x-axis)*R(approximately y-axis). For the purposes of the disclosed solution in exemplary ophthalmic imaging applications, yaw can be ignored. The solution can be carried out in an alternative manner without accessing the analytical model of the optical system of the digital camera 18. Instead, the transformation matrix 75 is applied by the ECU 50 during the subsequent motion sequence of the robot camera system 14 to calculate and display the true positions of the target object 19 in the image plane and the target points on the target object 19. Thus, the corresponding pixel positions in the displayed image of the target object 19 correspond to the pixel positions in the robot coordinate system 25.

[0042] The problems resolved by this disclosure when controlling the digital camera 18 using an extended optical working distance (WD) can be understood by briefly referring to Figure 2, which illustrates the robot coordinate system 25 and the camera coordinate system 125 of the digital camera 18 shown in Figure 1. Origin P O,実際 This is within the LTT reference system, i.e., the camera coordinate system 125. The z-axis of the robot coordinate system 25 is typically assumed to be the line-of-sight vector 100 of the digital camera 18, which is true if the line-of-sight vector 100 is not tilted. Thus, Figure 2 illustrates the possible positional errors ("errors") for the original kinematic model for controlling the robot camera system 14 in Figure 1 when the line-of-sight vector 100 is actually tilted at an angle (θ), and this kinematic model typically assumes that the line-of-sight vector 100 is perpendicular or orthogonal to the head unit 22.

[0043] That is, for points A, B, and C in the robot coordinate system 25, the origin P O,実際Similarly, the resulting pitch and roll offsets can also be offset by a certain distance from the x, y, and z axes. In other words, coordinate systems 25 and 125 are not perfectly aligned with the underlying model, or to put it another way, the orthogonal relationship that is usually assumed is not exactly the case. As the optical working distance (WD) increases, the resulting positional error also increases. For example, an optical working distance of 300 mm and a tilt angle (θ) of only 0.5 degrees can lead to a positional error of 5 mm to 10 mm, accompanied by a corresponding display error in the presented image of the target object 19. Therefore, this solution seeks to determine angular offsets, as well as x, y, and z axis offsets, in order to minimize such positional errors when translating the camera coordinate system 125 to the robot coordinate system 25 for use in subsequent motion control operations.

[0044] As is understood in the art, the LTT control function of the ECU 50 is performed when controlling the movement of the robot arm 24 using the attached digital camera 18 shown in Figure 1. The LTT function allows the ECU 50 to move the robot arm 24 while maintaining the starting image at the center of the display screen 20. This is done by "locking" the optical working distance (WD) in Figure 1, and the resulting motion of the digital camera 18 is spherical, with a radius equal to the estimated focal length of the digital camera 18. While such an LTT function of the ECU 50 improves accuracy, the main cause of image errors when using the LTT function is that the position of the projection center (CoP) and / or focal point (FP) based on the line of sight vector 100, and consequently the position of the target object 19 in the robot coordinate system 25 in Figure 2, is still unknown with an acceptable level of accuracy.

[0045] This teaching can be implemented as computer executable instructions performed for the purpose of calibrating a robot camera system of the type shown in Figure 1, i.e., a robot camera system having a digital camera connected to the end effector of a 6-degree-of-freedom serial robot, where the end effector and digital camera move or move freely within a robot motion coordinate system. Generally, the solutions included within the scope of this disclosure proceed by acquiring one or more reference images of a target object located on an image plane having an optical coordinate system, i.e., a coordinate system of the digital camera that is contrasted with the coordinate system of the serial robot. Input signals are received in the form of depth measurements indicating the linear distance between the digital camera and the target object, and a set of joint position signals that collectively describe the position of the digital camera within the robot motion coordinate system. As described above, the robot motion coordinate system has nominal x, y, and z axes.

[0046] A non-limiting and exemplary embodiment of the method is shown in Figure 3 and described below. A preferred implementation of the method includes a logic step of determining the roll and pitch offsets of a target point in a reference image, where “offset” refers to the angular difference or delta with respect to the robot motion coordinate system. Such offsets are determined while the serial robot is moving in a first calibrated motion sequence, and the method then determines the x-axis offset, y-axis offset, and z-axis offset of the target point with respect to the robot motion coordinate system, respectively. This is done while the serial robot is moving in a second calibrated motion sequence. From there, the method proceeds by storing the angular and linear / axial offsets in a transformation matrix, which is later used to control a third motion sequence of the serial robot, i.e., during the subsequent operation of the robot camera system.

[0047] Referring to FIG. 3 in conjunction with FIG. 4 showing a microscope 17 including a digital camera 18, an embodiment of the method outlined above, i.e., method 70, begins at block B72 (“start”). Method 70 can be performed on a given robotic camera system 14 to capture and correct for camera-specific variations. Block B72 completes when a digital camera 18 is securely attached to an end effector 26 located at the distal end E1 of a serial robot 16. The method 70 then proceeds to block B74.

[0048] Block B74 (“target alignment”) includes obtaining an image of a reference target 48 using the digital camera 18, and for clarity, such an image is referred to herein as a “reference image”. As shown in FIG. 4, the reference target 48 can be a pixelated target figure 48P, e.g., a two-dimensional rectangular checkered pattern image having alternating black and white pixels 49 with a y-axis dimension (y OPT ) and an x-axis dimension (x OPT ). During initial target alignment, the ECU 50 moves the serial robot 16 to a default joint angle and then attempts to rotate the serial robot 16 so that the x-axis and y-axis of the reference image are aligned with the x-axis and y-axis of the reference target 48. Since only the x-axis and y-axis are aligned in block B74 and the resulting joint angles are stored in the memory 54, the reference image need not be centered.

[0049] As part of block B74, the position of the focus motor 21 of FIG. 1 is initialized at a calibrated focal length. The calibrated focal length represents the focal length at which the ECU 50 subsequently performs blocks B76, B78, and B80. For example, in a possible embodiment, the ECU 50 can set the focal length to 50% of a default quadratic working distance curve stored in the memory 54 itself or to about 0.35 m to 0.45 m. Thus, block B74 optionally includes an input signal (arrow CC INUsing this, the ECU50 itself communicates with the serial robot 16 in Figure 1, and the input signal (arrow CC) is used by the ECU50. IN This involves receiving the input signal (arrow CC). The depth measurement indicates the straight-line distance to the target object 19 along the optical axis (LL) of the digital camera 18, and the digital camera 18 is shown as an integral part of a microscope 17, for example, an ophthalmic microscope, in possible embodiments. IN ) also provides a joint position signal (arrow CC in Figure 1) indicating the position of the distal end E1 of the serial robot 16 in the robot coordinate system 25. 30 ) also includes (see Figures 1 and 2). Next, method 70 proceeds to block B76.

[0050] Next, in block B76 ("Angular Offset"), the ECU 50 determines the roll and pitch offsets of the reference point in the aforementioned reference image, and does so relative to the robot coordinate system 25. The roll and pitch offsets can be determined by moving the end effector 26 of the serial robot 16 upward or downward along the z-axis and by observing and recording the distance by which the x and y positions of the central image 148 of Figure 4 deviate from their initial state using the ECU 50. For example, the ECU 50 may use arctangent calculations to calculate the roll and pitch angles independently with respect to the x and y axes using the difference or delta between the reference image and the measured displacement. The measurement of displacement can be achieved using the LTT logic described above, as will be understood by those skilled in the art. The ECU 50 may selectively perform several iterations of block B76 to improve accuracy, for example, by repeating for roll and pitch parameters within a range of ±0.5 degrees.

[0051] Block B78 ("X, Y Offset") may involve rotating the digital camera 18 around its z-axis. When this is done, the reference image traces a circular trajectory due to the x and y offsets at the center of projection (CoP). Thus, the Ax=b matrix can be used by the ECU 50 to calculate the deltas in the x and y directions measured on the reference image, thereby enabling the ECU 50 to calculate the starting x and y positions. Therefore, block B78 includes determining the x and y offsets of the reference point relative to the robot coordinate system 25 using the ECU 50. The ECU 50 then stores the x and y offsets, along with the roll and pitch offsets, in the transformation matrix 75 of Figure 1, either in the ECU 50's memory 54 or in another accessible location.

[0052] Block B80 involves determining the z-axis offset after adjusting the x and y offsets of Block B78. Briefly referring to Figures 5A and 5B, the Z-axis offset is one of the possible solutions for [f,z] to align the correct target Z-axis position for a given focal length (f). The actual CoP is represented by point 64. Along the optical axis, the reference point 62 is located where the optical axis intersects the image plane 60. However, before calibration using Method 70, ECU50 determines that the CoP is actually point 66 (CoP 50 ) is located, and the true position of the target is point 65 (TGT 50 It may operate based on the mistaken understanding that ). Therefore, the actual focal length (f) between point 64 and point 62 is the focal length estimate (f) of ECU50. 50 Unlike ), the amount of difference along the z-axis is expressed as Δz.

[0053] In block B80, the ECU 50 can perform an experiment in which the serial robot 16 moves the digital camera 18 by a known angle (θ') within a spherical range of motion, as shown by arrow 67 in Figure 5B. The new corresponding positions of points 62, 64, and 66 are shown in Figure 5B as points 162, 164, and 166, respectively. Several such movements can be performed in sequence, and the average delta can be used in the following equation: Thus, the measurement error (Δx) img ) occurs within the image plane.

[0054] The solution using the z-axis as shown in Figure 6 is x img The objective is to measure the angle (θ') and calculate the z-axis distance (r) using the known angle (θ'). The ECU50 then sets Δx=0 in the robot coordinate system 25 as follows:

number

[0055] Block B82 of Method 70 shown in Figure 3 may involve creating a focus-to-optical working distance (F vs WD) table in the memory 54 of the ECU 50. This operation can be achieved offline using values ​​extracted from the table, which will be used during subsequent operation of the robot camera system 14 after the adjustments of roll, pitch, and x, y, x-axis offsets described above. For example, the ECU 50 may be initialized with the calibrated focal length and then data may be added to the table. In a possible embodiment, the serial robot 16, at the direction of the ECU 50 or other preferred processing and control hardware, moves the digital camera 18 along the line-of-sight vector 100 in Figure 2 toward each limit by the amount to be calibrated, and then stores the corresponding table entry for encoder position or "count", optical working distance (WD) in meters, in the lookup table. The curve 90 of such data shown in Figure 8 represents the corresponding z-axis offset for a given optical working distance (WD). In other words, the ECU 50 generates the illustrated curve 90 using the position based on the count of the focus motor 21 (see Figure 1) and the corresponding optical working distance (WD).

[0056] When locking onto a target object 19 to perform a given LTT operation, the robot arm 24 in Figure 1 is given the distance from the CoP to the target position along the line of sight vector 100. This value can be obtained by inputting the raw encoder position of the focus motor 21. Thus, a key final result of this calibration method 70 is that positional errors are minimized by using the transformation matrix 75 when calculating the position of the target object 19. In a practical sense, each time the ECU 50 enters LTT mode, it calculates the position in the corresponding LTT control coordinate system and causes the digital camera 18 to consult an optical working distance table, e.g., the curve 90 in Figure 8, to determine the optical working distance (WD). The ECU 50 then generates the line of sight vector 100. The endpoint of the line of sight vector 100 is the target position, and the positional error of the target position is ultimately minimized by performing method 70. When considering a single optical working distance (WD), as in the case of a microscope 17 with a constant optical working distance, the ECU 50 still determines a single z-axis offset to minimize errors at that particular optical working distance. Therefore, a functional equivalent of such a technique is the creation of a lookup table or memory location with a single Z-offset entry.

[0057] Next, method 70 in Figure 3 follows block B84, in which the ECU 50 controls the motion sequence of the serial robot 16 during the subsequent operation of the robot camera system 14 using the transformation matrix 75 in Figure 1. Thus, when the serial robot 16 moves in response to a command from the ECU 50, which may be autonomously generated or commanded by a surgeon, the motion control logic of the ECU 50 recognizes the response of the incremental motion of the serial robot 16 in the robot coordinate system 25 to the camera coordinate system 125, i.e., on the image plane. Thus, a surgeon who wishes to visualize a specific point on a target object 19, for example, a point on the cornea or lens of a patient during ophthalmic surgery, is given a high degree of confidence that the target point will appear exactly where expected on the displayed digital image.

[0058] As will be understood by those skilled in the art in light of the foregoing disclosure, the calibration process or its logical modification enabled by the implementation of Method 70 is intended to compensate for slight variations between the expected alignment of the robot coordinate system 25 and the camera coordinate system 125. Whether due to adjustments to the line of sight vector by the surgeon, tolerances when mounting the digital camera 18 to the end effector 26 in Figure 1, or other factors, a given camera will not perfectly match the expected alignment in a given operating room 10. Thus, the calibration attempt described above is made on the connection between the given camera and the robot. Once the calibration step of Method 70 is complete, Method 70 can proceed to block B84, in which the surgeon or clinician can enjoy the benefit of minimizing positional errors during the subsequent operation of the robot camera system 14. These and other potential benefits will be readily understood by those skilled in the art in light of the foregoing disclosure.

[0059] While the detailed description and drawings support and illustrate this disclosure, the scope of this disclosure is defined solely by the claims. Although several best modes and other embodiments for carrying out the disclosure described in the claims have been described in detail, various alternative designs and embodiments exist for carrying out the disclosure as defined in the appended claims. Furthermore, the features of the embodiments shown in the drawings or the various embodiments referred to herein should not necessarily be understood as independent embodiments of each other. Rather, each of the characteristics described in one example of an embodiment can be combined with one or more other desirable characteristics from other embodiments, resulting in other embodiments that are not described in words or not described by reference to the drawings. Thus, such other embodiments are included within the framework of the appended claims.

Claims

1. A method for calibrating a robotic camera system having a digital camera connected to the end effector of a serial robot, wherein the end effector and the digital camera move within a robotic motion coordinate system, the digital camera is integrated with an ophthalmic microscope, and the method is The electronic control unit (ECU) communicating with the serial robot and the digital camera acquires a reference image of a target object in an image plane having an optical coordinate system using the digital camera, wherein the target object is to be used for calibration of the robot camera system before use on a patient. The ECU receives an input signal, the input signal includes a depth measurement indicating the straight-line distance between the digital camera and the target object, and a set of joint position signals collectively describing the position of the digital camera in the robot motion coordinate system, the robot motion coordinate system having x, y, and z axes, and the receiving of the signal. While the serial robot is moving in the first calibration motion sequence, the ECU determines the roll offset and pitch offset of the target point in the reference image relative to the robot motion coordinate system, After determining the roll offset and the pitch offset, while the serial robot is moving in the second calibration motion sequence, the ECU determines the x-axis offset, y-axis offset, and z-axis offset of the target point relative to the robot motion coordinate system. The ECU adds a z-axis offset indexed by a variable optical working distance and the rotational position of the focus motor or an encoder count to a lookup table, wherein the variable optical working distance is the distance between the digital camera and the image plane and is controlled by the focus motor. The ECU stores the roll offset, pitch offset, x-axis offset, y-axis offset, and z-axis offset in a transformation matrix in the ECU's memory. The ECU controls a third motion sequence of the serial robot during the subsequent operation of the robot camera system using the transformation matrix and the z-axis offset extracted from the lookup table based on the rotational position of the focus motor or encoder count that occurs during the subsequent operation. Methods that include...

2. The ECU receives the autofocus setting of the digital camera, The ECU processes the autofocus settings of the robot camera system and determines the depth measurement value, The method according to claim 1, further comprising:

3. The ECU measures the depth measurement using a depth sensor, The ECU measures the joint position signal using a corresponding set of joint position sensors of the serial robot, The method according to claim 1, further comprising:

4. The method according to claim 1, wherein acquiring the reference image of the target object includes the ECU collecting a digital image of a two-dimensional checkerboard pattern using the digital camera.

5. The method according to claim 1, further comprising the ECU displaying a three-dimensional image of the target object through one or more display screens during the subsequent operation.

6. It is a robotic camera system, A digital camera connectable to the end effector of a serial robot, wherein the end effector and the digital camera move within the robot's motion coordinate system, An ophthalmic microscope integrated with the aforementioned digital camera, An electronic control unit (ECU) that communicates with the aforementioned digital camera, Acquiring a reference image of a target object in an image plane having an optical coordinate system using the digital camera, wherein the target object is used for calibration of the robotic camera system before use on a patient. The receiving of an input signal includes a depth measurement indicating the straight-line distance between the digital camera and the target object, and a set of joint position signals that collectively describe the position of the digital camera in the robot motion coordinate system, wherein the robot motion coordinate system has x, y, and z axes. While the serial robot is moving in the first calibration motion sequence, the roll offset and pitch offset of the target point in the reference image relative to the robot motion coordinate system are determined. After determining the roll offset and pitch offset, while the serial robot is moving in the second calibration motion sequence, the x-axis offset, y-axis offset, and z-axis offset of the target point relative to the robot motion coordinate system are determined. The roll offset, pitch offset, x-axis offset, y-axis offset, and z-axis offset are stored in a transformation matrix in the memory of the ECU. The control of a third motion sequence of the serial robot using the transformation matrix during the subsequent operation of the robot camera system, wherein the subsequent operation of the robot camera system is performed as part of an ophthalmic surgery, and the target object whose image is acquired by the digital camera during the subsequent operation is the eye of a human patient. The electronic control unit (ECU) is configured to perform the following: A robotic camera system, including...

7. The robot camera system according to claim 6, further comprising the serial robot.

8. The robotic camera system according to claim 6, wherein the digital camera includes a focus motor and has a variable optical working distance between the digital camera and the image plane controlled by the focus motor, and the ECU is configured to extract the z-axis offset from a lookup table indexed by the variable optical working distance and the rotational position of the focus motor or an encoder count.

9. The robotic camera system according to claim 8, wherein the ECU is configured to add data to the lookup table while controlling the focus motor over a focal range corresponding to the variable optical working distance.

10. The robot camera system according to claim 6, wherein the ECU is configured to receive the autofocus setting of the digital camera and to determine the depth measurement using the autofocus setting.

11. The robotic camera system according to claim 6, further comprising a depth sensor operable to determine the depth measurement value.

12. The robot camera system according to claim 6, wherein the ECU is configured to acquire the reference image of the target object by collecting a digital image of a two-dimensional checkerboard pattern using the digital camera.

13. The robot camera system according to claim 6, further comprising one or more display screens, wherein the ECU is configured to display a three-dimensional image of the target object through the one or more display screens during the subsequent operation.

14. A computer-readable medium on which instructions are recorded, and when used with a robotic camera system of an ophthalmic microscope integrated with the digital camera, where the digital camera is connected to the end effector of a serial robot, the execution of the instructions by the processor causes the processor to: Acquiring a reference image on the image plane having an optical coordinate system from the digital camera, The receiving of an input signal includes a depth measurement indicating the straight-line distance between the digital camera and the patient's eye, and a set of joint position signals that collectively describe the position of the digital camera in a robot motion coordinate system, wherein the robot motion coordinate system has x, y, and z axes. While the serial robot is moving in the first calibration motion sequence, the roll offset and pitch offset of the target point in the reference image relative to the robot motion coordinate system are determined. After determining the roll offset and pitch offset, while the serial robot is moving in the second calibration motion sequence, the x-axis offset, y-axis offset, and z-axis offset of the target point relative to the robot motion coordinate system are determined. The roll offset, pitch offset, x-axis offset, y-axis offset, and z-axis offset are stored in a conversion matrix within the computer-readable medium, thereby calibrating the robot camera system having the digital camera. Controlling a third motion sequence of the serial robot using the transformation matrix during the subsequent operation of the robot camera system, wherein the subsequent operation of the robot camera system is performed as part of an ophthalmic surgery by acquiring an image of the patient's eye, and the control is as follows: A computer-readable medium that performs [something].

15. The digital camera is a stereoscopic camera, and the execution of the instructions by the processor, The computer-readable medium according to claim 14, wherein the processor displays a three-dimensional image of the patient's eye through one or more display screens during the subsequent operation.

16. The digital camera has a variable optical working distance between the digital camera and the image plane controlled by the focus motor, and the processor executes the instructions, Determining the rotational position or encoder count of the focus motor, A lookup table to which data has been added by the z-axis offset, wherein the z-axis offset is extracted from the lookup table indexed by the variable optical working distance and the rotation position of the focus motor or encoder count. A computer-readable medium according to claim 14, which performs the following.

Images