Method and system for calibrating a camera

Abstract

A method performed by a surgical system including a camera in an operating room. The system receives a first image from a first camera representing a first FOV with an object at a first location. The system receives a second image from a tracking camera with the object at the first location and determines a first pose of the first camera based on the first and second images. The system receives a third image captured by a second camera representing a second, different FOV and with the object at a second location. The system receives a fourth image captured by the tracking camera with the object at the second location and determines a second pose of the second camera based on the third and fourth images. The system determines a relative spatial transformation between the first camera and the second camera based on the first pose and the second pose.

Description

[Technical Field]

[0001] Various embodiments of the present disclosure relate generally to surgical systems and, more particularly, to surgical systems that calibrate one or more cameras. [Background technology]

[0002] Minimally invasive surgery (MIS), such as laparoscopic procedures, uses techniques aimed at reducing tissue damage during surgical procedures. Laparoscopic procedures typically require making multiple small incisions in a patient, for example, in their abdomen, through which multiple surgical tools, such as endoscopes, blades, graspers, and needles, are then inserted. Gas is insufflated into the abdomen, which inflates the abdomen, thereby providing more space around the tool tips and making it easier for the surgeon to view and manipulate tissue at the surgical site (via the endoscope). MIS can be performed faster and with less surgeon fatigue using a surgical robotic system in which surgical tools are operably attached to the distal end of a robotic arm and a control system actuates the arm and its attached tools. The tool tips mimic the movement of the position and orientation of a handheld user input device (UID) when the UID is manipulated by the surgeon. A surgical robotic system may have multiple surgical arms, one or more of which have an attached endoscope and other of which have attached surgical instruments for performing a particular surgical operation.

[0003] Control inputs from a user (e.g., a surgeon or other operator) are captured via one or more user input devices and then translated into control of the robotic system. For example, in response to user commands, a tool drive having one or more motors may actuate one or more degrees of freedom of a surgical tool when the surgical tool is positioned at a surgical site on a patient. Summary of the Invention [Means for solving the problem]

[0004] An increasing number of applications benefit from the use of images, e.g., video captured by multiple image sensors, such as video cameras and depth cameras. Examples of such applications may include telepresence, data collection for various computer training purposes (e.g., training the motion controller of an autonomous vehicle), and event detection during surgical procedures in operating rooms. For example, multiple video cameras may be used within a hospital facility, such as an operating room, to monitor events and surgical workflow during a surgical procedure (e.g., monitoring surgical tasks performed by a surgeon). Cameras within such settings may be used to monitor activity, detect events of interest, and analyze efficiency for training and improvement purposes. For example, in addition to detecting people or objects entering a room or space being observed by the cameras, a system may be capable of tracking the movement of an object through multiple cameras, such as a wheeled gurney being moved from room to room. As a result of maximizing the observable range of such a large room, some cameras may not have overlapping fields of view (FOV); one camera may be pointed toward a door and another camera may be pointed away from the door.

[0005] When collecting video data from multiple cameras, it may be desirable to correlate their spatial information (e.g., their position and orientation in space relative to each other) to better enable analysis of the video data to perform actions such as detecting activities or events. For example, when detecting the movement of an object, such as a patient bed or gurney in an operating room, the object may move from the FOV of one camera into the FOV of another camera. The spatial information indicating the relationship between the two cameras is used to maintain consistent tracking of the object as it moves between FOVs. This provides meaningful inferences and analysis of the detected events.

[0006] To correlate spatial information, a camera may be spatially calibrated by calculating the camera's intrinsic and extrinsic parameters (or characteristics). Intrinsic parameters may include characteristics associated with a particular camera, such as the camera's physical attributes. Examples of intrinsic characteristics may include the focal length of the camera (e.g., the camera's lens), the principal point (or optical center) of the camera (e.g., the camera's lens), the camera's skew, and the camera's lens distortion. These intrinsic characteristics may represent a transformation from a two-dimensional (2D) image coordinate system (e.g., pixel coordinates in a 2D captured image) to a three-dimensional (3D) coordinate system in the camera's 3D space (relative to the camera as the origin). Extrinsic parameters may represent (or include) a transformation (e.g., position and / or orientation) from the camera's 3D coordinates to a global (or world) coordinate system. In particular, the extrinsic parameters may represent the relative spatial transformation (or pose) between two cameras. The process of estimating the pose of a camera relative to another camera may be performed through spatial extrinsic calibration. Conventional calibration methods require multiple cameras to have at least partially overlapping FOVs, where the overlap includes a common reference point. However, these methods may not be able to accurately calibrate cameras with non-overlapping fields of view, particularly cameras that are part of a static (e.g., non-movable) image sensor setup. Therefore, there is a need for a surgical system configured to calibrate image sensors with both overlapping and non-overlapping FOVs (e.g., static).

[0007] The present disclosure provides a surgical system including a first camera, a second camera, and a tracking camera located within one or more operating rooms and configured to perform spatial calibration between the cameras. For example, the two cameras may be located within the room (e.g., mounted on separate walls) and may be used for event detection within the operating room. Specifically, the system receives a first image captured by the first camera, the first image representing a first FOV of the first camera with an object, such as a calibration pattern, at a first location within the operating room. The system receives a second image captured by the tracking camera, the second image representing the object at the first location, and determines a first pose of the first camera based on the first and second images. Specifically, the pose of the first camera may be relative to the tracking camera such that the position and / or orientation of the first camera is within the coordinate system of the tracking camera. The system receives a third image captured by the second camera, the third image representing a second FOV of the second camera that does not overlap with the first FOV and has an object at a second location within the operating room. The system receives a fourth image captured by the tracking camera, the fourth image having an object at a second location, and determines a second pose of the second camera based on the third and fourth images. Therefore, the poses of both the first and second cameras are in the coordinate system of the tracking camera. The system determines a relative spatial transformation, e.g., pose, between the first and second cameras based on the first and second poses. As a result, the system knows the relative spatial information between the two cameras, which allows the system to effectively track movement through all observations between the cameras, even when the cameras are configured to cover different room locations and have non-overlapping FOVs.

[0008] In one embodiment, the tracking camera includes a third FOV that includes both the first and second locations. For example, the tracking camera may be positioned on the ceiling of the operating room and may include a wide-angle lens to view the interior of the operating room. In one embodiment, the tracking camera remains stationary while the object is moved from the first location to the second location. In another embodiment, a third image and a fourth image are captured by the second camera and the tracking camera, respectively, before the first image and the second image are captured by the first camera and the tracking camera, respectively. In one embodiment, the first and second images are captured simultaneously by the first camera and the tracking camera, respectively, and the third and second images are captured simultaneously by the third camera and the tracking camera, respectively.

[0009] In one embodiment, the object remains stationary at the first location when the first and second images are captured by the first and tracking cameras, respectively, and the object remains stationary at the second location when the third and fourth images are captured by the third and tracking cameras, respectively. In another embodiment, the object is not attached to any of the first, second, or tracking cameras.

[0010] In one embodiment, determining a first pose of the first camera based on the first image and the second image includes determining a third pose of the first camera relative to the object at the first location using the first image and determining a fourth pose of the tracked camera relative to the object at the first location using the second image. In another embodiment, the relative spatial transformation indicates a position and orientation of the second camera relative to the first camera. In one embodiment, the object comprises a calibration pattern, and the calibration pattern is moved from the first location to the second location by a user in the operating room while the first camera, the second camera, and the tracked camera remain in fixed positions.

[0011] The present disclosure provides a surgical system including a first camera, a second camera, and (e.g., tracked) markers within an operating room for calibrating the two cameras. Specifically, the system receives a first image captured by the first camera, the first image representing a first FOV of the first camera including a marker at a first location within the operating room, and determines a first pose of the first camera based on the first image and the first location of the marker. The system receives a second image captured by a second camera, the second image representing a second FOV of the second camera that does not overlap with the first FOV and includes a marker at a second location within the operating room, and determines a second pose of the second camera based on the second image and the second location of the marker. The system determines a relative spatial transformation between the first camera and the second camera based on the first and second poses, and the first camera and the second camera remain stationary as the marker is moved from the first location to the second location.

[0012] In one embodiment, the marker is fixedly coupled to an object including a calibration pattern, and the object is at a first location captured in the first image and the second image. The system uses a tracking device to determine a third location where the marker is fixedly coupled to the calibration object, and determining the first pose of the first camera includes determining a third pose of the first camera relative to the calibration pattern while the object is at the first location and determining a fourth pose of the tracking device relative to the calibration pattern using the third location of the marker. In another embodiment, determining the fourth pose of the tracking device includes determining a fifth pose of the tracking device relative to the marker according to the third location of the marker, retrieving a sixth pose of the marker relative to the calibration pattern (e.g., from a memory of the surgical system), and adjusting the fifth pose based on the sixth pose. In one embodiment, the marker and the calibration pattern are a single integrated unit. In another embodiment, the tracking device is an infrared sensor, and the marker is an infrared tag. In another embodiment, the tracking device is a camera, and the marker is a visible pattern.

[0013] The present disclosure provides a surgical system including a first camera and a second camera in an operating room, the system receiving a first image captured by the first camera, the first image representing a first FOV including an object at a first location in the operating room, and determining a first pose of the first camera based on the first image and tracking markers in the operating room, such as those disposed on the ceiling of the operating room. The system receiving a second image captured by a second camera, the second image representing a second FOV of the second camera that does not overlap with the first FOV and includes an object at a second location in the operating room, and determining a second pose of the second camera based on the second image and the tracking markers. The system determines a relative spatial transformation between the first camera and the second camera based on the first pose and the second pose.

[0014] In one embodiment, the object includes a tracking device, the method includes using the tracking device to detect a tracking marker while the object is at a first location, a first pose being determined based on the detection of the tracking marker while the object is at the first location, and using the tracking device to detect the tracking marker while the object is at a second location, a second pose being determined based on the detection of the tracking marker while the object is at the second location.

[0015] In one embodiment, the method includes tracking movement of an object from a first location to a second location using a tracking device, wherein a second pose is determined based on the tracked movement of the object. In another embodiment, the tracking device and the object are an integrated unit. In another embodiment, the system further determines a third pose of the tracking marker relative to the object based on detection of the tracking marker while the object is at the first location, and determines a fourth pose of the first camera relative to the object based on the first image, wherein the first pose of the first camera is based on the third pose and the fourth pose. In one embodiment, determining the third pose of the tracking marker includes determining a fifth pose of the tracking marker relative to the tracking device based on detection of the tracking marker, receiving a sixth pose of the tracking device relative to the object, and adjusting the fifth pose according to the sixth pose.

[0016] In one embodiment, the tracking device is a tracking camera, and detecting the tracking marker while the object is at the first location includes receiving a third image captured by the tracking camera, the third image representing a third FOV of the tracking camera and including the tracking marker. In one embodiment, the tracking marker is an infrared (IR) tag, and the tracking device is an IR sensor. In another embodiment, the tracking marker is a radio frequency (RF) tag, and the tracking device is an RF sensor. In one embodiment, the object comprises a calibration pattern, and the calibration pattern is moved from the first location to the second location by a user or an autonomous robot in the operating room, while the first camera and the second camera remain in a fixed position.

[0017] The above summary does not contain an exhaustive list of all embodiments of the present disclosure. The present disclosure is contemplated to include all systems and methods that may be practiced from all suitable combinations of the various embodiments summarized above, as well as those disclosed in the following detailed description and particularly pointed out in the claims. Such combinations may have certain advantages not specifically described in the above summary. [Brief explanation of the drawings]

[0018] Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which like reference numerals refer to like elements. It should be noted that references to "an" or "one" embodiment of the present disclosure do not necessarily refer to the same embodiment, but rather to at least one. Also, for the sake of brevity and reducing the total number of figures, a given figure may be used to illustrate features of more than one embodiment, and not all elements in a figure may be required for a given embodiment. [Figure 1] 1 shows a pictorial view of an exemplary surgical system within a surgical field. [Figure 2] FIG. 1 is a block diagram of a surgical system for calibrating one or more cameras, according to one embodiment. [Figure 3]1 is a flowchart of a process for one embodiment of calibrating one or more cameras using a tracking device. [Figure 4] shows several stages illustrating the calibration of two cameras using a tracking device. [Figure 5] 1 is a flowchart of a process for one embodiment of calibrating one or more cameras using tracking devices and tracking markers. [Figure 6] 1 shows several steps illustrating the calibration of two cameras using tracking devices and tracking markers. [Figure 7] 1 is a flowchart of a process for one embodiment of calibrating one or more cameras using tracking devices and tracking markers. [Figure 8] 1 shows several steps illustrating the calibration of two cameras using tracking devices and tracking markers. DETAILED DESCRIPTION OF THE INVENTION

[0019] Some embodiments of the present disclosure are described with reference to the accompanying drawings. Whenever the shape, relative position, and other aspects of parts described in a given embodiment are not explicitly defined, the scope of the disclosure herein is not limited to the parts shown, which are intended solely for illustrative purposes. Also, while numerous details are described, it is understood that some embodiments can be practiced without these details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description. Furthermore, unless the meaning clearly indicates otherwise, all ranges described herein are intended to include the endpoints of each range.

[0020] In one embodiment, "pose" may refer to the relative position and / or orientation of one object or frame of reference with respect to another object or frame of reference. In particular, pose may be a six-degrees-of-freedom (6DOF) variable that indicates the three-dimensional (3D) location and orientation of an object relative to another object in a 3D space or frame of reference. For example, location may include three parameters, such as the X, Y, and Z coordinates of a 3D Cartesian coordinate system, and orientation may include three parameters, such as a set of three rotation angles about three axes of the coordinate system (e.g., the yaw, pitch, and roll Euler angles). In one embodiment, pose may be a relative spatial transformation, which may include a transformation matrix P, which may include a translation vector indicating the object's position with respect to the origin of the frame of reference in space and / or a rotation matrix indicating the object's orientation with respect to the frame of reference.

[0021] FIG. 1 shows a pictorial view of an exemplary (e.g., laparoscopic) surgical system (hereinafter, may be referred to as the “system”) 1 within a surgical field (or room). System 1 includes a user console 2, a control tower 3, and one or more surgical robotic arms 4 on a surgical robotic table (surgical table or surgical platform) 5. In one embodiment, the arms 4 may be attached to a table or bed on which a patient lies, as shown in the example of FIG. 1. In one embodiment, at least some of the arms 4 may be configured differently. For example, at least some of the arms may be mounted to another suitable structural support, such as a ceiling, a sidewall, or a cart separate from the table. System 1 may incorporate any number of devices, tools, or accessories used to perform surgery on patient 6. For example, system 1 may include one or more surgical tools (instruments) 7 used to perform the surgery (surgical procedure). The surgical tools 7 may be end effectors for performing the surgical procedure attached to the distal ends of the surgical arms 4.

[0022] Each surgical tool 7 may be manipulated manually, robotically, or both during surgery. For example, surgical tool 7 may be a tool used to enter, view, perform a surgical task, or manipulate the internal anatomical structures of patient 6. In one embodiment, surgical tool 7 is a grasper capable of grasping patient tissue. Surgical tool 7 may be controlled manually by a bedside operator 8 or robotically via actuated movement of a surgical robotic arm 4 to which the surgical tool is attached. For example, when manually controlled, an operator may (e.g., physically) hold a portion of the tool (e.g., a handle) and manually control the tool by moving the handle and / or pressing one or more input controls (e.g., buttons) on the tool (e.g., the tool's handle). In another embodiment, when robotically controlled, the surgical system can manipulate the surgical tool based on user input (e.g., received via user console 2 as described herein).

[0023] Generally, a remote operator 9, such as a surgeon or other operator, may use the user console 2 to remotely operate the arm 4 and / or attached surgical tool 7, e.g., during teleoperation. The user console 2 may be located in the same operating room as the rest of the system 1, as shown in FIG. 1 . However, in other environments, the user console 2 may be located in an adjacent or nearby room, or in a remote location, e.g., a different building, city, or country. The user console 2 may include one or more components, such as a seat 10, one or more foot-operated controls (or foot pedals) 13, one or more (handheld) user-input devices (UIDs) 14, and at least one display 15. The display may be configured to display, for example, a view of a surgical site within the patient 6. The display may be configured to display image data (e.g., still images and / or video). In one embodiment, the display may be any type of display, such as a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic LED (OLED) display, a head-mounted display (HMD), or the like. In some embodiments, the display may be a 3D immersive display for displaying a 3D (surgical) presentation. For example, during a surgical procedure, one or more endoscopes (e.g., endoscopic cameras) may capture image data of a surgical site that the display presents to the user in 3D. In one embodiment, the 3D display may be an autostereoscopic display that provides a 3D perception to the user without the need for special glasses. As another example, the 3D display may be a stereoscopic display that uses glasses (e.g., via active shutters or polarization) to provide a 3D perception.

[0024] In another embodiment, display 15 may be configured to display at least one graphical user interface (GUI) that may provide informative and / or interactive content to assist a user in performing a surgical procedure with one or more instruments in surgical system 1. For example, some of the displayed content may include image data captured by one or more endoscopic cameras, as described herein. In another embodiment, the GUI may include selectable UI items that, when manipulated by a user, cause the system to perform one or more actions. For example, the GUI may include UI items as interactive content for switching control between robotic arms. In one embodiment, the system may include input devices such as a keyboard, mouse, etc., to interact with the GUI. In another embodiment, a user may interact with the GUI using UID 14. For example, a user may manipulate the UID (e.g., with a cursor) to navigate through the GUI, or may hover the cursor over a UI item and manipulate the UID (e.g., select a control or button) to make a selection. In some embodiments, the display may be a touch-sensitive display screen. In this case, a user may make a selection by navigating and selecting through touching the display. In some embodiments, any method may be used to navigate and / or select UI items.

[0025] As shown, a remote operator 9 sits in a seat 10 and views a user display 15 while operating a foot-operated control 13 and a handheld UID 14 to remotely control one or more of the arm 4 and surgical tools 7 (mounted at the distal end of the arm 4).

[0026] In some embodiments, the bedside operator 8 can also operate the system 1 in an "over-the-bed" mode, in which a bystander (user) is currently next to the patient 6 and simultaneously operates a robotically driven tool (an end effector attached to the arm 4) with, for example, a handheld UID 14 held in one hand, and a manual laparoscopic tool. For example, the bedside operator's left hand may be operating the handheld UID to control the robotic components, while the bedside operator's right hand may be operating the manual laparoscopic tool. Thus, in these variations, the bedside operator may perform both robotically assisted minimally invasive surgery and manual laparoscopic procedures on the patient 6.

[0027] During an exemplary procedure (surgery), patient 6 is sterilely prepped and draped to achieve anesthesia. Initial access to the surgical site may be performed manually while the arms of system 1 are in a stowed or retracted configuration (thereby facilitating access to the surgical site). Once access is complete, initial positioning or preparation of system 1, including its arms 4, may be performed. A remote operator 9 at user console 2 then proceeds with the procedure by utilizing foot controls 13 and UID 14 to operate various end effectors and possibly an imaging system to perform the procedure. Manual assistance may also be provided at the procedure bed or table by sterile-gowned bedside personnel, such as bedside operator 8, who may perform tasks such as retracting tissue, performing manual repositioning, and replacing one or more tools on robotic arm 4. Non-sterile personnel may also be present to assist remote operator 9 at user console 2. When the procedure or surgery is completed, the system 1 and user console 2 may be configured or set to a state to facilitate post-operative procedures such as cleaning or sterilization and entering or printing medical records via the user console 2.

[0028] In one embodiment, the remote operator 9 holds and moves the UID 14 and provides input commands to actuate (move) one or more robotic arm actuators 17 (or drive mechanisms) within the system 1 for remote operation. The UID 14 may be communicatively coupled to the rest of the system 1, for example, via a console computer system 16 (or host). The UID 14 may generate spatial state signals corresponding to the movements of the UID 14, e.g., the position and orientation of the UID's handheld housing, which may be input signals for controlling the movement of the robotic arm actuators 17. The system 1 may use control signals derived from the spatial state signals to control the proportional movement of the actuators 17. In one embodiment, a console processor of the console computer system 16 receives the spatial state signals and generates corresponding control signals. Based on these control signals, which control how the actuators 17 are energized to drive segments or links of the arm 4, the movement of a corresponding surgical tool attached to the arm may mimic the movement of the UID 14. Similarly, interaction between the remote operator 9 and the UID 14 may generate a grasp control signal that, for example, causes the jaws of the graspers of the surgical tool 7 to close and grasp tissue of the patient 6 .

[0029] System 1 may include multiple UIDs 14, each generating a respective control signal for controlling the actuators and surgical tools (end effectors) of its respective arm 4. For example, a remote operator 9 may move a first UID 14 to control the movement of an actuator 17 in a left robotic arm, which responds by operating a linkage, gear, or the like in that arm 4. Similarly, movement of a second UID 14 by the remote operator 9 controls the movement of another actuator 17, which in turn drives other linkages, gears, or the like in system 1. System 1 may include a right arm 4 on the right side of a patient secured to a bed or table, and a left arm 4 on the left side of the patient. The actuators 17 may include one or more motors that are controlled to drive the rotation of a joint in the arm 4 to, for example, change the orientation of an endoscope or grasper of a surgical tool 7 attached to that arm relative to the patient. The movement of multiple actuators 17 in the same arm 4 may be controlled by spatial state signals generated from a particular UID 14. The UIDs 14 may also control the movement of the respective surgical tool graspers. For example, each UID 14 may generate a respective grasp signal to control the movement of an actuator, e.g., a linear actuator, that opens and closes the jaws of the grasper at the distal end of the surgical tool 7 to grasp tissue within the patient 6.

[0030] In some embodiments, communication between the surgical robot table 5 and the user console 2 may be via a control tower 3, which may translate user commands received from the user console 2 (more specifically, from the console computer system 16) into robotic control commands that are sent to the arm 4 on the surgical table 5. The control tower 3 may also transmit status and feedback from the surgical table 5 to the user console 2. The communication connections between the surgical table 5, the user console 2, and the control tower 3 may be via wired (e.g., fiber optic) and / or wireless links using any suitable of various wireless data communication protocols, for example, the BLUETOOTH protocol. Any wired connections may optionally be integrated into the floor and / or walls or ceiling of the operating room. The system 1 may provide video output to one or more displays, including displays in the operating room and remote displays accessible via the Internet or other network. The video output or feed may also be encrypted to ensure privacy, and all or portions of the video output may be stored on a server or electronic medical record system.

[0031] 2 is a block diagram of a surgical system 1 for calibrating one or more cameras, according to one embodiment. The system includes one or more (e.g., electronic) components or elements, such as a controller 20, a sensor 21, a tracking device 22, a calibration object 23, a first camera 26, a second camera 27, a display 28, a speaker 29, and a memory 90. In one embodiment, the system may include more or fewer elements, such as having one or more (different) sensors and / or not including a calibration object, a speaker, or a display. In another embodiment, at least some of the system's elements may be optional, such as tracking device 22, tracking marker 92, and / or tracking device 25. In one embodiment, the system may include one or more other elements not shown, such as having one or more robotic arms 4, as shown in FIG. 1.

[0032] In some embodiments, at least some of the elements may be part of a single electronic device. For example, controller 20 and memory 90 may be housed within control tower 3 shown in FIG. 1 . In other embodiments, at least some of the elements may be separate electronic devices or parts of separate electronic devices relative to one another. For example, tracking device 22 may be a separate electronic device located within, e.g., attached to, the operating room in which at least a portion of surgical system 1 is located.

[0033] In one embodiment, elements of the surgical system may be communicatively coupled to the controller 20 and / or to one another to exchange digital data. For example, the controller may be configured to communicate wirelessly with one or more elements, such as the tracking device 22, over a network and through a wireless connection. In one embodiment, the devices may communicate over any (computer) network, such as a wide area network (WAN), e.g., the Internet, a local area network (LAN), etc., through which the devices may exchange data among themselves and / or with one or more other electronic devices, such as a remote electronic server. In another embodiment, the network may be a wireless network, such as a wireless local area network (WLAN), a cellular network, etc., for exchanging digital data. With respect to a cellular network, the controller may be configured (e.g., via a network interface) to establish wireless (e.g., cellular) calls, which may include one or more cell towers that may be part of a communications network (e.g., a 4G Long Term Evolution (LTE) network) that supports data transmission (and / or voice calls) for electronic devices such as mobile devices (e.g., smartphones). In another embodiment, the devices may be configured to exchange data wirelessly via other networks, such as a Wireless Personal Area Network (WPAN) connection. For example, the controller 20 may be configured to establish a wireless communications link (connection) with an element (e.g., an electronic device including the sensor 21) via a wireless communications protocol (e.g., a BLUETOOTH protocol or any other wireless communications protocol).During the established wireless connection, the electronic device can transmit data, such as sensor data, to the controller 20 as data packets (eg, Internet Protocol (IP) packets).

[0034] In alternative embodiments, controller 20 may be communicatively coupled to one or more electronic devices via other methods. In particular, controller 20 may be coupled to one or more devices via wired connections. For example, controller 20 may be coupled to cameras 26 and 27 via a High-Definition Multimedia Interface (HDMI) connection to receive image data as a video stream including a series of one or more still images captured by the cameras, and may be coupled to display 28 (e.g., via another HDMI connection) to provide the one or more video streams to the display for display. In one embodiment, other video connections are possible, such as a Digital Video Interface (DVI), a Serial Digital Interface (SDI) connector, a composite video connector, or the like.

[0035] Each of first camera 26 and second camera 27 (e.g., a complementary metal-oxide-semiconductor (CMOS) image sensor) may be an electronic device configured to capture video (and / or image) data (e.g., as a series of still images). In particular, each of the cameras is positioned to capture images representing a respective field of view (FOV) of at least a portion of the environment in which the camera is located. In some embodiments, each of the cameras may be positioned within an environment such as an operating room, with both cameras having at least partially overlapping FOVs. In another embodiment, the cameras may be positioned such that neither of the cameras has an overlapping FOV, and the images captured by both cameras may not include similar (or the same) portions of the environment. For example, first camera 26 may have an FOV that captures one area of ​​the operating room (e.g., oriented toward one wall), and second camera 27 has another FOV that captures a different, non-overlapping area of ​​the operating room (e.g., oriented toward the wall opposite the wall toward which first camera 26 is oriented). In another embodiment, the cameras may have non-overlapping FOVs due to being in different environments, such as one camera in one room and another camera in a different adjacent room. In one embodiment, one or both of the camera(s) may include a wide-angle lens to maximize the camera's observable range. FOVs are described in more detail herein.

[0036] In one embodiment, one of the cameras may be an endoscope designed to capture video of a surgical site inside a patient's body during a surgical procedure. In one embodiment, at least one of the cameras may be a monocular camera (e.g., with a single camera sensor) that captures one digital (still) image at a time (e.g., as one video frame). In another embodiment, at least one of the cameras may be a stereoscopic camera with two (or more) lenses, each with a separate camera sensor for capturing individual still images to create 3D video (e.g., to generate separate video streams).

[0037] The sensor 21 may be any type of electronic device configured to detect (or sense) an environment, such as an operating room, and generate sensor data based on the environment. For example, the sensor 21 may include at least one microphone that may be configured to convert acoustic energy caused by sound wave propagation into an input microphone signal. In another embodiment, the sensor may be a proximity sensor, such as an optical sensor, that may be configured to detect the presence of one or more objects in the environment and / or the proximity of the sensor to one or more detected objects. In another embodiment, the sensor may be a temperature sensor that senses the ambient temperature in the environment in which the sensor is located as sensor data.

[0038] In some embodiments, sensor 21 may be a motion sensor, such as an inertial measurement unit (IMU), which may be designed to measure the position and / or orientation of the sensor (e.g., relative to the surrounding environment). For example, the IMU may be coupled to or part of first camera 26 and configured to detect camera movement, such as changes in the camera's position and / or orientation relative to a reference point, which may result from an operator manipulating the camera to show different perspectives of the surrounding environment. In some embodiments, the motion sensor may be a camera that captures images used by controller 20 to perform motion tracking operations (e.g., based on changes in the captured images).

[0039] The tracking device 22 may be any type of electronic device configured to track and / or detect one or more objects and generate tracking (or sensor) data (e.g., position relative to the tracking device) associated with the (e.g., detected) objects. In particular, the tracking device may capture tracking (sensor) data indicative of the object's position and / or orientation, which may be used by the controller to determine the pose of the tracked object relative to the tracking device in the environment. For example, the tracking device may be designed to track an object (e.g., its position and / or orientation in 3D space) relative to the tracking device as the object moves within a 3D space, such as an operating room, in which both the object and the tracking device may be located. As an example, the tracking device may track the spatial translation of the object relative to the tracking device as the object moves from one location to another. As a result, by tracking the object, the pose of the object that has moved from its original location to a new location may be determined (based on the tracking data), which may therefore be the relative spatial translation of the object between the first location and the second location. Further discussion of using a tracking device to estimate the pose of an object is provided herein.

[0040] In one embodiment, the pose of an object may be a transformation that indicates the position (e.g., as a translation vector or matrix) and / or orientation (e.g., as a rotation matrix) of the object relative to device 22. In another embodiment, the tracking device may include electronic components, such as one or more processors and memories, that are configurable to track an object within a threshold distance (e.g., radial distance) of the tracked object. In some embodiments, the tracking device may track an object while the object is within the FOV and / or line of sight of the device. In another embodiment, the tracking device may be configured to determine the pose of an object being tracked by device 22. The tracking device may then be configured to transmit the pose (as digital data) to controller 20.

[0041] In one embodiment, the tracking device may be a video (e.g., motion tracking) camera. For example, tracking camera 22 may be the same or similar type of camera as cameras 26 and 27. In another embodiment, the tracking device may be a proximity sensor, such as an optical sensor (e.g., an infrared (IR) sensor), a magnetic sensor, or the like, configured to detect the presence, position, and / or orientation of an object relative to the tracking device.

[0042] In another embodiment, the tracking device may be an electronic device capable of detecting the position and / or orientation of a tracking marker. For example, the tracking device may be a radio frequency (RF) position sensor (detector or receiver) capable of detecting (measuring) RF signals generated by an RF marker (or transmitter), such as an RF identifier (RFID). Such a device may determine the location of the marker by measuring the signal strength of the RF signal received from the marker. As another example, the tracking device may be an IR sensor capable of tracking an IR marker (or tag) in an environment. In another embodiment, the tracking device may be an electromagnetic sensor configured to track an electromagnetic marker (or tag).

[0043] In another embodiment, the tracking device may be any electronic device that can be configured to track an object moving in space based on data received from the object. For example, the tracking device may track an object based on position data received from the object. In that case, the object may include a position tracker (e.g., a global positioning system (GPS) device) that generates position data, and the position data may be transmitted by the object to the tracking device. As another example, the tracking device may determine the position of an object in space based on one or more electronic signals (RF signals) received from the object. For example, the tracking device may determine the position of the object in space based on the signal strength (e.g., received signal strength indicator (RSSI)) in the RF signals received from the object.

[0044] The calibration object 23 may be any type of object that can be used to perform external calibration of one or more cameras of the surgical system 1. In one embodiment, external calibration of the camera determines extrinsic parameters, such as the pose of the camera relative to a reference frame. For example, the first camera 26 may capture images of the calibration object, which may include the calibration pattern 24, and determine the relative pose of the camera with respect to the calibration pattern, and / or vice versa. The camera may be configured to determine the extrinsic parameters based on the images captured by the camera as a transformation from the camera's 3D coordinate system to a global coordinate system, which may be used to determine the pose of the first camera with respect to the reference frame. In one embodiment, the calibration pattern may be any type of (e.g., visual) pattern, such as a checkerboard pattern or a grid pattern.

[0045] In one embodiment, the calibration object 23 may be an object that may include a tracking device 25, which may be the same (or similar) type of electronic device as the tracking device 22. In another embodiment, the calibration object may include a tracking marker 92, which may be any type of marker designed to be tracked by the tracking device 22. In particular, the tracking marker may be similar (or the same) as at least one object described herein that may be tracked by the tracking device. For example, the marker may be a visual marker having a unique visual design. As another example, the marker may be any type of tag that can be tracked by any type of optical sensor, such as an IR tag that can be detected by an IR sensor. As another example, the marker may be a tracking tag (e.g., RFID) configured to be tracked by the tracking device 22. As yet another example, the tracking marker may be an electronic device that can be configured to be tracked by another electronic (e.g., tracking) device. For example, the tracking marker may include a GPS device, as described herein.

[0046] In some embodiments, the tracking marker 92, the calibration pattern 24, and / or the tracking device 25 may be part of (or fixedly attached to) the calibration object 23. For example, each (or at least one) of these elements may form an integrated unit with the calibration object 23. In one embodiment, at least some of the object's elements may be located in different locations on (or around) the object 23. For example, the calibration pattern 24 may be located on the side (or front) of the object, while the tracking marker and / or tracking device 25 may be located on (or around) the top surface of the object. Specifically, the patterns may be positioned to be visible to a camera in front of the object, while the tracking marker 92 may be positioned to be visible by a tracking device 22 that may be located above the object (e.g., on the ceiling of an operating room).

[0047] In one embodiment, at least some of the elements of the surgical system may be stationary (e.g., fixedly coupled) within the operating room environment. For example, the first camera 26 may be mounted on a wall or an object within a room, and the second camera 27 may be mounted on another wall or another object within the room (or a different room). As another example, both cameras may be mounted on the same object (e.g., the control tower 3 of the system 1 in FIG. 1 ) while facing in different (or similar) directions. As another example, the tracking device 22 may be mounted on the ceiling of the operating room. In another embodiment, the calibration object 23 may be a movable object and may be separate from (not attached to) at least some of the other elements of the surgical system 1. For example, the calibration object may be separate from the first camera 26, the second camera 27, and / or the tracking device 22. Specifically, the object may be sized to be held by a user, such that the user may place the object on a surface (e.g., on a table) in one location, lift the object, and place the calibration object in a different location on the surface (or on another surface).

[0048] The memory (e.g., a non-transitory machine-readable storage medium) 90 may be any type of electronic storage device. For example, the memory may include read-only memory, random-access memory, CD-ROM, DVD, magnetic tape, optical data storage devices, flash memory devices, and phase-change memory. While shown as separate from the controller 20, the memory may be part of the controller 20 (e.g., the controller's internal memory). As shown, the memory 90 includes one or more poses 91 of one or more objects that may be used for camera calibration of one or more cameras of the surgical system 1. In particular, the memory may include poses associated with a calibration object 23. As described herein, the object 23 includes a calibration pattern 24 that may be on (or part of) a surface (e.g., a front surface) of the object and may have a tracking marker 92. In that case, the tracking marker 92 may be on (or part of) another surface of the object, such as an upper surface. The memory may include the pose (or spatial relative transformation) of the calibration pattern 24 with respect to the tracking marker 92, and / or vice versa. Thus, the pose 91 may include at least one pose that accounts for a translation and / or rotation of the calibration pattern relative to the tracking markers. For example, if the object is square, the pose may indicate a 90° rotation about an axis extending parallel to the front surface on which the calibration pattern is attached and the top surface of the object on which the markers are attached. In one embodiment, the pose 91 of the elements of the calibration object may be predefined because the elements may be fixedly mounted (e.g., manufactured as a unit).

[0049] The controller 20 may be any type of electronic component configurable to perform one or more computational operations. For example, the controller may be a dedicated processor such as an application-specific integrated circuit (ASIC), a general-purpose microprocessor, a field-programmable gate array (FPGA), a digital signal controller, or a set of hardware logic structures (e.g., filters, arithmetic logic units, and dedicated state machines). The controller 20 is configured to spatially calibrate one or more cameras of the surgical system using sensor data, such as images captured by the cameras. Such operations enable the surgical system 1 to efficiently and effectively track movement through the different fields of view of the different cameras. Further explanation of how the controller performs calibration operations is provided herein.

[0050] In one embodiment, at least some of the operations performed by the controller may be performed in response to user input. For example, the controller may be configured to receive user input through one or more input (electronic) devices (not shown), such as a keyboard, a mouse, or a peripheral computing device such as a tablet computer. In another embodiment, user input may be received via a touch-sensitive display (e.g., display 28) that may display a graphical user interface (GUI) having one or more user interface (UI) items, and the device may generate one or more control signals (as user input) based on the user touching a portion of the display presenting the user items. In another embodiment, at least some of the operations may be performed automatically, such as without user intervention. For example, surgical system 1 may perform a calibration operation when a camera is connected to controller 20 (e.g., for the first time). As another example, the system may prompt a user (or operator) of surgical system 1 to calibrate one or more cameras (e.g., by presenting a notification, such as displaying a pop-up notification on display 28).

[0051] 3, 5, and 7 are flowcharts of a process for calibrating one or more cameras of surgical system 1. In one embodiment, at least some of the operations of at least one of the processes may be performed before surgical system 1 is used by an operator to perform a surgical (e.g., laparoscopic) procedure on a patient. In another embodiment, at least some of the operations may be performed intraoperatively (e.g., while an operator is performing a surgical procedure). In another embodiment, at least some of the operations may be performed postoperatively based on image data captured by one or more cameras during a surgical procedure. In some embodiments, at least some of the operations of the process may be performed by a surgical system (e.g., its controller 20) described herein.

[0052] Referring now to FIG. 3 , this figure shows a flowchart of a process 30 for one embodiment of calibrating one or more cameras 26 and / or 27 using a tracking device 22. In particular, the process 30 describes using the tracking device 22 to calibrate a first camera 26 and a second camera 27 according to a calibration object 23 (e.g., a calibration pattern 24 on the calibration object 23), the first camera 26 and the second camera 27 being located in one or more operating rooms. The controller 20 receives a first image captured by the first camera 26, the first image representing (or including) a first FOV of the first camera 26 with the object at the first location (block 31). For example, the calibration object 23 can be placed at a first location (e.g., on a table surface) in front of the first camera. In one embodiment, the object 23 can be positioned such that the calibration pattern 24 is within the first FOV. For example, the calibration pattern can be at the first location as described herein. In that case, the image captured by the camera may include the pattern and / or a portion of the environment surrounding the pattern. In one embodiment, the first camera 26 may capture a first image when a calibration object (e.g., its calibration pattern) is placed at a first location. In particular, the camera may capture the first image in response to user input (e.g., a user pressing a button on an input device that sends a control signal to controller 20 that causes the camera to capture an image). In another embodiment, the camera may detect that a calibration object 23 has been placed within its FOV (e.g., based on object recognition) and may capture the first image in response to detecting the object.

[0053] The controller 20 receives a second image captured by the tracking device 22, the second image having an object at the first location (block 32). In particular, the tracking device may be a tracking camera separate from the first and second cameras and may have an FOV that includes the first location and, therefore, includes the calibration pattern 24 of the calibration object 23 located at the first location. In one embodiment, the respective FOVs of both cameras may include the first location, while both cameras are located at different locations in the operating room and are stationary. In that case, both images may include different perspectives of the calibration object. For example, the FOV of the first camera may include the object's forward-facing calibration pattern, while the FOV of the target device may include an angled, top-down (bird's-eye) view of the calibration pattern. In another embodiment, the first camera 26 and the tracking device 22 may capture their respective images simultaneously. For example, both cameras may capture their respective images in response to receiving a single user input. In another embodiment, the tracking device may capture its image after (or before) the first camera captures its image.

[0054] The controller determines a first pose of the first camera based on the first image and the second image (block 33). In particular, the controller determines the pose P as a relative spatial transformation of the first camera with respect to the tracking device 22 (e.g., its frame of reference). C1 For example, when the tracking device is a camera, the origin of the coordinate system may be defined at the center of projection of the camera's lens, and one or more axes (orientations) may be defined based on the optical axis and the plane of the camera's imaging sensor. In one embodiment, P C1To determine ', the controller may implement a graph-based approach in which the pose is based on at least two other (e.g., estimated or known) poses of one or more elements in the environment, such as (e.g., calibration object 23, tracking device 22, and first camera 26 in the operating room). For example, the other poses may be relative to the same reference frame (or object) in the same coordinate system, and the determined first pose may be determined as a transformation from one pose to another (or the difference between them) in the space (e.g., 3D space) of the (e.g., 3D) coordinate system.

[0055] In one embodiment, P C1 To determine ', the controller may determine a spatial relationship between the first camera and the calibration object, and determine a spatial relationship between the tracking device and the calibration object. For example, the controller may determine a camera (or third) pose P of the calibration pattern 24 relative to the first camera while the object is at the first location. C1 and determining the tracking device (or fourth) orientation P of the calibration pattern 24 relative to the tracking device. TD1 In some embodiments, P C1 and / or P TD1 The one or more poses determined by the controller, such as P, may be the inverse spatial transformation of the spatial transformation determined by the controller based on the images of each camera, as described herein. For example, the controller may first determine a relative spatial transformation of the calibration pattern 24 with respect to the first camera 26 based on one or more images captured by the first camera 26. The controller 20 may then determine P as the inverse of the determined relative spatial transformation between the pattern and the camera so that the pose is relative to the calibration object (e.g., to change the frame of reference from the camera to the calibration pattern). C1 can be defined. In that case, P C1 and P TD1 may both be relative to the calibration pattern 24 of the object and thereby in the 3D coordinate system of the pattern.

[0056] In one embodiment, the pose P C1 and PTD1 Either (or both) of may be determined using one or more (predefined) graphical models of the calibration pattern of the object. In one embodiment, (the memory 90 of) the surgical system may include one or more 3D (e.g., computer-aided design (CAD)) models of one or more calibration patterns (and / or calibration objects), each model being a graphical mathematical coordinate-based representation (e.g., as one or more basis (or B) splines, such as non-uniform rational basis splines (NURBS)). In one embodiment, each model may include (or correspond to) one or more different poses (e.g., positions and / or orientations) of the calibration pattern within a 3D coordinate system (e.g., a Cartesian coordinate system) relative to a reference frame, such as at or on a camera. P C1 To determine P, the controller 20 may be configured to match the 3D model with the calibration pattern captured in the first image, which may provide an estimate of the spatially relative transformation of the calibration pattern with respect to the camera. In one embodiment, the pose of the calibration pattern may be estimated based on one or more (e.g., intrinsic) parameters (e.g., determined during camera calibration and / or retrieved from memory 90) and the matching 3D model. In particular, the 3D model may represent the calibration pattern in 3D model space, and the controller may use one or more (e.g., intrinsic) parameters of the camera to define the position and orientation of the calibration pattern with respect to the camera's position. In one embodiment, the controller may apply the intrinsic parameters and the 3D model to (e.g., as inputs to) a pose model, which generates as an output the pose of the calibration pattern. In one embodiment, the controller may calculate P as the inverse transformation of the estimated pose of the calibration pattern with respect to the first camera 26, as described herein. C1 In another embodiment, the pose model can be determined as P C1In another embodiment, the controller 20 may generate P from one or more images captured by the first camera 26 using any known (or future) method. C1 For example, the controller may determine P based on any external calibration algorithm that may be executed by the controller 20. C1 In some embodiments, the controller may determine P C1 By performing one or more of the operations described herein with respect to determining P TD1 can be determined.

[0057] In one embodiment, P C1 and P TD1 and the controller 20 calculates a first pose P of the first camera relative to the tracking device. C1 As described herein, the controller 20 may be configured to determine P'. As described herein, the controller 20 may implement a graph-based approach in which relative spatial transformations are projected into a 3D coordinate system relative to the same reference, which in this case may be the tracking device 22. For example, each spatial transformation may be a link between a reference (e.g., an origin) and an object (element) translated and / or rotated relative to the reference. In this case, P TD1 and P C1 ' may be with respect to the same reference, tracking device 22, while P C1 may be the transformation from the first camera to the calibration pattern. In this case, the controller TD1 and P C1 Based on the relationship between C1 For example, the controller can determine P C1 ', P TD1 and P C1 In another embodiment, the controller may determine P C1 and P TD1 is for the calibration object, P C1For example, in this case, both orientations may be relative to the calibration pattern 24, and the controller may be configured to determine the first orientation P C1 ', P TD1 From P C1 As a result, the controller can determine the spatial transformation P C1 ', P TD1 Inverse transformation of and P C1 It can be determined as a combination of

[0058] In another embodiment, the controller 20 may calibrate P by performing an extrinsic calibration method in which the intrinsic and / or extrinsic parameters of the camera are estimated using one or more images. C1 and / or P TD1 For example, the controller can determine the extrinsic parameters of the camera 26 using a calibration pattern of a calibration object captured in a first image by the first camera 26. For example, the controller can estimate PC1 using a 3D-2D correspondence between one or more known (e.g., coplanar) 3D points of the calibration pattern 24 (e.g., corners of a chessboard pattern, centroids of circles in a circular grid, etc.) and their corresponding 2D projections onto the image plane of the captured image. Because the projections depend on one or more 3D points, the controller can determine (or estimate) one or more (e.g., intrinsic and / or extrinsic) parameters using an optimization procedure (e.g., a nonlinear procedure such as the Levenberg-Marquardt algorithm). From (or using) the extrinsic parameters, the controller can calculate P C1 can be determined as the relative spatial transformation of the calibration object with respect to the camera. In one embodiment, the controller performs an extrinsic calibration method to determine P of the calibration object with respect to the tracking device using one or more images captured by the tracking device. TD1 can be determined.

[0059] The controller 20 receives a third image captured by a third camera (e.g., camera 27), the third image representing a second FOV of the second camera that does not overlap with the first FOV and has the calibration object at a second location (block 34). In particular, the calibration object 23 may be moved (transported) from a first location to a second location within the operating room. The controller 20 receives a fourth image captured by the tracking camera, the fourth image having the object at the second location (block 35). Thus, while the first camera 26, the second camera 27, and the target camera 22 remain stationary within the operating room, the object may be moved between the two locations. In that case, the target camera's FOV may include both the first and second locations without having to move (adjust its location and / or orientation). In one embodiment, the target camera 22 may include a wide-angle lens that expands the camera's FOV to include both locations. In this case, the target camera 22 and the second camera 27 may capture their respective images simultaneously. In one embodiment, the target camera images may be captured sequentially. For example, the first camera 26 and the target camera 22 may capture the first and second images, respectively, before the second camera 27 and the target camera 22 capture the third and fourth images. Conversely, the third and fourth images may be captured before the first and second images. In another embodiment, the calibration object 23 may remain stationary while the first camera 26, the second camera 27, and the target camera 22 capture their respective images. For example, the object may remain stationary at a first location when the first and second images are captured by the first camera and the tracking camera, respectively, and the object may remain stationary at a second location when the third and fourth images are captured by the third camera and the tracking camera, respectively.

[0060] The controller 20 determines a second pose of a third camera (e.g., the second camera 27) based on the third and fourth images (block 36). For example, the controller 20 may perform operations similar to those described with respect to block 33. For example, the controller may determine a pose P of the calibration object 23 at the second location relative to the second camera 27 (block 37). C2 can be determined, and the tracked camera pose P TD2 The controller may determine P TD2 and P C2 Based on this, the orientation P of the second camera 27 relative to the target camera 22 is calculated. C2 For example, the controller may perform operations similar to those described with respect to block 33 to determine P C2 ' can be determined.

[0061] The controller determines a relative spatial transformation (e.g., pose) between the first camera and the third camera based on the first pose and the second pose (block 37). In particular, the controller determines a relative transformation between the first camera 26 and the second camera 27. In one embodiment, the relative spatial transformation may be one or more extrinsic parameters of at least one of the first and second cameras. For example, the spatial transformation T of the first camera relative to the second camera may be C1 can be a relative transformation that indicates the position (e.g., translation matrix or vector) and / or orientation (e.g., rotation matrix) of the first camera relative to the second camera. Similarly, the spatial transformation T C2 may be for the first camera. In one embodiment, T C1、 Any of the relative spatial transformations such as P C1 ' and P C2 In some embodiments, P C1 ' and P C2 Since both T and T are relative to the same reference, the tracking camera, at least one of the determined spatial transformations can be inverted. For example, T C1 is P C1' and / or P C2 As described herein, the determined relative spatial transformation may include similar (or the same) data / information as the determined pose, such as including a translation matrix and / or a rotation matrix.

[0062] Some embodiments may implement variations on process 30 described herein. For example, certain operations of the process may not be performed in the exact order shown and described. Certain operations may not be performed in one continuous series of operations, and different embodiments may perform different specific operations. As described herein, the controller may receive the first camera 26 and the tracking camera 22 to determine the pose of the first camera relative to the tracking camera. In another embodiment, each of the cameras may capture one or more images of the calibration object. For example, when determining their respective poses, the controller may apply an external calibration method. In that case, this method may require multiple images of the object, and in each image, the position (and / or orientation) of the object may be adjusted. In that case, both cameras may capture one or more images of the object at different positions and / or orientations. In one embodiment, at least some of the operations described in process 30 may be performed sequentially. For example, at least some of the operations of blocks 31-33 may be performed before the execution of at least some of the operations of blocks 34-37. That is, the controller 20 may first determine the attitude of the first camera, and then determine the attitude of the second camera.

[0063] FIG. 4 illustrates several stages 40-42 illustrating the calibration of the first camera 26 and the second camera 27 using the tracking device 22 (e.g., to determine extrinsic parameters). In particular, this figure illustrates at least some of the operations described in process 30 of FIG. 3 . Each stage illustrates the first and second cameras and the tracking device, which in this example is a tracking camera in an operating room 43. As illustrated, the first camera 26 and the second camera 27 face opposite directions. Specifically, the first camera has a field of view 44 oriented in one direction, and the second camera has a field of view 45 oriented in the opposite direction. Therefore, the two FOVs may not overlap, which may be the case when both cameras are positioned around the operating room to maximize the observable range. Additionally, the tracking camera 22 is disposed above the first and second cameras and has a field of view 46 that includes the first and second cameras. For example, the tracking camera 22 may be fixed to the ceiling of the operating room, while the first and second cameras may be fixed to (e.g., respective) objects in the operating room 43 such as a table, cabinet, etc., or may be free-standing cameras (e.g., supported by tripods as shown).

[0064] The first stage 40 uses the calibration object 23 to determine the pose P of the first camera relative to the tracking device 22. C1 1 shows a diagram of estimating '. Specifically, the diagram shows the calibration object 23 at a (first) location 95, where the object, or more specifically the calibration pattern 24, is within the FOV 44 of the first camera 26 and the FOV 46 of the tracking device 22.

[0065] In addition, this stage also determines the pose P C1 ' can be estimated using the images captured by the first camera 26 and the tracking device 22. C1 and P TD1 In particular, this step shows that P C1 We present a graph-based approach in which two poses are estimated to determine the P C1is shown as a link (or projection) projecting from the first camera 26 onto the calibration object 23 (the calibration pattern 24 thereof), and P TD1 is shown as another link projecting from the tracking device 22 to (the calibration pattern 24 of) the calibration object 23. In addition, this stage also projects P as a link between the tracking device 22 and the first camera 26. C1 In one embodiment, P C1 ', P TD1 and P C1 For example, P C1 ' is, as described herein, P TD1 and P C1 It may also be based on a combination with the inverse transform of

[0066] The second stage 41 uses the calibration object 23 to calculate the pose P of the second camera relative to the tracked camera 22. C2 ' is shown. In particular, this diagram shows that the calibration object 23 has been moved from location 95 to another (second) location 96, which is now within the FOV 45 of the second camera 27. Additionally, the new location 96 is still within the FOV 46 of the tracking device. Thus, the calibration object 23 having the calibration pattern 24 has moved from the first location 95 to the second location 96, and is within the FOV 45. In one embodiment, the movement may be performed by a person (e.g., a technician). For example, the technician may pick up the object and move it between locations. As another example, the object 23 may be mounted on a movable platform (e.g., a cart) and moved between locations either manually by a person or automatically (e.g., without user intervention) via one or more motors or actuators of the movable cart. Additionally, as shown, the calibration object 23 is moving, while the first camera 26, the second camera 27, and the tracking camera 22 remain in fixed (stationary) positions within the operating room 43.

[0067] Additionally, the second stage shows the pose as a graphical link between the second camera 27 and the calibration object 23, and between the tracking device 22 and the calibration object. In particular, this figure shows the P C2 and P between the calibration object and the tracking device 22 TD2 In addition, this step uses P as the link between the second camera 27 and the tracking device 22. C2 ', which, as described herein, represents P C2 ' and P TD2 In one embodiment, at least some of the poses shown in this stage may be determined in a manner similar to the poses in the first stage 40 described herein.

[0068] The third stage 42 is the posture P C1 ' and P C2 '. In particular, it shows the spatial relative transformations of both the first camera 26 and the second camera 27 as links from the tracked camera, both of which are in the same coordinate system (e.g., relative to the tracked camera 22). As a result, the controller 20 determines the spatial relative transformation T between both cameras based on the difference between the two poses. C1 and T C2 For example, T C1 may be the spatially relative transformation of the first camera with respect to the second camera, and T C2 may be the spatially relative transformation of the second camera with respect to the first camera As shown herein, the FOV 46 of the tracking camera 22 includes both locations 95 and 96, as well as the first camera 26 and the second camera 27. In one embodiment, the FOV 46 may not include one or both of the cameras, but may have only the object locations within its field of view. In another embodiment, the camera positions and / or orientations may be different, for example, the cameras may be arranged in a circle (e.g., if there is more than one camera the surgical system is calibrating).

[0069] 5 is a flowchart of a process 50 for one embodiment of calibrating one or more cameras, such as cameras 26 and / or 27, using tracking device 22 and tracking marker 92, which may be part of calibration object 23. Specifically, tracking marker 92 may be part of (or fixedly coupled to) calibration object 23, which includes calibration pattern 24. Process 50 begins with controller 20 receiving a first image captured by first camera 26, the first image representing a first FOV of the first camera including marker 92 at a first location in the operating room (block 51). In particular, the calibration object may be at the first location (e.g., placed at the location by a user), and the captured image includes at least a portion of the calibration pattern. In one embodiment, tracking marker 92 may be disposed on a different side (or surface) than the calibration pattern and, therefore, may not be within the FOV of the first camera. As a result, the captured image may not include the tracking marker.

[0070] The controller 20 determines a first pose P of the first camera 26 based on the first image and the first location of the marker. C1 In particular, the controller determines P′ based on one or more poses estimated using (at least) the first image and / or marker detection. C1 For example, the controller may determine the pose P of the calibration pattern 24 relative to the first camera while the object 23 is at the first location, as described herein. C1 Furthermore, the controller can determine the pose P of the calibration pattern of the calibration object relative to the tracking device 22. TD1In one embodiment, to do this, the controller can determine the spatial relationship between the tracking device 22 and the tracking marker 92. In one embodiment, the tracking device may be a proximity (or location-detecting) sensor configured to detect position data (e.g., position and / or orientation) of the tracking marker. In particular, the controller may be configured to detect (determine) the location (and / or orientation) of the tracking marker 92 relative to the tracking device using tracking / sensor data generated by the device. For example, if the tracking marker includes an RF transmitter (or an RF tag such as an RFID) and the tracking device includes an RF sensor, the RF sensor may be configured to sense an RF signal generated (or reflected) from the RF tag and generate sensor data from the signal. In another embodiment, the tracking device may be an IR sensor, and the marker may be an IR tag. In that case, the tracking device can generate sensor data based on an infrared signal generated by the IR sensor that is reflected from the IR tag, which is indicative of the position characteristics of the marker. In one embodiment, the sensor data can be indicative of the position and / or orientation of the marker relative to the tracking device. The controller calculates the pose P of the tracked marker relative to the tracking device 22 according to the determined location of the marker (e.g., using sensor data). TD1 The controller may then use this information to determine (estimate) the P of the tracking marker 92 relative to the tracking device. TD1 ' can be estimated (e.g., by generating a transformation matrix that indicates the translation and / or rotation of the marker relative to the tracking device). TD1 ' may be the inverse transform of the pose of the tracking marker 92 determined relative to the tracking device as described herein.

[0071] In another embodiment, P TD1' may be estimated using one or more images, as described herein. For example, if the tracking device 22 is a tracking camera, the tracking camera may be configured to capture one or more images of the tracking marker 92, which may include one or more visible patterns (or objects), and the controller may estimate P as described herein. TD1 This may be the case when the calibration pattern 24 is not within the field of view of the tracking device 22.

[0072] As described herein, the tracking marker 92 is C1 The markers may be at a different location on the calibration object than the calibration pattern 24 that may be used to determine P. For example, the markers may be attached to the top surface of the object and the calibration pattern may be attached to the front surface of the object. In that case, the controller may determine P by considering the relative spatial relationship between the markers and the pattern. TD1 For example, the controller may be configured to determine the pose P of the tracked marker relative to the calibration pattern. TM can be retrieved (or received) (e.g., from the attitude 91 in memory 90), and P TM According to P TD1 As described herein, this adjustment may be performed using a graph-based approach. As a result, the controller, as described herein, C1 ', P TD1 and P C1 It can be estimated as a combination of

[0073] Controller 20 receives a second image captured by second camera 27, the second image representing a second FOV of the second camera that does not overlap with the first FOV of the first camera, and including a marker at a second location within the operating room (block 53). In particular, the object to which the marker is fixedly attached may be moved from one location to another within the operating room, as described herein.

[0074] The controller 20 determines a second pose P of the second camera 27 based on the second image and the second location of the marker. C2 ' (block 54). In one embodiment, the controller performs operations similar to those described for block 52 to determine P C2 For example, the controller may determine the pose P of the calibration pattern 24 of the object at the second location relative to the second camera. C2 and P of the calibration pattern 24 for the tracking device 22. TD2 In one embodiment, P TD2 can be estimated based on tracking data generated by a tracking device as the tracking marker (e.g., its calibration object) is moved from a first location to a second location. Tracking markers are further described herein. As a result, the controller determines the pose P of the tracking marker relative to the device 22. TD2 ', and this attitude P TM may be adjusted for.

[0075] The controller 20 controls the first attitude P C1 ' and the second position P C2 ' to determine the relative spatial transformation between the first camera 26 and the second camera 27 (block 55). Thus, in this embodiment, the controller uses the locations of the tracking markers on the calibration object to determine the relative spatial transformation, while the cameras and tracking devices remain stationary as the markers (e.g., the calibration object) are moved between locations in the operating room.

[0076] FIG. 6 illustrates several stages 60-62 illustrating the calibration of first camera 26 and second camera 27 using tracking device 22 and tracking marker 92 on calibration object 23. In particular, this figure illustrates at least some examples of the operations described in process 50 of FIG. 5. Each of these stages illustrates first camera 26, second camera 27, and tracking device 22, which may be any type of electronic device capable of detecting the location of an object (and / or marker) in space, such as an IR sensor or camera. Additionally, this figure illustrates that calibration object 23 includes calibration pattern 24 and tracking marker 92. In particular, this illustrates that calibration pattern 24 is at location 93 on (or around) calibration object 23, and tracking marker 92 is at another location 94. For example, as described herein, the calibration pattern may be on the front surface of the object, and the marker may be on the top surface of the object.

[0077] The first stage 60, as described herein, is C1 and P TD1 P C1 ' is shown. In particular, this diagram shows that the calibration object 23 is at a location 95 including the calibration pattern 24 that is within the FOV 44 of the first camera 26. Additionally, a tracking marker 92, such as an IR tag, may not be within the FOV 44 or may be at least partially within the FOV 44. This stage also shows that the tracking marker 92 is within the FOV 46 of the tracking device 22. This may be the case when the tracking device is a tracking camera. In another embodiment, the FOV 46 may represent the (e.g., radial) distance and / or line of sight over which the tracking device can track one or more objects, such as the marker 92. For example, when the tracking device is a proximity sensor, the FOV 46 may be the distance over which the sensor can detect the proximity of an object.

[0078] This first stage 60 involves generating a P C1In particular, this figure shows the pose determined by the surgical system to estimate P between the calibration pattern 24 and the first camera 26. C1 , and P between the calibration pattern 24 and the tracking device 22 TD1 , which is determined based on the detected location of the tracking marker 92 by the tracking device 22, as described herein. However, the calibration pattern may not be within the line of sight (or FOV 46) of the tracking device 22, so for example, the controller may determine P based on the detected location of the tracking marker 92. TD1 To do this, the controller determines P , the pose of the tracking marker 92 relative to the tracking device 22 as the tracking marker may be within the FOV 46 of the device 22. TD1 This figure also shows the P TM In this case, the controller TD1 , P TD1 ' and P TM It can be determined graphically as a combination (e.g., their product) of

[0079] The second stage 61 is P C2 ', which, as described herein, is C2 and P TD2 The positioning of the tracking marker 92 may be based on the spatial relationship between the first location 95 and the second location 96. In particular, the figure shows that the calibration object 23 is moved from a first location 95 to a location 96 and is within the FOV 45 of the second camera 27. In addition, the tracking marker 92 may be within the FOV 46 of the tracking device. In one embodiment, the controller may perform operations similar to those described for the first stage 60. In one embodiment, the tracking marker 92 may remain within the FOV 46 when moving from the first location 95 to the second location 96.

[0080] At this stage, a P C2 is shown, and P TD2 In one embodiment, PTD2 To determine P, the controller can determine the relationship between the tracking marker 92 and the tracking device 22. Thus, this diagram illustrates the P between the tracking marker 92 and the tracking device 22. TD2 ', which may be determined based on the positions of the markers detected by the tracking device, as described herein. Because the tracking markers 92 and the calibration pattern are rigidly coupled to the calibration object 23, the relative spatial transformation P TM remains the same as shown in the first stage 60. In one embodiment, as the tracking marker (e.g., its calibration object) moves between locations 95 and 96, P TD2 ', P TD1 The controller then determines P TD2 , P TD2 ' and P TM In some embodiments, tracking marker 92 may remain within FOV (e.g., line of sight) 46 of tracking device 22 at (and / or between) locations 95 and 96. In another embodiment, FOV 46 may be a threshold distance, such as a radial distance, from tracking device 22 where both locations may be located.

[0081] The third stage 62 is the posture P C1 ' and P C2 ', and the controller, as described herein, determines P C1 ' and P C2 ', and calculate the spatially relative transformation T of the first camera with respect to the second camera using a graph-based approach by identifying the translation and / or rotation between C1 , or the spatial relative transformation T of the second camera with respect to the first camera C2 can be determined.

[0082] 5 and 6 is a calibration method that uses a tracking device to track the location of a marker to determine the spatial relationship between at least two cameras of surgical system 1. In one embodiment, the method allows for high accuracy and tolerance to ambient conditions in the operating room (e.g., lighting therein) by enabling the tracking device to effectively track the marker from a first location to a second location.

[0083] The calibration methods described thus far involve estimating relative spatial transformations between spatially distributed stationary image sensors (e.g., RGB video cameras or RGBD depth sensors). In particular, both of the calibration methods described in FIGS. 3-6 may be “outside-in” tracking calibration methods in which a sensor, such as a stationary (e.g., stationary) tracking device 22 in an operating room tracks a calibration object 23 (e.g., its markers 92) between two or more locations. In another embodiment, a surgical system may be configured to perform one or more “inside-out” tracking calibration methods in which one or more tracking devices are movable in an operating room to track the movement of the tracking device based on detected positional changes of one or more (e.g., stationary) objects relative to the tracking device. In particular, FIGS. 7 and 8 relate to such inside-out tracking calibration methods.

[0084] 7, which illustrates a flowchart of an embodiment of a process 70 for performing inside-out calibration of one or more cameras, such as first camera 26 and / or second camera 27, using a tracking device and tracking markers. Specifically, process 70 describes calibrating a camera using a tracking device 25 coupled to a calibration object 23 (or a portion thereof) to track the movement of the calibration object 23 (e.g., calibration pattern 24 of calibration object 23) by detecting relative spatial changes of the tracking device 25 with respect to a separate tracking marker (e.g., tracking marker 98 as shown in FIG. 8), which may be separate from the calibration object 23. In one embodiment, the tracking marker 98 may be a static (stationary) object in the operating room, while the calibration object 23, including the tracking device 25, may be movable.

[0085] Process 70 begins by the controller receiving a first image captured by first camera 26, the first image representing a first FOV of first camera 26 that includes an object at a first location in the operating room (block 71). Thus, as described herein, the first camera's FOV may include a calibration object 23 at the first location, which may be placed there by a user. The controller determines a first pose P of first camera 26 based on the first image and tracking markers 98 in the operating room. C1 In particular, the controller 20 determines P′′, which may be the pose of the first camera 26 relative to the tracking marker 98, based on the one or more estimated poses. C1 Therefore, the controller can determine a first pose (e.g., P ) of the first camera 26 relative to the tracking device 22. C1 Unlike at least some of the other calibration methods described herein, which determine the pose of the first (and second) camera relative to the tracking marker 98, the controller 20 can determine the pose of the first (and second) camera relative to the tracking marker 98. As a result, P C1 '' may be determined based on detection of tracking markers by tracking device 25 of the calibration object, which may be at the first location. These operations are described further herein.

[0086] In one embodiment, the controller 20 calculates P C1 Specifically, the controller 20 may use (at least) the first image to determine P C1 The controller can determine P C1 and the pose P of the calibration object 23 (e.g., its calibration pattern 24) relative to the tracking marker 98. TM1 Based on (or a combination of) C1 In particular, P TM1may represent the spatial relative transformation of the calibration object 23 while at the first location from the tracking marker 98 at its particular (stationary) location in the operating room. In one embodiment, the transformation between the calibration object and the tracking marker 98 can change as the location of the object in the operating room changes. Further discussion of transformation changes is provided herein.

[0087] In one embodiment, P TM1 To determine the pose P of the tracking marker 98 relative to the tracking device 25, the controller TD1 Specifically, the controller may detect tracking marker 98 using tracking device 25 while the calibration object is at the first location, and determine P based on the detection of the tracking marker while the calibration object (e.g., its tracking device 25) is at the first location. TD1 In one embodiment, tracking device 25 may generate sensor data upon detecting tracking markers indicative of its position and / or orientation within the operating room, and using this data, controller 20 determines P TD1 In some embodiments, the controller may determine P'' from the sensor data. TD1 For example, when tracking device 25 is a tracking camera, tracking marker 98 is detected based on one or more images captured by the tracking camera, and from those one or more images, the controller can determine the pose of the tracking marker relative to tracking device 25, as described herein.

[0088] In one embodiment, P TM1To generate , the controller may be configured to take into account the relative spatial relationship between the tracking device 25 and the calibration pattern 24 of the calibration object 23. In one embodiment, the calibration pattern 24 is at one location on the calibration object 23 and the tracking device 22 is at another location on the calibration object 23. For example, the tracking device may be at or near a location on the top surface of the calibration object to have an upward FOV (or line of sight) to observe tracking markers that may be on the calibration object, while the location of the pattern may be towards a side of the object. As a result, the controller may calculate the pose P of the calibration pattern relative to the tracking device 25. TD (e.g., from pose 91 in memory 90). In one embodiment, the controller TD Considering the transformation of P TD1 By adjusting the TM1 For example, P TM1 is P TD1 '' and P TD In one embodiment, P TM1 and P C1 In this case, the controller C1 '', P TM1 and P C1 It may be configured to determine it as a combination of (the inverse transform of)

[0089] Returning to process 70, the controller 20 may track the movement of the object from a first location to a second (different) location (block 73). In particular, the controller may use a tracking device 25 coupled to the calibration object 23 to track the movement based on sensor data generated by the device. For example, as a user picks up the calibration object and moves it to a different location, the tracking device may capture sensor data (e.g., periodically, such as every second) indicating changes in the location and / or orientation of the tracking device (and / or object). As an example, if the tracking device 25 is a motion camera configured to capture one or more images of an environment, the controller may be configured to execute a camera motion tracking algorithm (e.g., a Simultaneous Localization and Mapping (SLAM) algorithm, a Visual Odometry (VO) algorithm, etc.) to track the movement of the device based on the movement of one or more points in a series of one or more video frames (images) captured by the camera. As described herein, the relative spatial transformation of the tracking device at the second location may be based on the tracked movement of the object from the first location in space relative to the tracking markers.

[0090] The controller 20 receives a second image captured by the second camera 27, the second image representing a second FOV of the second camera that does not overlap with the first FOV and having an object at a second location within the operating room (block 74). For example, the calibration object 23 may be manually moved by a user from the first location to another location within the operating room. In another embodiment, the object may be moved autonomously (e.g., by an autonomous robot within the room). In one embodiment, the tracking marker 98 may remain within the field of view of the tracking device while the object is moved.

[0091] The controller 20 determines a second pose P of the second camera 27 based on the second image and the tracking marker. C2In one embodiment, the controller performs at least some of the operations described herein, such as with respect to block 72 of this process, to determine P C2 For example, the controller may determine the pose P of the calibration pattern 24 of the object at the second location relative to the second camera. C2 , and P , the pose of the calibration pattern 24 of the calibration object while the object is at the second location, relative to the tracking marker 98. TM2 In one embodiment, the controller may determine P TM1 By performing at least some of the operations described herein with respect to TM2 In one embodiment, the controller may determine P based on the detected movement of the tracking device. TM2 For example, the controller may indicate changes in the location and / or orientation of the tracking device as it moves through space, and may use tracking data generated by the tracking device while the object is moved from a first location to a second location to determine P TD2 '' can be determined. In that case, P TD2 ″ may be the relative spatial translation of the tracking device 25 from the first location to the second location. In another embodiment, P TD2 '' may be estimated based on the tracking data and based on sensor data generated by the tracking device 25 at the second location. For example, the tracking data may indicate one or more translational parameters and / or one or more rotational parameters from the first location to the second location relative to the tracked marker, and other translational and / or rotational parameters may be determined based on sensor data captured by the tracking device at the second location. The controller may control P as described herein. C2 and P TM2 Based on P C2 The controller determines the first attitude P C1 '' and the second posture P C2' and ', a relative spatial transformation between the first camera 26 and the second camera 27 is determined (block 76). For example, a graph-based approach may be used to C1 To determine P, the controller may use the P C2 '' to P C1 In one embodiment, the controller may be configured to determine one or more extrinsic parameters of the first and second cameras based on the tracked movement of the cameras. For example, the tracked movement may indicate a distance between the first camera 26 and the second camera 27. In that case, the controller may determine a difference (or change) in position from T to T based on the determined distance. C1 and T C2 One or more translation parameters of a translation vector associated with the

[0092] FIG. 8 illustrates several steps 80-82 illustrating calibrating a first camera 26 and a second camera 27 using a tracking device 25 of a calibration object 23 to detect a tracking marker 98. In particular, this figure illustrates at least some of the operations described in process 70 of FIG. 7 . Each of these steps illustrates a first camera 26, a second camera 27, and a tracking marker 98 disposed within one or more operating rooms 43. Specifically, the tracking marker 98 is shown at a location 99 above both cameras. For example, the tracking marker 98 may be attached to the ceiling of the operating room, while the two cameras are supported on the floor of the room. In another embodiment, the tracking marker 98 may be located at a different location within the room (e.g., on a wall of the room). Additionally, this figure illustrates a calibration object 23 including a calibration pattern 24 at a first location 93 on the object 23 and a tracking device 25 at another location 97 on the object. In one embodiment, each of these locations may be on a different surface of the object (eg, location 93 is on the forward-facing surface, while location 97 is on the upward-facing surface).

[0093] The first stage 80 uses the tracking device 25 of the calibration object 23 to determine the pose P of the first camera relative to the tracking marker 98. C12 shows a diagram of estimating P′′. Specifically, this diagram shows the calibration object at location 95 where calibration pattern 24 is within FOV 44 of first camera 26. In addition, FOV 46 of tracking device 25 can include tracking marker 98 so that the tracking device can detect the marker. This stage also illustrates pose as a link between elements in, for example, the operating room, which is shown as the link between tracking marker 98 and first camera 26. C1 In particular, this step involves determining the P between the calibration pattern 24 and the first camera 26. C1 , which is determined based on images captured by first camera 26, as described herein. In addition, this example also shows P , the pose of tracking marker 98 relative to tracking device 25, which may be determined by the controller based on sensor data from tracking device 25, as described herein. TD1 ''. P TD is also shown, which may be the orientation of the calibration pattern 24 relative to the tracking device 25. In one embodiment, the controller 20 uses these two orientations to calculate P, which is shown as the link between the tracking marker 98 and the calibration pattern. TM1 As described herein, the controller can graph these poses in the coordinate system of the calibration pattern 24 to determine P C1 '' can be determined, which means that P TM1 and P C1 The method may be based on a combination of

[0094] The second step 81 is to calculate the estimated pose P of the second camera 27 relative to the tracking marker 98. C2 1 illustrates a diagram of estimating P′′, which may be performed while the calibration object 23 is in a new location. Specifically, the diagram shows that the object 23 has been moved from location 95 to location 96, which is within the FOV 45 of the second camera 27. As described herein, the controller 20 performs operations similar to those described with respect to the first pose 80 to determine one or more poses and estimate P′′. C2For example, this figure shows the pose P of the second camera 27 relative to the calibration pattern 24. C2 This figure also shows the P between the tracking device 25 and the tracking marker 98. TD2 ″, which may be estimated based on sensor data captured by the tracking device 25 at the location 96 and / or based on tracking data captured by the tracking device as it is moved from the first location 95 to the second location 96. In one embodiment, the controller TM2 To estimate P TD According to P TD2 The controller may then adjust P C2 and P TM2 Based on P C2 '' to determine.

[0095] The third stage 82 is the posture P C1 '' and P C2 ' ', and the controller, as described herein, determines P C1 '' and P C2 Calculate the spatially relative transformation T of the first camera with respect to the second camera using a graph-based approach by identifying the translation and / or rotation between C1 , or the spatial relative transformation T of the second camera with respect to the first camera C2 can be determined.

[0096] Some embodiments may implement variations on the process 30 described herein. For example, certain operations of the process may not be performed in the exact order shown and described. Certain operations may not be performed in one continuous series of operations, and different embodiments may perform different specific operations. In one embodiment, at least some of the operations described herein may be performed once to calibrate two or more cameras. For example, at least some operations and / or at least some elements illustrated herein with dashed boundaries may be optional, as described herein. In another embodiment, the process 70 may be performed to calibrate the first camera 26 and the second camera 27 during initial camera setup. Once calibrated, the surgical system 1 can use the estimated extrinsic parameters (relative spatial transformations) while performing vision processing operations such as motion detection. In another embodiment, at least some of the operations may be performed periodically to recalibrate one or more cameras over a period of time. For example, over time, one or more parameters of the cameras may drift (e.g., gradually change). As a result, the surgical system may perform at least some of these actions after a period of time to ensure the parameters are accurate.

[0097] As described herein, the surgical system may perform at least one of processes 30, 50, or 70 to calibrate one or more cameras. In one embodiment, the surgical system may perform two or more of the calibration methods described herein to optimize the estimated extrinsic parameters. For example, the surgical system may perform at least some of the operations of processes 30 and 70 of Figures 3 and 7, respectively, and may determine a relative spatial transformation between first camera 26 and second camera 27 based on two sets of transformations for each camera. In one embodiment, when performing two or more calibration methods, the surgical system may average the estimated relative spatial transformations between the cameras.

[0098] In one embodiment, at least some of the operations can be performed to calibrate the first camera 26 and the second camera 27 of the surgical system 1. In particular, the surgical system can perform one or more of the calibration operations described herein to calibrate three or more cameras. In that case, the surgical system 1 can be configured to estimate relative spatial transformations between pairs of cameras. For example, with three cameras, cameras "A," "B," and "C," the surgical system can use one or more of the calibration methods described herein to determine a first pair of relative spatial transformations between cameras A and B, a second pair of relative spatial transformations between cameras A and C, and / or a third pair of relative spatial transformations between cameras B and C. In some embodiments, the surgical system can determine each pair of spatial transformations between cameras in the same coordinate system using a graph-based approach, as described herein.

[0099] In one embodiment, the operations described herein may enable a surgical system to calibrate cameras having non-overlapping fields of view, hi another embodiment, at least some of the operations described herein may be used to calibrate cameras having at least partially overlapping fields of view.

[0100] As described herein, tracking devices 22 and 25 are positioned to detect tracking markers 92 and 98, respectively, to track the movement of the calibration object. In particular, the tracking devices may track movement by capturing sensor data at locations where the calibration object is positioned. In another embodiment, the tracking devices may track the movement of the calibration object as it moves around an operating room and / or between one or more operating rooms. By way of example, and referring to FIG. 8 , tracking device 25 may track movement (e.g., of the calibration object) as the object is moved (e.g., carried by a user) between locations. This may allow cameras in different operating rooms to be calibrated relative to one another. For example, a surgical system may use tracking device 25 to track movement from a first location 95 in one room to a second location 96 in another room. The surgical system may then use the tracked movement of the object to account for the distance between the two cameras. In one embodiment, the surgical system can use sensor data from either (or both) tracking devices 22 and 25 to track the movement of the calibration object and more accurately and effectively estimate one or more of the poses described herein.

[0101] In one embodiment, at least some of the operations described herein may be performed in "real time," meaning that the operations may be performed by the surgical system as one or more images are captured by one or more cameras to be calibrated. In one embodiment, the surgical system may provide feedback to the user as the user is calibrating the cameras. As an example, referring to FIG. 8, when the user places the calibration object at location 95, the surgical system may provide a notification to the user indicating whether the object is within the FOV of the first camera. For example, once placed, the surgical system may provide a pop-up notification indicating whether the calibration object needs to be adjusted. C1Once "' is estimated, the surgical system can provide a notification alerting the user to move the object to the second location 96. In this case, the surgical system may use the speaker 29 to output an audio notification saying, "Please move the calibration pattern in front of the next camera." Providing feedback can ensure that the surgical system is calibrated efficiently and effectively.

[0102] As described herein, the surgical system 1 may be configured to estimate the pose of an object relative to another object based on sensor data. In another embodiment, the estimated (or overall) pose may be based on one or more estimated poses, such that the estimated overall pose may be an average of one or more poses. For example, with reference to FIG. 4, the controller may be configured to control one or more P C1 In this case, the surgical system 1 may be configured to estimate P C1 Multiple P based on changes to other postures are used to estimate ' C1 For example, the surgical system may estimate a first P while the calibration pattern is at a first orientation relative to camera 1. C1 ', and a second P C1 In one embodiment, the calibration patterns may be at the same location 95 but with different orientations. The resulting P C1 ' is the first P C1 ' and the second P C1 ' (e.g., average). In one embodiment, estimating pose based on an average one or more poses can reduce noisy data and produce a better overall pose estimate.

[0103] As previously described, one embodiment of the present disclosure may be a non-transitory machine-readable medium (such as a microelectronic memory) storing instructions that program one or more data processing components (collectively referred to herein as a "processor") to automatically (e.g., without user intervention) calibrate one or more cameras using one or more images as described herein. In other embodiments, some of these operations may be performed by specific hardware components that include hardwired logic. These operations may alternatively be performed by any combination of programmed data processing components and fixed hardwired circuitry.

[0104] In order to assist the Patent Office and any reader of any patent issued on this application when interpreting the claims appended hereto, Applicant wishes to note that none of the appended claims or claim elements are intended to invoke 35 U.S.C. 112(f) unless the phrase "means for" or "step for" is expressly used in a particular claim.

[0105] While several embodiments have been described and illustrated in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of the broad disclosure and not limiting thereof, and that the disclosure is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those skilled in the art. Accordingly, the description should be regarded as illustrative instead of restrictive.

[0106] In some embodiments, the disclosure may include language such as, "at least one of [element A] and [element B]." This style may refer to one or more of the elements. For example, "at least one of A and B" may refer to "A," "B," or "A and B." Specifically, "at least one of A and B" may refer to "at least one of A and at least one of B" or "at least one of either A or B." In some embodiments, the disclosure may include language such as, "[element A], [element B], and / or [element C]." This style may refer to any of the elements or any combination thereof. For example, "A, B, and / or C" may refer to "A," "B," "C," "A and B," "A and C," "B and C," or "A, B, and C."

[0107] [Embodiment] (1) A method performed by a surgical system including a first camera, a second camera, and a tracking camera located in an operating room, the method comprising: receiving a first image captured by the first camera, the first image representing a first field of view (FOV) of the first camera having an object at a first location within the operating room; receiving a second image captured by the tracking camera, the second image having the object at the first location; determining a first pose of the first camera based on the first image and the second image; receiving a third image captured by the second camera, the third image representing a second FOV of the second camera that does not overlap with the first FOV and having the object at a second location within the operating room; receiving a fourth image captured by the tracking camera, the fourth image having the object at the second location; and determining a second pose of the second camera based on the third image and the fourth image; determining a relative spatial transformation between the first camera and the second camera based on the first pose and the second pose; A method comprising: (2) The method of embodiment 1, wherein the tracking camera includes a third FOV that includes both the first location and the second location, and the tracking camera is stationary while the object is moved from the first location to the second location. (3) The method of embodiment 1, wherein the third image and the fourth image are captured by the second camera and the tracking camera, respectively, before the first image and the second image are captured by the first camera and the tracking camera, respectively. (4) The method of embodiment 1, wherein the first image and the second image are captured simultaneously by the first camera and the tracking camera, respectively, and the third image and the second image are captured simultaneously by the third camera and the tracking camera, respectively. (5) the object remains stationary at the first location when the first image and the second image are captured by the first camera and the tracking camera, respectively; 2. The method of claim 1, wherein the object remains stationary at the second location when the third image and the fourth image are captured by the third camera and the tracking camera, respectively.

[0108] (6) The method of embodiment 1, wherein the object is not attached to any of the first camera, the second camera, and the tracking camera. (7) Determining the first attitude of the first camera based on the first image and the second image includes: determining a third pose of the first camera relative to the object at the first location using the first image; and 2. The method of claim 1, further comprising: using the second image to determine a fourth pose of the tracking camera relative to the object at the first location. (8) The method of embodiment 1, wherein the relative spatial transformation indicates the position and orientation of the second camera relative to the first camera. (9) The method of embodiment 1, wherein the object comprises a calibration pattern, and the calibration pattern is moved from the first location to the second location by a user in the operating room, while the first camera, the second camera, and the tracking camera remain in fixed positions. (10) A method performed by a surgical system including a first camera and a second camera located in an operating room, the method comprising: receiving a first image captured by the first camera, the first image representing a first field of view (FOV) of the first camera including a marker at a first location within the operating room; determining a first pose of the first camera based on the first image and the first locations of the markers; receiving a second image captured by the second camera, the second image representing a second FOV of the second camera that does not overlap with the first FOV and including the marker at a second location within the operating room; determining a second pose of the second camera based on the second image and the second locations of the markers; determining a relative spatial transformation between the first camera and the second camera based on the first pose and the second pose, wherein the first camera and the second camera remain stationary when the marker is moved from the first location to the second location; A method comprising:

[0109] (11) The marker is fixedly coupled to an object including a calibration pattern, the object being at the first location captured in the first image and the second image, the method further comprising: determining a third location at which the marker is fixedly coupled to the calibration object using a tracking device; and determining the first pose of the first camera includes: determining a third pose of the first camera relative to the calibration pattern while the object is at the first location; 11. A method as described in embodiment 10, comprising determining a fourth attitude of the tracking device relative to the calibration pattern using the third location of the marker. (12) Determining the fourth attitude of the tracking device includes: determining a fifth orientation of the tracking device relative to the marker according to the third location of the marker; deriving a sixth orientation of the marker relative to the calibration pattern; adjusting the fifth attitude according to the sixth attitude. (13) The method of embodiment 11, wherein the marker and the calibration pattern are one integrated unit. (14) The method of embodiment 11, wherein the tracking device is an infrared (IR) sensor and the marker is an IR tag. (15) The method of embodiment 11, wherein the tracking device is a camera and the marker is a visible pattern.

[0110] (16) A method performed by a surgical system including a first camera and a second camera located in an operating room, the method comprising: receiving a first image captured by the first camera, the first image representing a first field of view (FOV) of the first camera including an object at a first location within the operating room; determining a first pose of the first camera based on the first image and tracking markers in the operating room; receiving a second image captured by the second camera, the second image representing a second FOV of the second camera that does not overlap with the first FOV and having the object at a second location within the operating room; determining a second pose of the second camera based on the second image and the tracking marker; determining a relative spatial transformation between the first camera and the second camera based on the first pose and the second pose; A method comprising: (17) The object is equipped with a tracking device, and the method further comprises: detecting the tracking marker using the tracking device while the object is at the first location, wherein the first pose is determined based on the detection of the tracking marker while the object is at the first location; 17. The method of claim 16, further comprising: using the tracking device to detect the tracking marker while the object is at the second location, and the second attitude is determined based on the detection of the tracking marker while the object is at the second location. (18) The method of embodiment 17, further comprising using the tracking device to track movement of the object from the first location to the second location, and the second pose is determined based on the tracked movement of the object. (19) The method of embodiment 17, wherein the tracking device and the object are a single integrated unit. (20) The method comprises: determining a third pose of the tracking marker relative to the object based on the detection of the tracking marker while the object is at the first location; 18. The method of claim 17, further comprising determining a fourth pose of the first camera relative to the object based on the first image, wherein the first pose of the first camera is based on the third pose and the fourth pose.

[0111] (21) Determining the third attitude of the tracking marker includes: determining a fifth orientation of the tracking marker relative to the tracking device based on the detection of the tracking marker; receiving a sixth pose of the tracking device relative to the object; adjusting the fifth attitude according to the sixth attitude. (22) The method of embodiment 17, wherein the tracking device is a tracking camera, and detecting the tracking marker while the object is at the first location includes receiving a third image captured by the tracking camera, the third image representing a third FOV of the tracking camera and including the tracking marker. (23) The method of embodiment 17, wherein the tracking marker is an infrared (IR) tag and the tracking device is an IR sensor, or the tracking marker is a radio frequency (RF) tag and the tracking device is an RF sensor. (24) The method of embodiment 16, wherein the object comprises a calibration pattern, and the calibration pattern is moved from the first location to the second location by a user or an autonomous robot in the operating room, while the first camera and the second camera remain in a fixed position.

Claims

1. 1. A method performed by a surgical system including a first camera, a second camera, and a tracking camera located in an operating room, the method comprising: receiving a first image captured by the first camera, the first image representing a first field of view (FOV) of the first camera having an object at a first location within the operating room; receiving a second image captured by the tracking camera, the second image having the object at the first location; determining a first pose of the first camera based on the first image and the second image; receiving a third image captured by the second camera, the third image representing a second FOV of the second camera that does not overlap with the first FOV and having the object at a second location within the operating room; receiving a fourth image captured by the tracking camera, the fourth image having the object at the second location; determining a second pose of the second camera based on the third image and the fourth image; determining a relative spatial transformation between the first camera and the second camera based on the first pose and the second pose; A method comprising:

2. 10. The method of claim 1, wherein the tracking camera includes a third FOV that includes both the first location and the second location, and the tracking camera is stationary while the object is moved from the first location to the second location.

3. 2. The method of claim 1, wherein the third image and the fourth image are captured by the second camera and the tracking camera, respectively, before the first image and the second image are captured by the first camera and the tracking camera, respectively.

4. 2. The method of claim 1, wherein the first image and the second image are simultaneously captured by the first camera and the tracking camera, respectively, and the third image and the second image are simultaneously captured by the third camera and the tracking camera, respectively.

5. the object remains stationary at the first location when the first image and the second image are captured by the first camera and the tracking camera, respectively; The method of claim 1 , wherein the object remains stationary at the second location when the third image and the fourth image are captured by the third camera and the tracking camera, respectively.

6. The method of claim 1 , wherein the object is not attached to any of the first camera, the second camera, and the tracking camera.

7. Determining the first pose of the first camera based on the first image and the second image includes: determining a third pose of the first camera relative to the object at the first location using the first image; and and determining a fourth pose of the tracking camera relative to the object at the first location using the second image.

8. The method of claim 1 , wherein the relative spatial transformation indicates a position and orientation of the second camera relative to the first camera.

9. 10. The method of claim 1, wherein the object comprises a calibration pattern, and the calibration pattern is moved from the first location to the second location by a user in the operating room while the first camera, the second camera, and the tracking camera remain in a fixed position.

10. 1. A method performed by a surgical system including a first camera and a second camera located in an operating room, the method comprising: receiving a first image captured by the first camera, the first image representing a first field of view (FOV) of the first camera including a marker at a first location within the operating room; determining a first pose of the first camera based on the first image and the first locations of the markers; receiving a second image captured by the second camera, the second image representing a second FOV of the second camera that does not overlap with the first FOV and including the marker at a second location within the operating room; determining a second pose of the second camera based on the second image and the second locations of the markers; determining a relative spatial transformation between the first camera and the second camera based on the first pose and the second pose, wherein the first camera and the second camera remain stationary when the marker is moved from the first location to the second location; A method comprising:

11. The marker is fixedly coupled to an object including a calibration pattern, the object being at the first location captured in the first image and the second image, the method further comprising: determining, using a tracking device, a third location at which the marker is fixedly coupled to the calibration object; and determining the first pose of the first camera comprises: determining a third pose of the first camera relative to the calibration pattern while the object is at the first location; and determining a fourth pose of the tracking device relative to the calibration pattern using the third location of the marker.

12. Determining the fourth pose of the tracking device comprises: determining a fifth orientation of the tracking device relative to the marker according to the third location of the marker; deriving a sixth pose of the marker relative to the calibration pattern; and adjusting the fifth attitude according to the sixth attitude.

13. The method of claim 11 , wherein the marker and the calibration pattern are one integral unit.

14. The method of claim 11 , wherein the tracking device is an infrared (IR) sensor and the marker is an IR tag.

15. The method of claim 11 , wherein the tracking device is a camera and the marker is a visible pattern.

16. 1. A method performed by a surgical system including a first camera and a second camera located in an operating room, the method comprising: receiving a first image captured by the first camera, the first image representing a first field of view (FOV) of the first camera including an object at a first location within the operating room; determining a first pose of the first camera based on the first image and tracking markers in the operating room; receiving a second image captured by the second camera, the second image representing a second FOV of the second camera that does not overlap with the first FOV and having the object at a second location within the operating room; determining a second pose of the second camera based on the second image and the tracking marker; determining a relative spatial transformation between the first camera and the second camera based on the first pose and the second pose; A method comprising:

17. the object comprises a tracking device, and the method comprises: detecting the tracking marker using the tracking device while the object is at the first location, wherein the first pose is determined based on the detection of the tracking marker while the object is at the first location; 17. The method of claim 16, further comprising: detecting, using the tracking device, the tracking marker while the object is at the second location, and wherein the second pose is determined based on the detection of the tracking marker while the object is at the second location.

18. 20. The method of claim 17, further comprising: using the tracking device to track movement of the object from the first location to the second location, and wherein the second pose is determined based on the tracked movement of the object.

19. The method of claim 17 , wherein the tracking device and the object are one integrated unit.

20. The method comprises: determining a third pose of the tracking marker relative to the object based on the detection of the tracking marker while the object is at the first location; 18. The method of claim 17, further comprising: determining a fourth pose of the first camera relative to the object based on the first image, the first pose of the first camera being based on the third pose and the fourth pose.

21. Determining the third pose of the tracking marker comprises: determining a fifth pose of the tracking marker relative to the tracking device based on the detection of the tracking marker; receiving a sixth pose of the tracking device relative to the object; and adjusting the fifth attitude according to the sixth attitude.

22. 18. The method of claim 17, wherein the tracking device is a tracking camera, and wherein detecting the tracking marker while the object is at the first location includes receiving a third image captured by the tracking camera, the third image representing a third FOV of the tracking camera and including the tracking marker.

23. 18. The method of claim 17, wherein the tracking marker is an infrared (IR) tag and the tracking device is an IR sensor, or the tracking marker is a radio frequency (RF) tag and the tracking device is an RF sensor.

24. 17. The method of claim 16, wherein the object comprises a calibration pattern, and the calibration pattern is moved from the first location to the second location by a user or an autonomous robot in the operating room, while the first camera and the second camera remain in a fixed position.

Images