Method and system for calibrating a camera

Abstract

A method performed by a surgical system that includes a camera within an operating room. The system receives a first image from a first camera, the first image representing a first FOV having an object at a first position. The system receives a second image having the object at the first position from a tracking camera, and determines a first pose of the first camera based on the first image and the second image. The system receives a third image captured by the second camera, the third image representing a different second FOV and having the object at a second position. The system receives a fourth image captured by the tracking camera having the object at the second position, and determines a second pose of the second camera based on the third image and the fourth image. The system determines a relative spatial transformation between the first camera and the second camera based on the first pose and the second pose.

Description

Technical Field

[0001] Various embodiments of the present disclosure relate generally to surgical systems, and more particularly to surgical systems that calibrate one or more cameras. Background Art

[0002] Minimally invasive surgery (MIS), such as laparoscopic surgery, uses techniques designed to reduce tissue damage during surgical procedures. Laparoscopic procedures typically require making multiple small incisions in a patient's body (e.g., in the abdomen), through which several surgical tools (such as an endoscope, knife, grasper, and needle) are inserted. Gas is injected into the abdomen, inflating it and providing more space around the ends of the tools, making it easier for the surgeon to see (via the endoscope) and manipulate tissue at the surgical site. MIS can be performed faster and with less surgeon fatigue using a surgical robotic system in which surgical tools are operably attached to the distal ends of robotic arms, and a control system actuates the arms and their attached tools. When the surgeon manipulates a handheld user input device (UID), the ends of the tools simulate their position and orientation movements. A surgical robotic system can have multiple surgical arms, one or more of which have attached endoscopes, and other surgical arms have attached surgical instruments for performing certain surgical actions.

[0003] Control inputs from a user (e.g., a surgeon or other operator) are captured via one or more user input devices and then translated into control of the robotic system. For example, when a surgical tool is positioned at a surgical site on a patient, a tool driver having one or more motors may actuate one or more degrees of freedom of the surgical tool in response to user commands. Summary of the Invention

[0004] An increasing number of applications can benefit from the use of images (e.g., videos captured by multiple image sensors such as cameras and depth cameras). Examples of such applications may include telepresence, data collection for various computer training purposes (e.g., training motion controllers for autonomous vehicles), and event detection in operating rooms during surgical procedures. For example, multiple cameras may be used inside a hospital facility (such as an operating room) to monitor events and surgical workflows (e.g., monitoring surgical tasks performed by a surgeon) during surgical procedures. Cameras within such an environment can be used to monitor activity, detect events of interest, and analyze efficiency for training and improvement purposes. For example, in addition to detecting people or objects entering a room or space observed by a camera, the system may also be able to track the movement of an object, such as the movement of a gurney from one room to another, through multiple cameras. Due to maximizing the observable range of such a large room, some cameras may not have overlapping fields of view (FOVs), where one camera may be pointing towards a door, while another camera may be pointing in the opposite direction of the door.

[0005] When collecting video data from multiple cameras, it may be desirable to correlate the spatial information of these cameras (e.g., their positions and orientations relative to each other in space) to better enable analysis of the video data and to perform actions such as detecting activities or events. For example, when detecting the movement of an object (such as a patient's bed or gurney) within an operating room, the object may move out of the FOV of one camera and into the FOV of another camera. The spatial information indicating the relationship between the two cameras is then used to maintain continuous tracking of the object as it moves between the FOVs. This provides meaningful inferences and analysis of detected events.

[0006] To correlate spatial information, cameras can be spatially calibrated by calculating intrinsic and extrinsic parameters (or characteristics) of the cameras. Intrinsic parameters may include characteristics associated with a particular camera, such as physical properties of the camera. Examples of intrinsic characteristics may include at least one of the focal length of the camera (e.g., the camera's lens), the principal point (or optical center) of the camera (e.g., the camera's lens), the skew of the camera, and the distortion of the camera's lens. These intrinsic characteristics may represent a transformation from a two-dimensional (2D) image coordinate system (e.g., pixel coordinates in a 2D captured image) to a 3D coordinate system in the camera's three-dimensional (3D) space (relative to the camera as the origin). Extrinsic parameters may represent (or include) a transformation (e.g., position and / or orientation) from the camera's 3D coordinates to a global (or world) coordinate system. Specifically, extrinsic parameters may represent a relative spatial transformation (or pose) between two cameras. The process of estimating the pose of a camera relative to another camera can be performed through spatial extrinsic calibration. Conventional calibration methods require that multiple cameras have at least partially overlapping FOVs, where the overlapping portion includes a common reference point. However, these methods may not accurately calibrate cameras with non-overlapping fields of view, especially for such cameras that are part of a static (e.g., non-movable) image sensor setup. Therefore, there is a need for a surgical system that is configured to calibrate (e.g., static) image sensors with both overlapping and non-overlapping FOVs.

[0007] The present disclosure provides a surgical system comprising a first camera, a second camera, and a tracking camera located in one or more operating rooms, and configured to perform spatial calibration between the cameras. For example, two cameras may be located in a room (e.g., mounted on separate walls) and may be used for event detection in the operating room. Specifically, the system receives a first image captured by the first camera, the first image representing a first FOV of the first camera, the first FOV having an object, such as a calibration pattern, at a first position in the operating room. The system receives a second image captured by the tracking camera having the object at the first position, and determines a first pose of the first camera based on the first image and the second image. Specifically, the pose of the first camera may be relative to the tracking camera such that the position and / or orientation of the first camera is within the coordinate system of the tracking camera. The system receives a third image captured by the second camera, wherein the third image represents a second FOV of the second camera that does not overlap with the first FOV and has the object at a second position in the operating room. The system receives a fourth image captured by the tracking camera, wherein the fourth image has the object at the second position, and determines a second pose of the second camera based on the third image and the fourth image. Therefore, both the pose of the first camera and the pose of the second camera are within the coordinate system of the tracking camera. The system determines a relative spatial transformation, such as a pose, between the first camera and the second camera based on the first pose and the second pose. Therefore, the system is able to know the relative spatial information between the two cameras, which allows the system to effectively track movement through all observations between the cameras even when the cameras are arranged so that they cover different room locations and do not have overlapping FOVs.

[0008] In one embodiment, the tracking camera includes a third FOV that includes both the first position and the second position. For example, the tracking camera may be positioned on the ceiling of an operating room and include a wide-angle lens to allow for viewing of the interior of the operating room. In one embodiment, the tracking camera is stationary while the object moves from the first position to the second position. In another embodiment, the second camera and the tracking camera capture the third image and the fourth image, respectively, before the first camera and the tracking camera capture the first image and the second image, respectively. In one embodiment, the first camera and the tracking camera capture the first image and the second image, respectively, simultaneously, and the third camera and the tracking camera capture the third image and the second image, respectively, simultaneously.

[0009] In one embodiment, the object remains stationary at the first location when the first camera and the tracking camera capture the first image and the second image, respectively, and the object remains stationary at the second location when the third camera and the tracking camera capture the third image and the fourth image, respectively. In another embodiment, the object is not attached to any of the first camera, the second camera, and the tracking camera.

[0010] In one embodiment, determining the first pose of the first camera based on the first image and the second image includes: using the first image to determine a third pose of the first camera relative to the object at the first location; and using the second image to determine a fourth pose of the tracking camera relative to the object at the first location. In another embodiment, the relative spatial transformation indicates the position and orientation of the second camera relative to the first camera. In one embodiment, the object includes a calibration pattern that is moved by a user from the first location to the second location in the operating room while the first camera, the second camera, and the tracking camera remain in place.

[0011] The present disclosure provides a surgical system comprising a first camera, a second camera, and a (e.g., tracking) marker in an operating room for calibrating the two cameras. Specifically, the system receives a first image captured by the first camera, the first image representing a first FOV of the first camera, the first FOV including a marker at a first position in the operating room, and the system determines a first pose of the first camera based on the first image and the first position of the marker. The system receives a second image captured by the second camera, the second image representing a second FOV of the second camera that does not overlap with the first FOV and including the marker at a second position in the operating room, and the system determines a second pose of the second camera based on the second image and the second position of the marker. The system determines a relative spatial transformation between the first camera and the second camera based on the first pose and the second pose, wherein the first camera and the second camera remain stationary when the marker moves from the first position to the second position.

[0012] In one embodiment, the marker is fixedly coupled to an object comprising a calibration pattern, wherein the object is at the first position captured in the first image and the second image. The system uses a tracking device to determine a third position of the marker fixedly coupled to the calibration object, wherein determining the first pose of the first camera includes: determining a third pose of the first camera relative to the calibration pattern when the object is in the first position; and using the third position of the marker to determine a fourth pose of the tracking device relative to the calibration pattern. In another embodiment, determining the fourth pose of the tracking device includes: determining a fifth pose of the tracking device relative to the marker based on the third position of the marker; retrieving a sixth pose of the marker relative to the calibration pattern (e.g., from a memory of the surgical system); and adjusting the fifth pose based on the sixth pose. In one embodiment, the marker and the calibration pattern are an integral unit. In another embodiment, the tracking device is an infrared sensor and the marker is an infrared tag. In another embodiment, the tracking device is a camera and the marker is a visible pattern.

[0013] The present disclosure provides a surgical system comprising a first camera and a second camera in an operating room, wherein the system receives a first image captured by the first camera, the first image representing a first FOV including an object at a first position in the operating room, and the system determines a first pose of the first camera based on the first image and a tracking marker in the operating room (such as set on the ceiling of the operating room). The system receives a second image captured by the second camera, the second image representing a second FOV of the second camera that does not overlap with the first FOV and has the object at a second position in the operating room, and the system determines a second pose of the second camera based on the second image and the tracking marker. The system determines a relative spatial transformation between the first camera and the second camera based on the first pose and the second pose.

[0014] In one embodiment, the object includes a tracking device. The method comprises detecting the tracking marker using the tracking device when the object is in the first position, the first pose being determined based on the detection of the tracking marker when the object is in the first position; and detecting the tracking marker using the tracking device when the object is in the second position, the second pose being determined based on the detection of the tracking marker when the object is in the second position.

[0015] In one embodiment, the method includes: using the tracking device to track the movement of the object from the first position to the second position, wherein the second pose is determined based on the tracked movement of the object. In another embodiment, the tracking device and the object are an integral unit. In another embodiment, the system further performs the following operations: determining a third pose of the tracking marker relative to the object based on the detection of the tracking marker when the object is in the first position; and determining a fourth pose of the first camera relative to the object based on the first image, wherein the first pose of the first camera is based on the third pose and the fourth pose. In one embodiment, determining the third pose of the tracking marker includes: determining a fifth pose of the tracking marker relative to the tracking device based on the detection of the tracking marker; receiving a sixth pose of the tracking device relative to the object; and adjusting the fifth pose according to the sixth pose.

[0016] In one embodiment, the tracking device is a tracking camera, wherein detecting the tracking marker when the object is in the first position comprises receiving a third image captured by the tracking camera, the third image representing a third FOV of the tracking camera and including the tracking marker. In one embodiment, the tracking marker is an infrared (IR) tag and the tracking device is an IR sensor. In another embodiment, the tracking marker is a radio frequency (RF) tag and the tracking device is an RF sensor. In one embodiment, the object comprises a calibration pattern that is moved from the first position to the second position within the operating room by a user or an autonomous robot while the first camera and the second camera remain in position.

[0017] The above summary does not constitute an exhaustive list of all embodiments of the present disclosure. It is contemplated that the present disclosure includes all systems and methods that can be implemented by all suitable combinations of the various embodiments summarized above, as well as those disclosed in the following detailed description and particularly pointed out in the claims. Such combinations may have particular advantages not specifically recited in the above summary. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings, wherein like reference numerals indicate similar elements. It should be noted that references to "one" or "an" embodiment of the present disclosure are not necessarily to the same embodiment, and that they refer to at least one. In addition, for the sake of brevity and to reduce the total number of figures, a given figure may be used to illustrate features of more than one embodiment, and not all elements in a figure may be required for a given embodiment.

[0019] Figure 1 A pictorial view of an example surgical system in a surgical setting is shown.

[0020] Figure 2 is a block diagram of a surgical system for calibrating one or more cameras, according to one embodiment.

[0021] Figure 3 is a flow chart of a process for an embodiment of using a tracking device to calibrate one or more cameras.

[0022] Figure 4 Several stages are illustrated showing the calibration of two cameras using a tracking device.

[0023] Figure 5 is a flow chart of a process for an embodiment of calibrating one or more cameras using a tracking device and tracking markers.

[0024] Figure 6 Several stages are shown illustrating the calibration of two cameras using a tracking device and tracking markers.

[0025] Figure 7 is a flow chart of a process for an embodiment of calibrating one or more cameras using a tracking device and tracking markers.

[0026] Figure 8 Several stages are shown illustrating the calibration of two cameras using a tracking device and tracking markers. DETAILED DESCRIPTION

[0027] Several embodiments of the present disclosure will now be explained with reference to the accompanying drawings. Whenever the shapes, relative positions, and other embodiments of the parts described in a given embodiment are not explicitly defined, the scope of the present disclosure is not limited to the parts shown, which are shown for illustrative purposes only. In addition, although many details are set forth, it should be understood that some embodiments may be practiced without these details. In other cases, well-known circuits, structures, and technologies are not shown in detail to avoid obscuring the understanding of this specification. In addition, unless the meaning is clearly to the contrary, all ranges listed herein are deemed to include the endpoints of each range.

[0028] In one embodiment, "pose" may refer to the relative position and / or orientation of one object or reference frame relative to another object or reference frame. Specifically, the pose may be a six-degree-of-freedom (6DOF) variable that indicates the three-dimensional (3D) position and 3D orientation of an object relative to an object or reference frame in 3D space. For example, the position may include three parameters, such as the X, Y, and Z coordinates of a 3D Cartesian coordinate system, and the orientation may include three parameters, such as a set of three rotation angles around the three axes of the coordinate system (e.g., the yaw, pitch, and roll angles in Euler angles). In one embodiment, the pose may be a relative spatial transformation that may include a transformation matrix P that may include a translation vector indicating the position of the object relative to the origin of the reference frame in space and / or a rotation matrix indicating the orientation of the object relative to the reference frame.

[0029] Figure 1 A pictorial view of an example (e.g., laparoscopic) surgical system (which may be referred to hereinafter as a "system") 1 in a surgical setting (or operating room) is shown. The system 1 includes a user console 2, a control tower 3, and one or more surgical robotic arms 4 at a surgical robotic table (surgical table or surgical platform) 5. In one embodiment, the arms 4 may be mounted to a Figure 1 , or a combination thereof. The system 1 may include an operating table or bed on which the patient is positioned as shown in the example of . In one embodiment, at least some of the arms 4 may be configured differently. For example, at least some of the arms may be mounted on a ceiling, a side wall, or another suitable structural support (such as a cart separate from the operating table). The system 1 may incorporate any number of devices, tools, or accessories for performing surgery on the patient 6. For example, the system 1 may include one or more surgical tools (instruments) 7 for performing a surgical procedure. The surgical tool 7 may be an end effector attached to the distal end of the surgical arm 4 for performing the surgical procedure.

[0030] Each surgical tool 7 can be manually manipulated, robotically manipulated, or both during a surgical procedure. For example, a surgical tool 7 can be a tool for entering, viewing, performing surgical tasks, and / or manipulating the internal anatomical structures of a patient 6. In one embodiment, the surgical tool 7 is a gripper that can grasp the patient's tissue. The surgical tool 7 can be manually controlled by a bedside operator 8; or it can be robotically controlled via actuation of a surgical robot arm 4 to which it is attached. For example, when manually controlled, the operator can (e.g., physically) hold a portion of the tool (e.g., a handle) and can manually control the tool by moving the handle and / or pressing one or more input controls (e.g., buttons) on the tool (e.g., the handle of the tool). In another embodiment, when robotically controlled, the surgical system can manipulate the surgical tool based on user input (e.g., received via a user console 2, as described herein).

[0031] Generally, a remote operator 9 (such as a surgeon or other operator) can use the user console 2 to remotely manipulate the arm 4 and / or attached surgical tools 7, for example during a teleoperation. The user console 2 can be located in the same operating room as the rest of the system 1, such as in a conventional operating room. Figure 1 As shown. However, in other environments, the user console 2 may be located in an adjacent or nearby room, or it may be located in a remote location, for example, in a different building, city, or country. The user console 2 may include one or more components, such as a seat 10, one or more foot controls (or foot pedals) 13, one or more (handheld) user input devices (UIDs) 14, and at least one display 15. The display is configured to display a view of a surgical site within a patient 6, for example. The display may be configured to display image data (e.g., static images and / or video). In one embodiment, the display may be any type of display, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, a head-mounted display (HMD), etc. In some embodiments, the display may be a 3D immersive display for displaying a 3D (surgical) presentation. For example, during a surgical procedure, one or more endoscopes (e.g., an endoscopic camera) may capture image data of the surgical site, and the display presents the image data to the user in 3D. In one embodiment, the 3D display may be an autostereoscopic display that provides a 3D perception to the user without the need for special glasses. As another example, the 3D display may be a stereoscopic display that provides 3D perception through the use of glasses (eg, via active shutters or polarization).

[0032] In another embodiment, the display 15 may be configured to display at least one graphical user interface (GUI) that provides information and / or interactive content to assist the user in performing surgical procedures using one or more instruments in the surgical system 1. For example, some of the displayed content may include image data captured by one or more endoscopic cameras, as described herein. In another embodiment, the GUI may include selectable UI items that, when manipulated by the user, cause the system to perform one or more operations. For example, the GUI may include UI items as interactive content to switch control between robotic arms. In one embodiment, the system may include an input device (such as a keyboard, mouse, etc.) to interact with the GUI. In another embodiment, the user may use a UID 14 to interact with the GUI. For example, the user may manipulate the UID to navigate through the GUI (e.g., using a cursor), and to make a selection, the user may hover the cursor over a UI item and manipulate the UID (e.g., select a control or button). In some embodiments, the display may be a touch-sensitive display. In this case, the user may perform a selection by navigating and selecting via the touch display. In some embodiments, any method may be used to navigate and / or select UI items.

[0033] As shown, teleoperator 9 sits in chair 10 and views user display 15 while manipulating foot controls 13 and handheld UID 14 to remotely control one or more of arm 4 and surgical tool 7 (which is mounted on the distal end of arm 4).

[0034] In some embodiments, the bedside operator 8 can also operate the system 1 in an "in bed" mode, wherein the bedside operator (user) is now positioned to the side of the patient 6 and simultaneously manipulates the robotically driven tools (the end effector attached to the arm 4), for example, holding the handheld UID 14 and the manual laparoscopic tools with one hand. For example, the bedside operator's left hand can manipulate the handheld UID to control the robotic components, while the bedside operator's right hand can manipulate the manual laparoscopic tools. Thus, in these variations, the bedside operator can perform both robotic-assisted minimally invasive surgery and manual laparoscopic surgery on the patient 6.

[0035] During the example procedure (surgical operation), the patient 6 is prepared for surgery and covered with a sterile drape in a sterile manner to achieve anesthesia. When the arms of the system 1 are in a stowed or withdrawn configuration, the initial access to the surgical site (to facilitate access to the surgical site) can be performed manually. Once access is completed, the initial positioning or preparation of the system 1 (including its arms 4) can be performed. Then, the surgical operation continues, wherein the remote operator 9 at the user console 2 utilizes the foot control 13 and the UID 14 to manipulate various end effectors and possible imaging systems to perform the surgical operation. Manual assistance can also be provided at the operating bed or operating table by a bedside person (e.g., a bedside operator 8) wearing a sterile surgical gown. The bedside person can perform tasks on one or more of the robotic arms 4, such as retracting tissue, performing manual repositioning, and tool replacement. Non-sterile personnel may also be present to assist the remote operator 9 at the user console 2. When the procedure or surgical operation is complete, the system 1 and user console 2 may be configured or set to a state to facilitate completion of post-operative procedures, such as cleaning or disinfection, and entry or printing of healthcare records via the user console 2 .

[0036] In one embodiment, the remote operator 9 holds and moves the UID 14 to provide input commands to drive (move) one or more robotic arm actuators 17 (or drive mechanisms) in the robotic system 1 for remote operation. The UID 14 can be communicatively coupled to the rest of the system 1, for example, via the console computer system 16 (or host). The UID 14 can generate spatial state signals corresponding to the movement of the UID 14, such as the position and orientation of the handheld housing of the UID, and the spatial state signals can be input signals that control the movement of the robotic arm actuators 17. The system 1 can use control signals derived from the spatial state signals to control the proportional movement of the actuators 17. In one embodiment, the console processor of the console computer system 16 receives the spatial state signals and generates corresponding control signals. Based on these control signals that control how the actuators 17 are energized to drive the segments or connectors of the arm 4, the movement of the corresponding surgical tool attached to the arm can simulate the movement of the UID 14. Similarly, interaction between the teleoperator 9 and the UID 14 may generate, for example, a grasp control signal that causes the jaws of the grasper of the surgical tool 7 to close and grasp tissue of the patient 6 .

[0037] System 1 may include several UIDs 14, with corresponding control signals generated for each UID controlling the actuators and surgical tools (end effectors) of a corresponding arm 4. For example, a teleoperator 9 may move a first UID 14 to control the movement of an actuator 17 located in the left robotic arm, where the actuator responds by moving a linkage, gear, etc. in arm 4. Similarly, movement of a second UID 14 by the teleoperator 9 controls the movement of another actuator 17, which in turn drives other linkages, gears, etc. in system 1. System 1 may include a right arm 4 secured to a bed or table on the right side of a patient, and a left arm 4 located on the left side of the patient. Actuators 17 may include one or more motors controlled so that they drive rotation of a joint of arm 4 to, for example, change the orientation of an endoscope or gripper of a surgical tool 7 attached to the arm relative to the patient. The movement of several actuators 17 in the same arm 4 may be controlled by a spatial state signal generated from a particular UID 14. A UID 14 may also control the movement of a corresponding surgical tool gripper. For example, each UID 14 may generate a corresponding grasping signal to control the movement of an actuator (eg, a linear actuator) that opens or closes the jaws of a grasper at the distal end of the surgical tool 7 to grasp tissue within the patient 6 .

[0038] In some embodiments, communication between the surgical robotic table 5 and the user console 2 can be through a control tower 3, which can translate user commands received from the user console 2 (and more specifically from the console computer system 16) into robotic control commands sent to the arm 4 on the surgical table 5. The control tower 3 can also send status and feedback from the surgical table 5 back to the user console 2. The communication connection between the surgical table 5, the user console 2, and the control tower 3 can be via a wired link (e.g., fiber optic) and / or a wireless link, using any suitable data communication protocol from a variety of wireless data communication protocols, such as the Bluetooth protocol. Any wired connection can optionally be built into the floor and / or walls or ceiling of the operating room. The system 1 can provide video output to one or more displays (including displays within the operating room and remote displays accessible via the Internet or other network). The video output or feed can also be encrypted to ensure privacy, and all or part of the video output can be saved to a server or electronic health record system.

[0039] Figure 2is a block diagram of a surgical system 1 for calibrating one or more cameras according to one embodiment. The system includes one or more (e.g., electronic) components or elements, such as a controller 20, a sensor 21, a tracking device 22, a calibration object 23, a first camera 26, a second camera 27, a display 28, a speaker 29, and a memory 90. In one embodiment, the system may include more or fewer elements, such as having one or more (different) sensors and / or not including a calibration object, a speaker, or a display. In another embodiment, at least some of the elements of the system may be optional, such as a tracking device 22, tracking markers 92, and / or a tracking device 25. In one embodiment, the system may include one or more other elements not shown, such as having one or more robotic arms 4, such as Figure 1 shown.

[0040] In some embodiments, at least some of these elements may be part of a single electronic device. For example, the controller 20 and the memory 90 may be housed in a Figure 1 3. In another embodiment, at least some of these elements may be separate electronic devices relative to one another or part of these separate electronic devices. For example, tracking device 22 may be a separate electronic device located (e.g., mounted) within an operating room where at least a portion of surgical system 1 is located.

[0041] In one embodiment, the components of the surgical system can be communicatively coupled to the controller 20 and / or to each other to exchange digital data. For example, the controller can be configured to wirelessly communicate with one or more components (such as the tracking device 22) via a network and through a wireless connection. In one embodiment, the devices can communicate via any (computer) network, such as a wide area network (WAN) (e.g., the Internet), a local area network (LAN), etc., through which the devices can exchange data with each other and / or with one or more other electronic devices (such as a remote electronic server). In another embodiment, the network can be a wireless network, such as a wireless local area network (WLAN), a cellular network, etc., to exchange digital data. With respect to the cellular network, the controller (e.g., via a network interface) can be configured to establish a wireless (e.g., cellular) call, wherein the cellular network can include one or more cellular towers, which can be part of a communication network (e.g., a 4G Long Term Evolution (LTE) network) that supports data transmission (and / or voice calls) for electronic devices such as mobile devices (e.g., smartphones). In another embodiment, the devices can be configured to wirelessly exchange data via other networks, such as wireless personal area networks (WPAN) connections. For example, the controller 20 may be configured to establish a wireless communication link (connection) with an element (e.g., an electronic device including the sensor 21) via a wireless communication protocol (e.g., the Bluetooth protocol or any other wireless communication protocol). During the established wireless connection, the electronic device may send data (such as sensor data) as data packets (e.g., Internet Protocol (IP) packets) to the controller 20.

[0042] In another embodiment, the controller 20 can be communicatively coupled to one or more electronic devices via other methods. Specifically, the controller can be coupled to one or more devices via a wired connection. For example, the controller 20 can be coupled to the cameras 26 and 27 via a High Definition Multimedia Interface (HDMI) connection to receive image data as a video stream comprising a series of one or more still images captured by the cameras, and can be coupled to the display 28 (e.g., another HDMI connection) to provide one or more video streams to the display for display. In one embodiment, other video connections are also possible, such as a digital video interface (DVI), a serial digital interface (SDI) connector, a composite video connector, etc.

[0043] Each of first camera 26 and second camera 27 (e.g., complementary metal oxide semiconductor (CMOS image sensors)) can be an electronic device configured to capture video (and / or image) data (e.g., as a series of still images). Specifically, each of these cameras is arranged to capture an image of a respective field of view (FOV) representing at least a portion of the environment in which the camera is located. In some embodiments, each of these cameras may be located within an environment, such as an operating room, with both cameras having at least partially overlapping FOVs. In another embodiment, the cameras may be arranged such that neither camera has an overlapping FOV, wherein the images captured by the two cameras may not include similar (or identical) portions of the environment. For example, first camera 26 may have a FOV that captures one area of the operating room (e.g., directed toward one wall), while second camera 27 has a different FOV that captures a different, non-overlapping area of the operating room (e.g., directed toward a wall opposite the wall to which first camera 26 is directed). In another embodiment, the cameras may have non-overlapping FOVs due to being located in different environments (e.g., one camera in one room and the other in a different, adjacent room). In one embodiment, one or both of the cameras may include a wide angle lens to maximize the camera's observable range. More on FOV is described herein.

[0044] In one embodiment, one of the cameras can be an endoscope designed to capture video of a surgical site within a patient's body during a surgical procedure. In one embodiment, at least one of the cameras can be a monocular camera (e.g., having a single camera sensor that captures one digital (still) image at a time (e.g., as a video frame). In another embodiment, at least one of the cameras can be a stereoscopic (stereoscopic) camera having two (or more) lenses, each lens having a separate camera sensor for capturing a separate still image (e.g., to produce a separate video stream) to create a 3D video.

[0045] The sensor 21 may be any type of electronic device configured to detect (or sense) an environment (such as an operating room) and generate sensor data based on the environment. For example, the sensor 21 may include at least one microphone that may be configured to convert acoustic energy caused by propagation of sound waves into an input microphone signal. In another embodiment, the sensor may be a proximity sensor, such as an optical sensor, that may be configured to detect the presence of one or more objects within the environment and / or the proximity of the sensor to the one or more detected objects. In another embodiment, the sensor may be a temperature sensor that senses the ambient temperature within the environment in which the sensor is located as sensor data.

[0046] In some embodiments, the sensor 21 can be a motion sensor, such as an inertial measurement unit (IMU), which can be designed to measure the position and / or orientation of the sensor (e.g., relative to the surrounding environment). For example, the IMU can be coupled to the first camera 26 or a portion of the first camera and can be configured to detect camera motion, such as changes in the position and / or orientation of the camera relative to a reference point, which changes may be caused by an operator manipulating the camera to show different perspectives of the surrounding environment. In some embodiments, the motion sensor can be a camera that captures images used by the controller 20 to perform motion tracking operations (e.g., based on changes in the captured images).

[0047] The tracking device 22 can be any type of electronic device that is arranged to track and / or detect one or more objects and generate tracking (or sensor) data associated with the (e.g., detected) objects (e.g., their positions relative to the tracking device). Specifically, the tracking device can capture tracking (sensor) data indicating the position and / or orientation of the objects, which the controller can use to determine the pose of the tracked objects relative to the tracking device within the environment. For example, the tracking device can be designed to track the object relative to the tracking device (e.g., the position and / or orientation of the object in 3D space relative to the tracking device) as the object moves within a 3D space (such as an operating room, in which both the object and the tracking device can be located). As an example, as the object moves from one location to another, the tracking device can track the spatial transformation of the object relative to the tracking device. Thus, by tracking the object, the pose of the object that has moved from an original location to a new location can be determined (based on the tracking data); thus, the pose can be the relative spatial transformation of the object between the first location and the second location. More information about using a tracking device to estimate the pose of an object is described herein.

[0048] In one embodiment, the pose of an object can be a transformation indicating the position (e.g., as a translation vector or matrix) and / or orientation (e.g., as a rotation matrix) of the object relative to device 22. In another embodiment, the tracking device can include electronic components, such as one or more processors and memory, that can be configured to track an object within a threshold distance (e.g., a radial distance) of the tracked object. In some embodiments, the tracking device can track the object when the object is within the FOV and / or line of sight of the device. In another embodiment, the tracking device can be configured to determine the pose of the object tracked by device 22. In this case, the tracking device can be configured to send the pose (as digital data) to controller 20.

[0049] In one embodiment, the tracking device can be a video (e.g., motion tracking) camera. For example, tracking camera 22 can be the same or similar type of camera as cameras 26 and 27. In another embodiment, the tracking device can be a proximity sensor, such as an optical sensor (e.g., an infrared (IR) sensor), a magnetic sensor, etc., arranged to detect the presence of an object, the position and / or orientation of the object relative to the proximity sensor.

[0050] In another embodiment, the tracking device can be an electronic device capable of detecting the position and / or orientation of a tracking marker. For example, the tracking device can be a radio frequency (RF) position sensor (detector or receiver) capable of detecting (measuring) RF signals generated by an RF tag (or transmitter), such as an RF identifier (RFID). Such a device can determine the location of the marker by measuring the signal strength of the RF signal received from the marker. As another example, the tracking device can be an IR sensor capable of tracking an IR marker (or tag) within an environment. In another embodiment, the tracking device can be an electromagnetic sensor configured to track an electromagnetic marker (or tag).

[0051] In another embodiment, the tracking device can be any electronic device that can be configured to track an object moving in space based on data received from the object. For example, the tracking device can track the object based on location data received from the object. In this case, the object can include a location tracker (e.g., a global positioning system (GPS) device) that generates location data, which can be sent by the object to the tracking device. As another example, the tracking device can determine the location of the object in space based on one or more electronic signals (RF signals) received from the object. For example, the tracking device can determine the location of the object in space based on the signal strength (e.g., a received signal strength indicator (RSSI)) in the RF signal received from the object.

[0052] The calibration object 23 can be any type of object that can be used to perform external calibration of one or more cameras of the surgical system 1. In one embodiment, the external calibration of the camera determines external parameters, such as the pose of the camera relative to a reference frame. For example, a first camera 26 can capture an image of the calibration object, which image can include a calibration pattern 24, and the first camera determines the relative pose of the camera relative to the calibration pattern and / or vice versa. The camera can be configured to determine the external parameters as a transformation from the camera's 3D coordinate system to a global coordinate system based on the image captured by the camera, which transformation can be used to determine the pose of the first camera relative to the reference frame. In one embodiment, the calibration pattern can be any type of (e.g., visual) pattern, such as a checkerboard pattern or a grid pattern.

[0053] In one embodiment, the calibration object 23 can be an object that can include a tracking device 25, which can be an electronic device of the same (or similar) type as the tracking device 22. In another embodiment, the calibration object can include a tracking marker 92, which can be any type of marker designed to be tracked by the tracking device 22. Specifically, the tracking marker can be similar (or the same) as at least one object described herein that can be tracked by a tracking device. For example, the marker can be a visual marker with a unique visual design. As another example, the marker can be any type of tag that can be tracked by any type of optical sensor, such as an IR tag that can be detected by an IR sensor. As another example, the marker can be a tracking tag (e.g., an RFID) that is arranged to be tracked by the tracking device 22. As yet another example, the tracking marker can be an electronic device that can be configured to be tracked by another electronic (e.g., tracking) device. For example, the tracking marker can include a GPS device, as described herein.

[0054] In some embodiments, the tracking markers 92, calibration pattern 24, and / or tracking device 25 can be part of (or fixedly attached to) the calibration object 23. For example, each (or at least one) of these elements can form an integral unit with the calibration object 23. In one embodiment, at least some of the elements of the object can be positioned at different locations on (or around) the object 23. For example, the calibration pattern 24 can be positioned on a side (or in front of) the object, while the tracking markers and / or tracking device 25 can be positioned on the top side (or around) the object. Specifically, the pattern can be positioned so as to be visible to a camera in front of the object, while the tracking markers 92 can be positioned so as to be visible to the tracking device 22, which can be located above the object (e.g., on the ceiling of an operating room).

[0055] In one embodiment, at least some of the elements of the surgical system may be stationary (e.g., fixedly coupled) within the operating room environment. For example, first camera 26 may be attached to a wall or object within a room, while second camera 27 may be attached to another wall or object within the room (or a different room). As another example, both cameras may be attached to the same object (e.g., Figure 13 of the control tower 3 of the surgical system 1), while pointing in different (or similar) directions. As another example, the tracking device 22 may be attached to the ceiling of the operating room. In another embodiment, the calibration object 23 may be a movable object and may be separated from (not attached to) at least some of the other elements of the surgical system 1. For example, the calibration object may be separated from the first camera 26, the second camera 27, and / or the tracking device 22. Specifically, the object may be sized to be held by a user so that the user can place the object at one location on a surface (e.g., an operating table), pick up the object, and place the calibration object at a different location on that surface (or another surface).

[0056] Memory (e.g., non-transitory machine-readable storage medium) 90 can be any type of electronic storage device. For example, the memory can include read-only memory, random access memory, CD-ROM, DVD, magnetic tape, optical data storage device, flash memory device, and phase change memory. Although the memory is shown as being separate from the controller 20, the memory can be part of the controller 20 (e.g., internal memory of the controller). As shown, the memory 90 includes one or more poses 91 of one or more objects that can be used for camera calibration of one or more cameras of the surgical system 1. Specifically, the memory can include poses associated with the calibration object 23. As described herein, the object 23 can include a calibration pattern 24 that can be located on a surface (e.g., the front side) (or a portion of such surface) of the object and has tracking markers 92. In this case, the tracking markers 92 can be located on another surface (or a portion of such surface) of the object, such as a surface on the top side. The memory can include the pose (or spatial relative transformation) of the calibration pattern 24 relative to the tracking marker 92 and / or vice versa. Thus, the poses 91 can include at least one pose that describes a translation and / or rotation of the calibration pattern relative to the tracking marker. For example, when the object is square, the pose may indicate a 90° rotation about an axis extending parallel to the front side to which the calibration pattern is attached and the top side of the object to which the marker is attached. In one embodiment, the pose 91 of the elements of the calibration object may be predefined because the elements may be fixedly attached (e.g., manufactured as a unit).

[0057] The controller 20 can be any type of electronic component that can be configured to perform one or more computing operations. For example, the controller can be a dedicated processor, such as an application-specific integrated circuit (ASIC), a general-purpose microprocessor, a field-programmable gate array (FPGA), a digital signal controller, or a set of hardware logic structures (e.g., filters, arithmetic logic units, and a dedicated state machine). The controller 20 is configured to spatially calibrate one or more cameras of the surgical system using sensor data (such as images captured by the cameras). Such operations allow the surgical system 1 to efficiently and effectively track movement through the different fields of view of the different cameras. More information about how the controller performs calibration operations is described herein.

[0058] In one embodiment, at least some of the operations performed by the controller may be performed in response to user input. For example, the controller may be configured to receive user input via one or more input (electronic) devices (not shown) (such as a keyboard, a mouse) or a peripheral computer device (such as a tablet computer). In another embodiment, user input may be received via a touch-sensitive display (e.g., display 28), which may display a graphical user interface (GUI) having one or more user interface (UI) items, wherein the device may generate one or more control signals (as user input) based on the user touching a portion of the display that is presenting the user items. In another embodiment, at least some of these operations may be performed automatically, such as without user intervention. For example, the surgical system 1 may perform a calibration operation when a camera is connected to the controller 20 (e.g., for the first time). As another example, the system may instruct the user (or operator) of the surgical system 1 (e.g., by presenting a notification, such as displaying a pop-up notification on the display 28) to calibrate one or more cameras.

[0059] Figure 3 、 Figure 5 and Figure 7 is a flow chart of a process for calibrating one or more cameras of surgical system 1. In one embodiment, at least some of the operations in at least some of these processes may be performed before the surgical system 1 is used by an operator to perform a surgical (e.g., laparoscopic) procedure on a patient. In another embodiment, at least some of these operations may be performed intraoperatively (e.g., while the operator is performing a surgical procedure). In another embodiment, at least some of these operations may be performed postoperatively based on image data captured by one or more cameras during the surgical procedure. In some embodiments, at least some of the operations in these processes may be performed by the surgical system described herein (e.g., the controller 20 of the surgical system).

[0060] Now go to Figure 3, which illustrates a flow chart of a process 30 for an embodiment of using a tracking device 22 to calibrate one or more cameras 26 and / or 27. Specifically, process 30 describes using a tracking device 22 to calibrate a first camera 26 and a second camera 27 based on a calibration object 23 (e.g., a calibration pattern 24 of the calibration object), with the tracking device located in one or more operating rooms. The controller 20 receives a first image captured by the first camera 26, the first image representing (or including) a first field of view (FOV) of the first camera 26, the first FOV having an object at a first position (at block 31). For example, the calibration object 23 may be placed at a first position in front of the first camera (e.g., on the surface of an operating table). In one embodiment, the object 23 may be positioned such that the calibration pattern 24 is within the first FOV. For example, the calibration pattern may be in the first position, as described herein. In this case, the image captured by the camera may include the pattern and / or a portion of the environment surrounding the pattern. In one embodiment, once the calibration object (e.g., the calibration pattern of the calibration object) is placed at the first position, the first camera 26 may capture the first image. Specifically, the camera may capture the first image in response to user input (e.g., a user pressing a button on an input device that sends a control signal to controller 20 causing the camera to capture an image). In another embodiment, the camera may detect that calibration object 23 has been placed in the camera's FOV (e.g., based on object recognition) and, in response to detecting the object, may capture the first image.

[0061] Controller 20 receives a second image captured by tracking device 22, the second image showing the object at a first location (at block 32). Specifically, the tracking device may be a tracking camera separate from the first and second cameras and may have a field of view (FOV) that includes the first location and, therefore, the calibration pattern 24 of calibration object 23 placed at the first location. In one embodiment, the respective FOVs of the two cameras may include the first location, with the two cameras being located in different locations within the operating room and stationary. In this case, the two images may include different perspectives of the calibration object. For example, the FOV of the first camera may include the calibration pattern facing forward, while the FOV of the target device may include an angled, top-down (bird's-eye) view of the calibration pattern. In another embodiment, first camera 26 and tracking device 22 may capture their respective images simultaneously. For example, both cameras may capture their respective images in response to receiving a user input. In another embodiment, the tracking device may capture an image after (or before) the first camera captures the image.

[0062] The controller determines a first pose of the first camera based on the first image and the second image (at block 33). Specifically, the controller sets the pose P C1' is determined as a relative spatial transformation of the first camera relative to the tracking device 22 (e.g., the reference frame of the tracking device). For example, when the tracking device is a camera, the origin of the coordinate system can be defined at the projection center of the camera's lens, and one or more axes (orientations) can be defined based on the optical axis and the plane of the camera's imaging sensor. In one embodiment, to determine P C1 ', the controller may perform a graph-based approach in which the pose is based on at least two other (e.g., estimated or known) poses of one or more elements within the environment (such as, for example, the calibration object 23, the tracking device 22, and the first camera 26 within the operating room). For example, the other poses may be relative to the same reference frame (or object) within the same coordinate system, wherein the determined first pose may be determined as a transformation from one pose to the other pose (or a difference between the two poses) within the space (e.g., 3D space) of the (e.g., 3D) coordinate system.

[0063] In one embodiment, to determine P C1 ', the controller may determine the spatial relationship between the first camera and the calibration object, and determine the spatial relationship between the tracking device and the calibration object. For example, the controller may determine the camera (or third) pose P of the calibration pattern 24 relative to the first camera when the object is in the first position. C1 , and determining the tracking device (or fourth) pose P of the calibration pattern 24 relative to the tracking device when the object is in the first position TD1 In some embodiments, one or more poses (e.g., P C1 and / or P TD1 ) may be the inverse spatial transformation of the spatial transformation determined by the controller based on the images of each camera, as described herein. For example, the controller may first determine the relative spatial transformation of the calibration pattern 24 relative to the first camera 26 based on one or more images captured by the first camera 26. The controller 20 may determine the inverse spatial transformation of the calibration pattern 24 relative to the first camera 26 based on the one or more images captured by the first camera 26. C1 is defined as the inverse of the determined relative spatial transformation between the pattern and the camera so that the pose is relative to the calibration object (e.g., in order to change the reference frame from the camera to the calibration pattern). In this case, P C1 and P TD1 Both may be relative to the calibration pattern 24 of the object and thus lie within the 3D coordinate system of the pattern.

[0064] In one embodiment, the pose P may be determined using one or more (predefined) graphical models of the object's calibration pattern. C1 and P TD1In one embodiment, the surgical system (memory 90) may include one or more 3D (e.g., computer-aided design (CAD)) models of one or more calibration patterns (and / or calibration objects), each of which may be a graphical mathematical coordinate-based representation (e.g., as one or more basis (or B) splines, such as non-uniform rational basis splines (NURBS)). In one embodiment, each model may include (or correspond to) one or more different poses (e.g., positions and / or orientations) of the calibration pattern within a 3D coordinate system (e.g., a Cartesian coordinate system) relative to a reference system (e.g., at or on a camera). In order to determine P C1 , the controller 20 may be configured to match the 3D model to the calibration pattern captured within the first image, which may provide an estimate of the spatial relative transformation of the calibration pattern with respect to the camera. In one embodiment, the pose of the calibration pattern may be estimated based on one or more (e.g., intrinsic) parameters (e.g., which are determined during calibration of the camera and / or retrieved from memory 90) and the matched 3D model. Specifically, the 3D model may represent the calibration pattern within the 3D model space, and the controller uses the one or more (e.g., intrinsic) parameters of the camera to define the position and orientation of the calibration pattern relative to the position of the camera. In one embodiment, the controller may apply the intrinsic parameters and the 3D model to a pose model (e.g., as input thereto), which produces a pose of the calibration pattern as output. In one embodiment, the controller may P C1 is determined as the inverse transformation of the estimated pose of the calibration pattern relative to the first camera 26, as described herein. In another embodiment, the pose model may generate P C1 In another embodiment, the controller 20 may use any known (or future) method to determine P from one or more images captured by the first camera 26. C1 For example, the controller may determine P based on any external calibration algorithm that may be executed by the controller 20. C1 In some embodiments, the controller may perform the following steps with respect to determining P C1 One or more of the operations described herein determine P TD1 .

[0065] In one embodiment, the controller 20 may be configured to utilize P C1 and P TD1 The two determine the first position P of the first camera relative to the tracking device. C1As described herein, the controller 20 may perform a graph-based approach in which relative spatial transformations are projected within a 3D coordinate system relative to the same reference, which in this case may be the tracking device 22. For example, each spatial transformation may be a link between a reference (e.g., an origin) and an object (element) that has been translated and / or rotated relative to the reference. In this case, P TD1 and P C1 ' can be relative to the same reference (ie tracking device 22), and P C1 It can be a transformation from the first camera to the calibration pattern. In this case, the controller can be based on P TD1 and P C1 The relationship between P C1 '. For example, the controller can C1 'Determined to be P C1 The inverse of P TD1 In another embodiment, the controller can be configured to C1 and P TD1 When P is determined relative to the calibration object, C1 For example, in this case, the two poses may be relative to the calibration pattern 24, and the controller may position the first pose P C1 'Determine from P TD1 to P C1 Therefore, the controller can transform P C1 'Determined to be P TD1 The inverse of P C1 combination.

[0066] In another embodiment, the controller 20 may determine P by performing an external calibration method. C1 and / or P TD1 In this external calibration method, one or more images are used to estimate the intrinsic parameters and / or extrinsic parameters of the camera. For example, the controller may determine the extrinsic parameters of the camera 26 using a calibration pattern of a calibration object captured by the first camera 26 within the first image. For example, the controller may estimate PC1 using a 3D-2D correspondence between one or more known (e.g., coplanar) 3D points of the calibration pattern 24 (e.g., corners of a checkerboard pattern, centroids of circles in a circular grid, etc.) and their corresponding 2D projections on the image plane of the captured image. Since the projection depends on the one or more 3D points, the controller may use an optimization process (e.g., a nonlinear process such as the Levenberg-Marquardt algorithm) to determine (or estimate) one or more (e.g., intrinsic and / or extrinsic) parameters. Based on (or using) the extrinsic parameters, the controller may estimate PC1. C1In one embodiment, the controller may determine the relative spatial transformation of the calibration object relative to the camera by performing an external calibration method using one or more images captured by the tracking device. TD1 .

[0067] Controller 20 receives a third image captured by a third camera (e.g., camera 27), which represents the second FOV of the second camera that does not overlap with the first FOV and has the calibration object at a second location (at block 34). Specifically, calibration object 23 may be moved (transported) from the first location to the second location within the operating room. Controller 20 receives a fourth image captured by the tracking camera, which shows the object at the second location (at block 35). Thus, the object may move between the two locations while first camera 26, second camera 27, and target camera 22 remain stationary within the operating room. In this case, the target camera's FOV may include both the first and second locations without having to move (adjust its position and / or orientation). In one embodiment, target camera 22 may include a wide-angle lens that expands the camera's FOV to include both locations. In this case, target camera 22 and second camera 27 may capture their respective images simultaneously. In one embodiment, the target camera's images may be captured sequentially. For example, the first camera 26 and the target camera 22 may capture the first and second images, respectively, before the second camera 27 and the target camera 22 capture the third and fourth images. Conversely, the third and fourth images may be captured before the first and second images. In another embodiment, the calibration object 23 may remain stationary while the first camera 26, the second camera 27, and the target camera 22 capture their respective images. For example, the object may remain stationary at the first location while the first camera and the tracking camera capture the first and second images, respectively, and may remain stationary at the second location while the third camera and the tracking camera capture the third and fourth images, respectively.

[0068] The controller 20 determines a second pose of the third camera (e.g., the second camera 27) based on the third image and the fourth image (at block 36). For example, the controller 20 may perform operations similar to those described with respect to block 33. For example, the controller may determine a pose P of the calibration object 23 at the second position relative to the second camera 27. C2 , and the tracking camera pose P of the calibration object 23 at the second position relative to the tracking camera can be determined TD2 The controller can be based on P TD2 and P C2 To determine the pose P of the second camera 27 relative to the target camera 22 C2'. For example, the controller may perform operations similar to those described with respect to block 33 to determine P C2 '.

[0069] The controller determines a relative spatial transformation (e.g., pose) between the first camera and the third camera based on the first pose and the second pose (at block 37). Specifically, the controller determines the relative transformation between the first camera 26 and the second camera 27. In one embodiment, the relative spatial transformation can be one or more external parameters of at least one of the first camera and the second camera. For example, the spatial transformation T of the first camera relative to the second camera C1 It can be a relative transformation indicating the position (e.g., translation matrix or vector) and / or orientation (e.g., rotation matrix) of the first camera relative to the second camera. Similarly, the spatial transformation T of the second camera is C2 Can be relative to the first camera. In one embodiment, any of the relative spatial transformations (such as T C1 ) can be P C1 ' and P C2 'The result of the combination. In some embodiments, due to P C1 ' and P C2 Both are relative to the same reference (ie the tracking camera), so at least one of the determined spatial transformations can be reversed. For example, T C1 It can be P C1 ' and / or P C2 As described herein, the determined relative spatial transformation may include similar (or identical) data / information to the determined pose, such as including a translation matrix and / or a rotation matrix.

[0070] Some embodiments may perform variations of process 30 described herein. For example, certain operations of the process may not be performed in the exact order shown and described. Certain operations may not be performed in a series of consecutive operations, and different specific operations may be performed in different embodiments. As described herein, the controller may receive first camera 26 and tracking camera 22 to determine the pose of the first camera relative to the tracking camera. In another embodiment, each of these cameras may capture one or more images of a calibration object. For example, when determining the respective poses of these cameras, the controller may apply an extrinsic calibration method. In this case, the method may require multiple images of the object, wherein the position (and / or orientation) of the object in each image may be adjusted. In this case, the two cameras may capture one or more images of the object at different positions and / or orientations. In one embodiment, at least some of the operations described in process 30 may be performed sequentially. For example, at least some of the operations in blocks 31 through 33 may be performed before performing at least some of the operations in blocks 34 through 37. In other words, the controller 20 may first determine the pose of the first camera and then determine the pose of the second camera.

[0071] Figure 4 Several stages 40 to 42 are illustrated that show the use of the tracking device 22 to calibrate the first camera 26 and the second camera 27 (e.g., to determine extrinsic parameters). Figure 3 30 . Each stage shows a first camera and a second camera and a tracking device, which in this example is a tracking camera in an operating room 43. As shown, the first camera 26 and the second camera 27 each point in opposite directions. Specifically, the first camera has a FOV 44 pointing in one direction, while the second camera has a FOV 45 pointing in the opposite direction. Thus, the two FOVs can be non-overlapping, which may be the case when the two cameras are placed around the operating room to maximize the observable range. In addition, the tracking camera 22 is disposed above the first camera and the second camera and has a FOV 46 that includes the first camera and the second camera. For example, the tracking camera 22 can be fixed to the ceiling of the operating room, while the first camera and the second camera can be fixed to (e.g., corresponding) objects in the operating room 43, such as an operating table, a cabinet, etc., or can be independent cameras (e.g., supported by a tripod, as shown).

[0072] The first stage 40 shows the estimation of the pose P of the first camera relative to the tracking device 22 using the calibration object 23. C1 In particular, the figure shows the calibration object 23 in a (first) position 95 , wherein the object or more particularly the calibration pattern 24 is in the FOV 44 of the first camera 26 and in the FOV 46 of the tracking device 22 .

[0073] In addition, this stage shows the pose P of the first camera C1 ' can be based on P which can be estimated using images captured by the first camera 26 and the tracking device 22 C1 and P TD1 Specifically, this stage illustrates a graph-based approach in which two poses are estimated in order to determine P C1 '. Specifically, P C1 is shown as a link or (projection) projected from the first camera 26 to the calibration object 23 (the calibration pattern 24 of the calibration object), and P TD1 is shown as another link projected from the tracking device 22 to the calibration object 23 (the calibration pattern 24 of the calibration object). In addition, this stage shows P as a link between the tracking device 22 and the first camera 26. C1 '. In one embodiment, P C1 'Based on P TD1 With P C1 The translation and / or rotation differences between the two can be determined graphically. For example, P C1 'Based on P C1 The inverse of P TD1 combinations as described herein.

[0074] The second stage 41 shows the use of the calibration object 23 to estimate the pose P of the second camera relative to the tracking camera 22 C2 ' is an illustration of a method of moving the calibration object from position 95 to another (second) position 96, which is now within the FOV 45 of the second camera 27. In addition, the new position 96 is still within the FOV 46 of the tracking device. Thus, the calibration object 23 with the calibration pattern 24 has been moved from the first position 95 to the second position 96 and is within the FOV 45. In one embodiment, the movement can be performed by a person (e.g., a technician). For example, the technician may have picked up the object and moved it between positions. As another example, the object 23 can be mounted on a movable platform (e.g., a cart) and moved between positions manually by a person or automatically (e.g., without user intervention) via one or more motors or actuators of the movable cart. In addition, as shown, although the calibration object 23 has been moved, the first camera 26, the second camera 27, and the tracking camera 22 remain in place (stationary) within the operating room 43.

[0075] In addition, the second stage shows the pose as a graphical link between the second camera 27 and the calibration object 23 and between the tracking device 22 and the calibration object. Specifically, the figure shows the P between the calibration object (the pattern of the calibration object) and the second camera 27. C2, and P between the calibration object and the tracking device 22 TD2 In addition, this stage shows P as the link between the second camera 27 and the tracking device 22 C2 ', which can be based on P C2 'with P TD2 In one embodiment, at least some of the poses shown in this stage may be determined in a similar manner to the poses in the first stage 40 described herein.

[0076] The third stage 42 shows the determination of the pose P C1 ' and P C2 ' result. Specifically, this stage shows the spatial relative transformation of both the first camera 26 and the second camera 27 (which are links sent from the tracking camera), both of which are in the same coordinate system (for example, both are relative to the tracking camera 22). Therefore, the controller 20 can determine the spatial relative transformation T between the two cameras based on the difference between the two poses. C1 and T C2 For example, T C1 can be the spatial relative transformation of the first camera relative to the second camera, and T C2 It can be a spatial relative transformation of the second camera with respect to the first camera.

[0077] As shown herein, the FOV 46 of the tracking camera 22 includes both positions 95 and 96, as well as the first camera 26 and the second camera 27. In one embodiment, the FOV 46 may not include one or both of the cameras, but may only have the position of the object in its field of view. In another embodiment, the position and / or orientation of the cameras may be different, such as when the cameras are positioned in a circular manner (e.g., when there are two or more cameras that the surgical system is being calibrated).

[0078] Figure 51 is a flow chart of a process 50 for calibrating one or more cameras (such as cameras 26 and / or 27) using a tracking device 22 and a tracking marker 92 that may be part of a calibration object 23. Specifically, the tracking marker 92 may be part of a calibration object 23 that includes a calibration pattern 24 (fixedly coupled to the calibration object). Process 50 begins with the controller 20 receiving a first image captured by a first camera 26, the first image representing a first FOV of the first camera that includes the marker 92 at a first location in the operating room (at block 51). Specifically, the calibration object may be in a first location (e.g., placed at the location by a user), wherein the captured image includes at least a portion of the calibration pattern. In one embodiment, the tracking marker 92 may be located on a different side (or surface) than the calibration pattern and, therefore, may not be within the FOV of the first camera. Consequently, the captured image may not include the tracking marker.

[0079] The controller 20 determines a first position P of the first camera 26 based on the first image and the first position of the marker. C1 ' (at block 52). Specifically, the controller may determine P based on one or more poses estimated using (at least) the first image and / or detection of the marker. C1 For example, the controller may determine the pose P of the calibration pattern 24 relative to the first camera when the object 23 is in the first position. C1 , as described herein. Additionally, the controller may determine a pose P of the calibration pattern of the calibration object relative to the tracking device 22 TD1 . In one embodiment, to this end, the controller may determine the spatial relationship between the tracking device 22 and the tracking marker 92. In one embodiment, the tracking device may be a proximity (or position detection) sensor configured to detect position data (e.g., position and / or orientation) of the tracking marker. Specifically, the controller may be configured to use the tracking / sensor data generated by the tracking device to detect (determine) the position (and / or orientation) of the tracking marker 92 relative to the device. For example, when the tracking marker includes an RF transmitter (or RF tag, such as an RFID) and the tracking device includes an RF sensor, the RF sensor may be arranged to sense RF signals generated (or reflected) by the RF tag and generate sensor data based on these signals. In another embodiment, the tracking device may be an IR sensor and the marker may be an IR tag. In this case, the tracking device may generate sensor data based on infrared signals generated by the IR sensor, which are reflected back from the IR tag that indicates the position characteristics of the marker. In one embodiment, the sensor data may indicate the position and / or orientation of the marker relative to the tracking device. The controller may be configured to determine (estimate) the pose P of the tracking marker relative to the tracking device 22 based on the determined position of the marker (e.g., using the sensor data).TD1 In this case, the controller can use this information to estimate the P of the tracking mark 92 relative to the tracking device. TD1 ' (e.g., by generating a transformation matrix that indicates the translation and / or rotation of the marker relative to the tracking device). In one embodiment, P TD1 ' may be an inverse transform of the pose of the tracking marker 92 determined relative to the tracking device, as described herein.

[0080] In another embodiment, one or more images may be used to estimate P TD1 ', as described herein. For example, when the tracking device 22 is a tracking camera, the tracking device may be arranged to capture one or more images of the tracking marker 92, which may include one or more visible patterns (or objects), and the controller may determine P TD1 This may be the case when the calibration pattern 24 is not within the field of view of the tracking device 22.

[0081] As described herein, tracking markers 92 may be located on the calibration object and may be used to determine P C1 The calibration pattern 24 may be located at different positions on the object. For example, the marker may be attached to the top surface of the object, while the calibration pattern may be attached to the front surface of the object. In this case, the controller may be configured to determine P by considering the relative spatial relationship between the marker and the pattern. TD1 For example, the controller may retrieve (or receive) the pose P of the tracking marker relative to the calibration pattern. TM (e.g., retrieved from pose 91 in memory 90), and can be based on P TM To adjust P TD1 '. As described herein, a graph-based approach can be used to perform this adjustment. Thus, the controller can C1 'Estimated to be P TD1 and P C1 combinations as described herein.

[0082] Controller 20 receives a second image captured by second camera 27, the second image representing a second FOV of the second camera that does not overlap with the first FOV of the first camera and including a marker at a second location in the operating room at block 53. Specifically, the object to which the marker is fixedly attached may be moved from one location to another in the operating room, as described herein.

[0083] The controller 20 determines a second pose P of the second camera 27 based on the second image and the second position of the marker. C2 ' (at block 54). In one embodiment, the controller may perform operations similar to those described for block 52 to determine P C2For example, the controller may determine the pose P of the calibration pattern 24 of the object at the second position relative to the second camera. C2 , and determine the P of the calibration pattern 24 relative to the tracking device 22 TD2 In one embodiment, P may be estimated based on tracking data generated by a tracking device when a tracking marker (e.g., a calibration object for the tracking marker) moves from a first position to a second position. TD2 More information about tracking markers is described herein. Thus, the controller may be configured to estimate a pose P of the tracking marker relative to the device 22. TD2 ', and can be relative to P TM to adjust the pose.

[0084] The controller 20 is based on the first position P C1 ' and the second pose P C2 ' to determine the relative spatial transformation between the first camera 26 and the second camera 27 (at block 55). Thus, in this embodiment, the controller uses the positions of the tracking markers on the calibration object to determine the relative spatial transformation, while the cameras and tracking device remain stationary as the markers (e.g., the calibration object for the markers) move between positions within the operating room.

[0085] Figure 6 Several stages 60 to 62 are illustrated showing the calibration of the first camera 26 and the second camera 27 using the tracking device 22 and the tracking markers 92 of the calibration object 23. In particular, the figure shows Figure 5 50 . Each of these stages shows a first camera 26, a second camera 27, and a tracking device 22, which can be any type of electronic device capable of detecting the position of an object (and / or a marker) within a space, such as an IR sensor or a camera. Additionally, the figure shows that the calibration object 23 includes a calibration pattern 24 and a tracking marker 92. Specifically, the figure shows the calibration pattern 24 at a location 93 on (or around) the calibration object 23, while the tracking marker 92 is at another location 94. For example, the calibration pattern can be on the front surface of the object, while the marker can be on the top surface of the object, as described herein.

[0086] The first stage 60 shows the estimation of P C1 ', the estimate can be based on P C1 With P TD144 , as described herein. Specifically, the figure shows the calibration object 23 at position 95, which includes the calibration pattern 24 in the FOV 44 of the first camera 26. Additionally, a tracking marker 92 (such as an IR tag) may not be within the FOV 44, or may be at least partially within the FOV 44. The stage also shows the tracking marker 92 within the FOV 46 of the tracking device 22. This may be the case when the tracking device is a tracking camera. In another embodiment, the FOV 46 may represent the (e.g., radial) distance and / or line of sight within which the tracking device can track one or more objects, such as the marker 92. For example, when the tracking device is a proximity sensor, the FOV 46 may be the distance at which the sensor can detect the proximity of the object.

[0087] This first stage 60 illustrates the determination by the surgical system to estimate P C1 ' pose (e.g., using a graph-based approach, as described herein). Specifically, the figure shows the P between the calibration pattern 24 and the first camera 26. C1 , and shows the P between the calibration pattern 24 and the tracking device 22 TD1 , which is determined based on the position of tracking marker 92 detected by tracking device 22, as described herein. However, because the calibration pattern may not be in the line of sight (or within FOV 46) of tracking device 22, for example, the controller determines P based on the detected position of tracking marker 92. TD1 To this end, the controller determines P TD1 ', which is the pose of the tracking marker 92 relative to the tracking device 22, since the tracking marker may be within the FOV 46 of the device 22. The figure also shows the P between the calibration pattern 24 and the tracking marker 92 TM In this case, the controller can graphically TD1 Determined to be P TD1 ' and P TM combinations (e.g., products) of .

[0088] The second stage 61 shows the estimation of P C2 ', the estimate can be based on P C2 With P TD2 46 . In one embodiment, the controller may perform operations similar to those described for the first stage 60. In one embodiment, the tracking marker 92 may remain within the FOV 46 as it moves from the first position 95 to the second position 96.

[0089] This stage shows the P between the calibration pattern 24 and the second camera 27 C2 , and shows the P between the calibration pattern 24 and the tracking device 22 TD2 In one embodiment, to determine P TD2 , the controller can determine the relationship between the tracking marker 92 and the tracking device 22. Therefore, the figure shows the P between the tracking marker 92 and the tracking device 22. TD2 ', which can be determined based on the position of the marker detected by the tracking device, as described herein. Since the tracking marker 92 and the calibration pattern are fixedly coupled to the calibration object 23, the relative spatial transformation P TM Remaining the same as shown in the first stage 60. In one embodiment, P TD2 'It is different from P TD1 ', because the tracking mark (eg, the calibration object of the tracking mark) has moved between position 95 and position 96. The controller then sets P TD2 Determined to be P TD2 'with P TM In some embodiments, tracking marker 92 may remain within the FOV (e.g., line of sight) 46 of tracking device 22 at (and / or between) locations 95 and 96. In another embodiment, FOV 46 may be a threshold distance from tracking device 22, such as a radial distance, within which both locations may be located.

[0090] The third stage 62 shows the determination of the pose P C1 ' and P C2 'results, where the controller can use a graph-based approach by identifying P C1 'with P C2 'The translation and / or rotation between them determines the spatial relative transformation T of the first camera relative to the second camera C1 Or the spatial relative transformation T of the second camera relative to the first camera C2 , as described in this article.

[0091] Figure 5 and Figure 6 The calibration operation described in the foregoing is a calibration method for determining the spatial relationship between at least two cameras of the surgical system 1 by tracking the position of a marker using a tracking device. In one embodiment, the method is capable of effectively tracking a marker from a first position to a second position by the tracking device, so that the method has high accuracy and high tolerance with respect to environmental conditions (e.g., lighting) in the operating room.

[0092] The calibration methods described so far involve estimating the relative spatial transformation between spatially distributed static image sensors (e.g., RGB cameras or RGBD depth sensors). Specifically, Figures 3 to 6 Both of the described calibration methods can be "outside-in" tracking calibration methods, in which a static (e.g., stationary) sensor (such as tracking device 22) within the operating room tracks a calibration object 23 (e.g., the calibration object's markers 92) between two or more locations. In another embodiment, the surgical system can be configured to perform one or more "inside-out" tracking calibration methods, in which one or more tracking devices can be moved within the operating room so as to track the movement of the tracking devices based on detected position changes of one or more (e.g., stationary) objects relative to the tracking devices. Specifically, Figure 7 and Figure 8 This inside-out tracking calibration approach is involved.

[0093] Now go to Figure 7 , which shows a flow chart of a process 70 for an embodiment of performing inside-out calibration of one or more cameras (such as the first camera 26 and / or the second camera 27) using a tracking device and tracking markers. Specifically, the process 70 describes calibrating the cameras using the tracking device 25, which is coupled to the calibration object 23 (or a portion of the calibration object) so as to detect the movement of the tracking device 25 relative to individual tracking markers (e.g., tracking marker 98, such as Figure 8 The tracking marker 98 can be a static (stationary) object in the operating room, while the calibration object 23 including the tracking device 25 can be movable.

[0094] The process 70 begins with the controller receiving a first image captured by the first camera 26, the first image representing a first FOV of the first camera 26, the first FOV including an object at a first location in the operating room (at block 71). Thus, as described herein, the FOV of the first camera may include the calibration object 23 at the first location, which may be placed by a user at the first location. The controller determines a first position P of the first camera 26 based on the first image and the tracking markers 98 in the operating room. C1 ” (at block 72). Specifically, the controller 20 may determine P based on one or more estimated poses. C1 ”, which may be the pose of the first camera 26 relative to the tracking marker 98. Thus, unlike the method described herein where the controller determines a first pose of the first camera 26 relative to the tracking device 22 (e.g., P C1Unlike at least some of the other calibration methods of FIG'), the controller 20 may determine the pose of the first (and second) cameras relative to the tracking marker 98. Thus, P C1 " may be determined based on detection of the tracking marker by the tracking device 25 of the calibration object, which may be at the first position. More on these operations is described herein.

[0095] In one embodiment, the controller 20 may determine P as follows C1 Specifically, the controller 20 may use (at least) the first image to determine P C1 , as described herein. The controller may be based on P C1 and the pose P of the calibration object 23 (eg, the calibration pattern 24 of the calibration object) relative to the tracking marker 98. TM1 To determine P C1 ” (or determine it as a combination of the above two). Specifically, P TM1 ] may represent a spatial relative transformation of calibration object 23 when in a first position relative to tracking marker 98 at its specific (stationary) position within the operating room. In one embodiment, the transformation between the calibration object and tracking marker 98 may change as the position of the object within the operating room changes. More information on transformations is described herein.

[0096] In one embodiment, to determine P TM1 , the controller can determine the position P of the tracking marker 98 relative to the tracking device 25 TD1 Specifically, the controller detects the tracking mark 98 using the tracking device 25 when the calibration object is in the first position, and determines P based on the detection of the tracking mark when the calibration object (e.g., the tracking device 25 of the calibration object) is in the first position. TD1 In one embodiment, the tracking device 25 may generate sensor data upon detecting tracking markers indicating the position and / or orientation of the tracking device within the operating room, and using this data, the controller 20 may determine P TD1 In some embodiments, the controller may perform one or more operations described herein to determine P based on sensor data. TD1 For example, when the tracking device 25 is a tracking camera, the tracking marker 98 is detected based on one or more images captured by the tracking camera, and based on the one or more images, the controller can determine a pose of the tracking marker relative to the tracking device 25, as described herein.

[0097] In one embodiment, to produce P TM1, the controller may be configured to take into account the relative spatial relationship between the tracking device 25 and the calibration pattern 24 of the calibration object 23. In one embodiment, the calibration pattern 24 is at one location on the calibration object 23, while the tracking device 22 is at another location on the calibration object 23. For example, the tracking device may be at or near a location on the top surface of the calibration object so as to have an upward FOV (or line of sight) to observe tracking markers that may be above the calibration object, while the pattern may be located toward one side of the object. Thus, the controller may retrieve the pose P of the calibration pattern relative to the tracking device 25 TD (e.g., retrieved from pose 91 in memory 90). In one embodiment, the controller can adjust P TD1 "To consider P TD The transformation of P TM1 For example, P TM1 Based on P TD1 ”(the inverse of) and P TD In one embodiment, using P TM1 and P C1 , the controller can be configured to P C1 "Identified as P TM1 and P C1 (the inverse of) combination.

[0098] Returning to process 70, the controller 20 may track the movement of the object from the first location to the second (different) location (at block 73). Specifically, the controller may use the tracking device 25 coupled to the calibration object 23 to track the movement based on sensor data generated by the device. For example, as the user picks up the calibration object and moves it to different locations, the tracking device may (e.g., periodically, such as every second) capture sensor data indicating changes in the position and / or orientation of the tracking device (and / or the object). For example, when the tracking device 25 is a motion camera arranged to capture one or more images of an environment, the controller may be configured to execute a camera motion tracking algorithm (e.g., a simultaneous localization and mapping (SLAM) algorithm, a visual odometry (VO) algorithm, etc.) to track the movement of the device based on the movement of one or more points within a sequence of one or more video frames (images) captured by the camera. As described herein, the relative spatial transformation of the tracking device at the second location may be based on the tracked movement of the object relative to the tracking marker from the first location in space.

[0099] The controller 20 receives a second image captured by the second camera 27, the second image representing a second FOV of the second camera that does not overlap with the first FOV and having an object at a second location in the operating room (at block 74). For example, the calibration object 23 may be manually moved by a user from the first location to another location within the operating room. In another embodiment, the object may be moved autonomously (e.g., by an autonomous robot within the room). In one embodiment, the tracking marker 98 may remain within the field of view of the tracking device as the object moves.

[0100] The controller 20 determines a second pose P of the second camera 27 based on the second image and the tracking marker. C2 ” (at block 75). In one embodiment, the controller may perform at least some of the operations described herein (such as with respect to block 72 of the process) to determine P C2 For example, the controller may determine the pose P of the calibration pattern 24 of the object at the second position relative to the second camera. C2 , and determine P TM2 , which is the pose of the calibration pattern 24 of the calibration object relative to the tracking marker 98 when the object is in the second position. In one embodiment, the controller may execute the method described herein relative to P TM1 At least some of the operations described herein determine P TM2 In one embodiment, the controller may determine P based on the detected movement of the tracking device. TM2 For example, the controller can use the tracking data to determine P TD2 ”, the tracking data may indicate changes in position and / or orientation of the tracking device as the tracking device moves in space, and the tracking data is generated by the tracking device when the object moves from a first position to a second position. In this case, P TD2 " may be the relative spatial transformation of the tracking device 25 from the first position to the second position. In another embodiment, P may be estimated based on the tracking data and based on sensor data generated by the tracking device 25 at the second position. TD2 For example, the tracking data may indicate one or more translation parameters and / or one or more rotation parameters relative to the tracking marker from a first position to a second position, and other translation and / or rotation parameters may be determined based on sensor data captured by the tracking device at the second position. The controller may determine the position of the tracking marker based on P C2 and P TM2 To determine P C2 ", as described in this paper. The controller is based on the first pose P C1 ” and the second pose P C2 ” to determine the relative spatial transformation between the first camera 26 and the second camera 27 (at block 76). For example, to determine T using a graph-based approachC1 , the controller can determine the C2 ” to P C1 ", as described herein. In one embodiment, the controller may be configured to determine one or more external parameters of the first camera and the second camera based on the tracked movement of the cameras. For example, the tracked movement may indicate the distance between the first camera 26 and the second camera 27. In this case, the controller may determine the distance from T based on the determined distance. C1 and T C2 One or more translation parameters of the associated translation vector.

[0101] Figure 8 Several stages 80 to 82 are illustrated showing the use of the tracking device 25 of the calibration object 23 to detect the tracking markers 98 and thereby calibrate the first camera 26 and the second camera 27. Specifically, the figure shows Figure 7 7. Each of these stages shows a first camera 26, a second camera 27, and a tracking marker 98 disposed within one or more operating rooms 43. Specifically, the tracking marker 98 is shown at a position 99, which is above the two cameras. For example, the tracking marker 98 may be attached to the ceiling of the operating room, while the two cameras are supported on the floor of the room. In another embodiment, the tracking marker 98 may be located at a different location within the room (e.g., on a wall of the room). Additionally, the figure shows a calibration object 23 that includes a calibration pattern 24 at a first location 93 on the object 23 and a tracking device 25 at another location 97 on (or at) the object. In one embodiment, each of these locations may be on a different surface of the object (e.g., location 93 on a forward-facing surface, while location 97 is on a top-facing surface).

[0102] The first stage 80 shows the use of the tracking device 25 of the calibration object 23 to estimate the pose P of the first camera relative to the tracking marker 98 C1 Specifically, the figure shows a calibration object at position 95, where the calibration pattern 24 is in the FOV 44 of the first camera 26. In addition, the FOV 46 of the tracking device 25 may include a tracking marker 98, so that the tracking device can detect the marker. The stage also shows poses, for example, as links between elements in the operating room, which are determined by the controller 20 to estimate P C1 ”, which is shown as a link between the tracking marker 98 and the first camera 26. Specifically, this stage shows the P between the calibration pattern 24 and the first camera 26 C1 , which is determined based on the image captured by the first camera 26, as described herein. In addition, this example also shows PTD1 ”, which is the pose of the tracking marker 98 relative to the tracking device, which can be determined by the controller based on sensor data from the tracking device 25, as described herein. TD is also shown and may be the pose of the calibration pattern 24 relative to the tracking device 25. In one embodiment, the controller 20 may use these two poses to determine P TM1 , which is shown as a link between the tracking marker 98 and the calibration pattern. As described herein, the controller can plot these poses in the coordinate system of the calibration pattern 24 to determine P C1 ”, which can be based on P TM1 and P C1 combination.

[0103] The second stage 81 shows the estimation of P C2 "Example, P C2 ” is an estimated pose of the second camera 27 relative to the tracking marker 98, which may be performed when the calibration object 23 is in a new position. Specifically, the figure shows that the object 23 has moved from position 95 to position 96, which is within the FOV 45 of the second camera. As described herein, the controller 20 may perform operations similar to those described with respect to the first camera 80 to determine one or more poses to estimate P C2 For example, the figure shows the pose P of the second camera 27 relative to the calibration pattern 24. C2 The figure also shows the P between the tracking device 25 and the tracking mark 98. TD2 ”, which may be estimated based on sensor data captured by the tracking device 25 at the location 96 and / or based on tracking data captured by the tracking device as it moves from the first location 95 to the second location 96. In one embodiment, the controller may estimate the value of P according to TD To adjust P TD2 ”, in order to estimate P TM2 Then, the controller is based on P C2 and P TM2 To determine P C2 ”, as described in this article.

[0104] The third stage 82 shows the determination of the pose P C1 ” and P C2 ” results, where the controller can use a graph-based approach to identify P C1 ” and P C2 The spatial relative transformation T of the first camera relative to the second camera is determined by the translation and / or rotation between C1 Or the spatial relative transformation T of the second camera relative to the first camera C2 , as described in this article.

[0105] Some embodiments may perform variations of process 30 described herein. For example, specific operations of the process may not be performed in the exact order shown and described. Specific operations may not be performed in a series of continuous operations, and different specific operations may be performed in different embodiments. In one embodiment, at least some of the operations described herein may be performed once to calibrate two or more cameras. For example, at least some of the operations and / or at least some of the elements illustrated herein with dashed boundaries may be optional, as described herein. In another embodiment, process 70 may be performed to calibrate the first camera 26 and the second camera 27 under the initial settings of the cameras. Once calibrated, the surgical system 1 may use the estimated external parameters (relative spatial transformation) when performing visual processing operations (such as motion detection). In another embodiment, at least some of these operations may be performed periodically to recalibrate one or more cameras over a period of time. For example, over time, one or more parameters of a camera may drift (e.g., gradually change). Therefore, the surgical system may perform at least some of these operations after a period of time to ensure that the parameters are accurate.

[0106] As described herein, the surgical system may perform at least one of processes 30, 50, or 70 to calibrate one or more cameras. In one embodiment, the surgical system may perform two or more calibration methods described herein to optimize the estimated extrinsic parameters. For example, the surgical system may perform Figure 3 The process 30 and Figure 7 The surgical system may perform at least some of the operations of process 70 and determine the relative spatial transformation between the first camera 26 and the second camera 27 based on the two sets of transformations for each camera. In one embodiment, when performing two or more calibration methods, the surgical system may average the estimated relative spatial transformation between the cameras.

[0107] In one embodiment, at least some of these operations may be performed to calibrate the first camera 26 and the second camera 27 of the surgical system 1. Specifically, the surgical system may perform one or more calibration operations described herein to calibrate three or more cameras. In this case, the surgical system 1 may be configured to estimate the relative spatial transformations between pairs of cameras. For example, in the case of three cameras (cameras "A," "B," and "C"), the surgical system may use one or more of the calibration methods described herein to determine a first pair of relative spatial transformations between cameras A and B, a second pair of relative spatial transformations between cameras A and C, and / or a third pair of relative spatial transformations between cameras B and C. In some embodiments, the surgical system may use a graph-based approach to determine each pair of spatial transformations between cameras within the same coordinate system, as described herein.

[0108] In one embodiment, the operations described herein can allow a surgical system to calibrate cameras with non-overlapping fields of view. In another embodiment, at least some of the operations described herein can be used to calibrate cameras with at least partially overlapping fields of view.

[0109] As described herein, tracking devices 22 and 25 are arranged to detect tracking markers 92 and 98, respectively, in order to track the movement of the calibration object. Specifically, the tracking devices can track the movement by capturing sensor data at the location where the calibration object is placed. In another embodiment, the tracking devices can track the movement of the calibration object as it moves around an operating room and / or between one or more operating rooms. As an example, referring to Figure 8 , as an object (e.g., carried by a user) moves between locations, tracking device 25 can track the movement (e.g., the movement of the calibration object). This can allow cameras in different operating rooms to be calibrated relative to each other. For example, a surgical system can use tracking device 25 to track the movement of a first location 95 in one room to a second location 96 in another room. In this case, the surgical system can use the tracked movement of the object to account for the distance between the two cameras. In one embodiment, the surgical system can use sensor data from either (or both) tracking devices 22 and 25 to track the movement of the calibration object to more accurately and efficiently estimate one or more poses described herein.

[0110] In one embodiment, at least some of the operations described herein may be performed in "real time," meaning that the surgical system may perform the operations while the one or more cameras to be calibrated are capturing one or more images. In one embodiment, the surgical system may provide feedback to the user while the user is calibrating the cameras. As an example, go to Figure 8 Once the user places the calibration object at position 95, the surgical system may provide a notification to the user indicating whether the object is within the FOV of the first camera. For example, once placed, the surgical system may provide a pop-up notification indicating whether the calibration object needs to be adjusted. Once P is estimated C1 ”, the surgical system may provide a notification alerting the user to move the object to the second position 96. In this case, the surgical system may use the speaker 29 to output an audible notification: “Please move the calibration pattern in front of the next camera.” Providing feedback ensures that the surgical system is calibrated efficiently and effectively.

[0111] As described herein, the surgical system 1 can be configured to estimate the pose of an object relative to another object based on sensor data. In another embodiment, the estimated (or overall) pose can be based on one or more estimated poses, such that the estimated overall pose can be an average of the one or more poses. For example, referring to Figure 4, the controller may be configured to estimate one or more P C1 '. In this case, the surgical system 1 can be based on the method for estimating P C1 'Other pose changes to estimate multiple P C1 For example, when the calibration pattern is in a first orientation relative to the camera 1, the surgical system may estimate a first P C1 ', and when the calibration pattern is in a second orientation relative to the camera 1, the surgical system can estimate a second P C1 '. In one embodiment, the calibration pattern can be in the same position 95, but with different orientations. C1 'Based on the first P C1 ' and the second P C1 ' (eg, the average of the two). In one embodiment, estimating a pose based on an average of one or more poses may reduce noisy data and may produce a better overall pose estimate.

[0112] As previously explained, one embodiment of the present disclosure can be a non-transitory machine-readable medium (such as a microelectronic memory) having stored thereon instructions that program one or more data processing components (collectively referred to herein as "processors") to automatically (e.g., without user intervention) use one or more images to calibrate one or more cameras, as described herein. In other embodiments, some of these operations may be performed by specific hardware components containing hard-wired logic. These operations may alternatively be performed by any combination of programmed data processing components and fixed, hard-wired circuit components.

[0113] To assist the Patent Office and any reader of any patent issuing on this application in interpreting the claims appended hereto, Applicant wishes to note that unless the words "means for" or "step for" are expressly used in a particular claim, Applicant does not intend that any of the appended claims or claim elements invoke 35 U.S.C. § 112(f).

[0114] Although certain embodiments have been described and shown in the accompanying drawings, it should be understood that such embodiments are merely illustrative of the present disclosure and not restrictive, and that the present disclosure is not limited to the specific constructions and arrangements shown and described, as various other modifications may occur to those skilled in the art. Accordingly, this description should be regarded as illustrative and not restrictive.

[0115] In some embodiments, the present disclosure may include language such as, “at least one of [element A] and [element B]”. The language may refer to one or more of the elements. For example, “at least one of A and B” may refer to “A”, “B”, or “A and B”. Specifically, “at least one of A and B” may refer to “at least one of A and at least one of B” or “at least one of A or B”. In some embodiments, the present disclosure may include language such as, “[element A], [element B], and / or [element C]”. The language may refer to any one of these elements or any combination thereof. For example, “A, B, and / or C” may refer to “A”, “B”, “C”, “A and B”, “A and C”, “B and C”, or “A, B, and C”.

Claims

1. A method performed by a surgical system comprising a first camera, a second camera, and a tracking camera located within an operating room, the method comprising: receiving a first image captured by the first camera, the first image representing a first field of view (FOV) of the first camera, the first field of view (FOV) having an object at a first location in the operating room; receiving a second image captured by the tracking camera, the second image having the object at the first location; determining a first pose of the first camera based on the first image and the second image; receiving a third image captured by the second camera, the third image representing a second FOV of the second camera that does not overlap with the first FOV and having the object at a second location in the operating room; receiving a fourth image captured by the tracking camera, the fourth image having the object at the second location; determining a second pose of the second camera based on the third image and the fourth image; as well as A relative spatial transformation between the first camera and the second camera is determined based on the first pose and the second pose.

2. The method according to claim 1, wherein The tracking camera includes a third FOV that includes both the first position and the second position, and wherein the tracking camera is stationary when the object moves from the first position to the second position.

3. The method according to claim 1, wherein The second camera and the tracking camera capture the third image and the fourth image, respectively, before the first camera and the tracking camera capture the first image and the second image, respectively.

4. The method according to claim 1, wherein The first camera and the tracking camera simultaneously capture the first image and the second image, respectively, and the third camera and the tracking camera simultaneously capture the third image and the second image, respectively.

5. The method according to claim 1, in, The object remains stationary at the first location while the first camera and the tracking camera capture the first image and the second image, respectively, and Wherein, when the third camera and the tracking camera capture the third image and the fourth image, respectively, the object remains stationary at the second position.

6. The method according to claim 1, wherein The object is not attached to any of the first camera, the second camera, and the tracking camera.

7. The method according to claim 1, wherein Determining the first pose of the first camera based on the first image and the second image includes: determining a third pose of the first camera relative to the object at the first location using the first image; and The second image is used to determine a fourth pose of the tracking camera relative to the object at the first location.

8. The method according to claim 1, wherein The relative spatial transformation indicates a position and orientation of the second camera relative to the first camera.

9. The method according to claim 1, wherein The object comprises a calibration pattern that is moved by a user from the first position to the second position within the operating room while the first camera, the second camera, and the tracking camera remain in place.

10. A method performed by a surgical system comprising a first camera and a second camera located within an operating room, the method comprising: receiving a first image captured by the first camera, the first image representing a first field of view (FOV) of the first camera, the first field of view (FOV) including a marker at a first location in the operating room; determining a first pose of the first camera based on the first image and the first position of the marker; receiving a second image captured by the second camera, the second image representing a second FOV of the second camera that does not overlap with the first FOV and including the marking at a second location in the operating room; determining a second pose of the second camera based on the second image and the second position of the marker; as well as A relative spatial transformation between the first camera and the second camera is determined based on the first pose and the second pose, wherein the first camera and the second camera remain stationary when the marker moves from the first position to the second position.

11. The method according to claim 10, wherein: The marker is fixedly coupled to an object comprising a calibration pattern, wherein the object is at the first position captured in the first image and the second image, wherein the method further comprises determining, using a tracking device, a third position at which the marker is fixedly coupled to the calibration object, wherein determining the first pose of the first camera comprises: determining a third pose of the first camera relative to the calibration pattern when the object is in the first position; and A fourth pose of the tracking device relative to the calibration pattern is determined using the third position of the marker.

12. The method according to claim 11, wherein Determining the fourth pose of the tracking device includes: determining a fifth pose of the tracking device relative to the marker based on the third position of the marker; retrieving a sixth pose of the marker relative to the calibration pattern; and The fifth pose is adjusted according to the sixth pose.

13. The method according to claim 11, wherein The marking and the calibration pattern are an integrated unit.

14. The method according to claim 11, wherein The tracking device is an infrared (IR) sensor and the marker is an IR tag.

15. The method according to claim 11, wherein The tracking device is a camera and the marker is a visible pattern.

16. A method performed by a surgical system comprising a first camera and a second camera located within an operating room, the method comprising: receiving a first image captured by the first camera, the first image representing a first field of view (FOV) of the first camera, the first field of view (FOV) including an object at a first location in the operating room; determining a first pose of the first camera based on the first image and tracking markers within the operating room; receiving a second image captured by the second camera, the second image representing a second FOV of the second camera that does not overlap with the first FOV and having the object at a second location in the operating room; determining a second pose of the second camera based on the second image and the tracking markers; as well as A relative spatial transformation between the first camera and the second camera is determined based on the first pose and the second pose.

17. The method according to claim 16, wherein The object comprises a tracking device, wherein the method further comprises: detecting the tracking marker using the tracking device when the object is in the first position, wherein the first pose is determined based on the detection of the tracking marker when the object is in the first position; and The tracking marker is detected using the tracking device when the object is in the second position, wherein the second pose is determined based on the detection of the tracking marker when the object is in the second position.

18. The method according to claim 17, further comprising: Movement of the object from the first position to the second position is tracked using the tracking device, wherein the second pose is determined based on the tracked movement of the object.

19. The method according to claim 17, wherein The tracking device and the object are an integral unit.

20. The method according to claim 17, in, The method further includes determining a third pose of the tracking marker relative to the object based on the detection of the tracking marker when the object is in the first position; and A fourth pose of the first camera relative to the object is determined based on the first image, wherein the first pose of the first camera is based on the third pose and the fourth pose.

21. The method according to claim 20, wherein Determining the third pose of the tracking marker includes: determining a fifth pose of the tracking marker relative to the tracking device based on the detecting of the tracking marker; receiving a sixth pose of the tracking device relative to the object; and The fifth pose is adjusted according to the sixth pose.

22. The method according to claim 17, wherein The tracking device is a tracking camera, wherein detecting the tracking marker when the object is in the first position includes receiving a third image captured by the tracking camera, the third image representing a third FOV of the tracking camera and including the tracking marker.

23. The method according to claim 17, wherein The tracking tag is an infrared (IR) tag and the tracking device is an IR sensor, or the tracking tag is a radio frequency (RF) tag and the tracking device is an RF sensor.

24. The method according to claim 16, wherein The object comprises a calibration pattern that is moved by a user or an autonomous robot from the first position to the second position within the operating room while the first camera and the second camera remain in position.

Images