Camera control system and method for computer-assisted surgery system

Abstract

A camera control system may access surgical data for a surgical procedure including the performance of one or more operations performed by a computer-assisted surgery system. The camera control system may identify an event associated with the surgical procedure based on the surgical data and may determine a location associated with the event based on the surgical data. In response to determining the location of the event, the camera control system may direct automatic adjustment of a field of view of a camera to capture a specific field of view of the location associated with the event.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 799,729, filed on January 31, 2019, and entitled “CAMERA CONTROL SYSTEMS AND METHODS FOR A COMPUTER-ASSISTED SURGICAL SYSTEM,” the contents of which are incorporated herein by reference in their entirety. Background Art

[0003] During a computer-assisted surgical procedure (such as a minimally invasive surgical procedure using a computer-assisted surgical system), a camera can capture images of a scene of the computer-assisted surgical system, for example, outside the patient's body. The computer-assisted surgical system can display the captured images to medical personnel (e.g., to a surgeon and / or other members of the surgical team) to provide visualization of the computer-assisted surgical system. The visualized images assist the medical personnel in performing the surgical procedure. However, the user typically must manually position and move the camera to capture the desired view, which takes time and interrupts the flow of the surgical procedure. Therefore, the camera may not be fully utilized during the surgical procedure. Summary of the Invention

[0004] An exemplary system may include a memory storing instructions and a processor, the processor being communicatively coupled to the memory and configured to execute the instructions to access surgical data for a surgical session, the surgical session comprising performance of one or more operations performed by a computer-assisted surgery system; based on the surgical data, identifying an event associated with the surgical session; based on the surgical data, determining a location associated with the event; and, in response to the determination of the location of the event, directing automatic adjustment of a field of view of a camera to capture a particular field of view associated with the location associated with the event.

[0005] An exemplary method may include accessing, by a camera control system, surgical data for a surgical procedure that includes performance of one or more operations performed by a computer-assisted surgery system; identifying, by the camera control system, an event associated with the surgical procedure based on the surgical data; determining, by the camera control system, a location associated with the event based on the surgical data; and directing, by the camera control system, automatic adjustment of a camera's field of view to capture a specific field of view of the location associated with the event in response to the determination of the location of the event.

[0006] An exemplary system includes a memory storing instructions and a processor, the processor being communicatively coupled to the memory and configured to execute the instructions to receive user input indicating workflow segmentation of a surgical procedure from a user device communicatively paired with a computer-assisted surgery system during a surgical procedure; identify an event associated with the surgical procedure based at least in part on the user input; determine a location associated with the identified event based on surgical data for the surgical procedure; and, in response to the determination of the location of the event, direct automatic adjustment of a field of view of a camera to capture a particular field of view of the location associated with the event. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings illustrate various embodiments and are part of the specification. The embodiments shown are examples only and do not limit the scope of the present disclosure. Throughout the drawings, the same or similar reference numerals represent the same or similar elements.

[0008] Figure 1 An exemplary computer-assisted surgery system according to the principles described herein is shown.

[0009] Figure 2 Another exemplary computer-assisted surgery system according to the principles described herein is shown.

[0010] Figure 3 An exemplary camera control system according to the principles described herein is shown.

[0011] Figure 4 shows a schematic diagram of a circuit according to the principles described herein. Figure 3 An exemplary embodiment of a camera control system is shown in FIG.

[0012] Figure 5 shows a schematic diagram of a circuit according to the principles described herein. Figure 3 Another exemplary embodiment of a camera control system is shown in .

[0013] Figures 6 to 8 Illustrative manners in which events may be identified based on surgical data and / or additional data according to the principles described herein are shown.

[0014] Figures 9 and 10 An exemplary event location table according to the principles described herein is shown.

[0015] Figure 11 An exemplary camera control method according to the principles described herein is shown.

[0016] Figure 12 An exemplary computing system according to the principles described herein is shown. DETAILED DESCRIPTION

[0017] A camera control system and method for a computer-assisted surgery system are described herein. As will be explained in more detail below, an exemplary camera control system can access surgical data for a surgical procedure that includes the performance of one or more operations performed by a computer-assisted surgery system. The camera control system can identify an event associated with the surgical procedure based on the surgical data and can determine a location associated with the event based on the surgical data. In response to determining the location of the event, the camera control system can direct automatic adjustment of the field of view of the camera to capture a specific field of view of the location associated with the event. In some examples, images captured by the camera can be displayed by a display device associated with the computer-assisted surgery system.

[0018] To illustrate, during a minimally invasive surgical procedure performed at a surgical facility using a computer-assisted surgery system, a camera control system can detect that a first manipulator arm attached to a surgical instrument is colliding with a second manipulator arm, thereby preventing movement of the tip of the surgical instrument coupled to the first manipulator arm. The camera control system can thus determine that the location of the collision event is the position of the first manipulator arm and direct automatic adjustment of a camera positioned within the surgical facility to capture a specific field of view for the first manipulator arm. The video from the camera can be displayed by a display device associated with a user control system used by the surgeon. Thus, the surgeon can quickly understand why the tip of the surgical instrument is no longer moving and can quickly identify corrective actions, all without having to leave their position at the user control system.

[0019] Various benefits can be provided by the camera control systems and methods described herein. For example, the systems and methods described herein can automatically guide the adjustment of a camera positioned within a surgical facility to capture images based on a detected context of a surgical procedure. Automatic adjustment of the camera can improve the efficiency of surgical procedures by eliminating interruptions caused by manually adjusting the camera. Additionally, the systems and methods described herein can provide relevant contextual information to surgical team members in real-time during a surgical procedure, which can lead to more effective and efficient collaboration and coordination among surgical team members, and which can allow surgical team members to quickly and efficiently resolve issues. Furthermore, the systems and methods can predict events that may occur during a surgical procedure and guide the presentation of contextual visual content associated with such events, thereby allowing surgical team members to prepare for and / or resolve such events before they occur.

[0020] Many technical computing benefits can also be achieved by the systems and methods described herein. For example, the systems and methods described herein can be configured to access, convert, and process data from different computing systems in a manner that allows the systems and methods to provide timely (e.g., real-time) information to various users through various computing platforms. To this end, the systems and methods described herein can be seamlessly integrated with one or more dedicated computing devices to process various types of data (e.g., by applying kinematic data, image data, sensor data, and / or surgical instrument data to one or more machine learning models) in order to identify events that occurred or may occur during a surgical operation and / or identify background information associated with the event. In addition, the systems and methods described herein can utilize historical surgical data generated during a surgical operation prior to the current surgical operation to determine the background of the surgical operation with reference to other previous surgical operations. In this way, the systems and methods described herein can perform operations that would be impossible for a person to perform alone. In addition, the systems and methods described herein can improve the operation of computer-assisted surgical systems by improving their efficiency, accuracy, and effectiveness.

[0021] Various embodiments will now be described in greater detail with reference to the accompanying drawings.The systems and methods described herein can provide one or more of the benefits noted above and / or various additional and / or alternative benefits that will become apparent herein.

[0022] In some embodiments, the camera control system can be operated as part of or in conjunction with a computer-assisted surgery system. Accordingly, an exemplary computer-assisted surgery system will now be described. The exemplary computer-assisted surgery system described is illustrative and not limiting. The camera control system can be operated as part of or in conjunction with the computer-assisted surgery system described herein and / or other suitable computer-assisted surgery systems.

[0023] Figure 1 An exemplary computer-assisted surgery system 100 ("surgical system 100") is shown. As shown, surgical system 100 may include a manipulation system 102, a user control system 104, an auxiliary system 106, and an auxiliary camera system 108 communicatively coupled to one another. In some examples, the camera control systems described herein may be implemented by one or more of these components.

[0024] Surgical system 100 can be utilized by a surgical team to perform a computer-assisted surgical procedure on a patient 110 positioned on an operating table 112. As shown, the surgical team may include a surgeon 114-1, a nurse 114-2, an assistant 114-3, and an anesthesiologist 114-4, all of which may be collectively referred to as "surgical team members 114." Additional or alternative surgical team members may be present during the surgical procedure as may be necessary for a particular embodiment. Surgical system 100 and surgical team members 114 are located at a surgical facility 116. As used herein, a "surgical facility" may include any area in which surgical system 100 and surgical team members 114 are located while a computer-assisted surgical procedure is being performed. For example, a surgical facility may include an operating room, a training facility, a specific area within a room, etc. In some examples, surgeon 114-1 may be located in a room or location separate from the room or location in which surgical system 102, auxiliary system 106, and / or other surgical team members 114 are located during the surgical procedure. In these examples, the surgical facility may include one or both locations.

[0025] Although Figure 1 A minimally invasive surgical procedure is shown being performed at a surgical facility 116, but the surgical system 100 may similarly be used to perform open surgical procedures or other types of surgical procedures that may similarly benefit from the accuracy and convenience of the surgical system 100. Additionally, it will be understood that surgical procedures that may employ the surgical system 100 may include more than just the surgical phase of a surgical procedure, such as Figure 1 , and may include pre-operative, post-operative, and / or other suitable stages of a surgical procedure. A surgical procedure may include any procedure performed on a patient using manual and / or instrumental techniques to investigate, diagnose, or treat a medical condition of the patient. Additionally, a surgical procedure may include any procedure not performed on a live patient, such as a calibration procedure, a training procedure, and an experimental or research procedure.

[0026] like Figure 1 As shown in , the manipulation system 102 may include a plurality of manipulator arms 118 (e.g., manipulator arms 118-1 to 118-4) that may be coupled to a plurality of surgical instruments (not shown). Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool having tissue interaction functionality), medical tool, monitoring instrument (e.g., an endoscope), sensing instrument (e.g., a force sensing surgical instrument), diagnostic instrument, etc. that can be used for a computer-assisted surgical procedure (e.g., by being at least partially inserted into a patient 110 and manipulated to perform a computer-assisted surgical procedure on the patient 110). Although the manipulation system 102 is depicted and described herein as including four manipulator arms 118, it will be appreciated that the manipulation system 102 may include only a single manipulator arm 118 or any other number of manipulator arms as may serve a particular embodiment.

[0027] The surgical instruments coupled to the manipulator arm 118 can each be positioned at a surgical field associated with the patient. In some examples, the "surgical field" can be located entirely within the patient's body and can include an area within the patient's body at or near the location where a surgical procedure is planned, being performed, or has been performed. For example, for a minimally invasive surgical procedure performed on tissue within the patient's body, the surgical field can include the tissue, the anatomical structures beneath the tissue, and the space surrounding the tissue where, for example, surgical instruments used to perform the surgical procedure are located. In other examples, the surgical field can be at least partially located outside the patient's body at or near the location where a surgical procedure is planned, being performed, or has been performed on the patient. For example, the surgical system 100 can be used to perform an open surgical procedure such that a portion of the surgical field (e.g., the tissue being operated on) is within the patient's body, while another portion of the surgical field (e.g., the space surrounding the tissue where one or more surgical instruments can be located) is outside the patient's body. When at least a portion of a surgical instrument (e.g., a distal portion of the surgical instrument) is located within the surgical field, the surgical instrument can be said to be positioned or located at or within the surgical field.

[0028] The user control system 104 can be configured to facilitate control by the surgeon 114-1 of the manipulator arm 118 and the surgical instruments coupled to the manipulator arm 118. For example, the surgeon 114-1 can interact with the user control system 104 to remotely move or manipulate the manipulator arm 118 and the surgical instruments. To this end, the user control system 104 can provide the surgeon 114-1 with an image (e.g., a high-definition stereoscopic image) of the surgical area associated with the patient 110 as captured by an imaging device (e.g., an endoscope). In some examples, the user control system 104 can include a stereoscopic viewer having two displays, wherein the stereoscopic image of the surgical area associated with the patient 110 and generated by the stereoscopic imaging system can be viewed by the surgeon 114-1. The surgeon 114-1 can utilize the image to perform one or more procedures with one or more surgical instruments coupled to the manipulator arm 118.

[0029] To facilitate control of surgical instruments, user control system 104 may include a set of master controls (not shown). These master controls may be manipulated by surgeon 114-1 to control the movement of surgical instruments (e.g., by utilizing robotics and / or teleoperation technology). The master controls may be configured to detect various hand, wrist, and finger movements performed by surgeon 114-1. In this manner, surgeon 114-1 may intuitively perform surgical procedures using one or more surgical instruments.

[0030] The user control system 104 may also be configured to facilitate control by the surgeon 114-1 of other components of the surgical system 100. For example, the surgeon 114-1 may interact with the user control system 104 to change the configuration or operating mode of the surgical system 100, change the display mode of the surgical system 100, generate additional control signals for controlling surgical instruments attached to the manipulator arm 118, facilitate switching control from one surgical instrument to another, or perform any other suitable operation. To this end, the user control system 104 may also include one or more input devices (e.g., foot pedals, buttons, switches, etc.) configured to receive input from the surgeon 114-1.

[0031] The auxiliary system 106 may include one or more computing devices configured to perform the primary processing operations of the surgical system 100. The one or more computing devices included in the auxiliary system 106 may control and / or coordinate operations performed by various other components of the surgical system 100 (e.g., the manipulator system 102, surgical instruments attached to the manipulator arms 118, and / or the user control system 104). For example, a computing device included in the user control system 104 may transmit instructions to the manipulator system 102 via one or more computing devices included in the auxiliary system 106. As another example, the auxiliary system 106 may receive and process image data from the manipulator system 102 representing an image captured by an imaging device attached to one of the manipulator arms 118.

[0032] In some examples, the assistance system 106 can be configured to present visual content to surgical team members 114 who may not have access to the images provided to the surgeon 114-1 at the user control system 104. To this end, the assistance system 106 can include a display monitor 120 that is configured to display one or more user interfaces, such as an image of the surgical area (e.g., a 2D image), information associated with the patient 110 and / or the surgical procedure, and / or any other visual content as may serve a particular embodiment. For example, the display monitor 120 can display an image of the surgical area and additional content (e.g., graphical content, background information, etc.) displayed simultaneously with the image. In some embodiments, the display monitor 120 is implemented by a touch screen display with which the surgical team members 114 can interact (e.g., via touch gestures) to provide user input to the surgical system 100.

[0033] The auxiliary camera system 108 may be configured to capture and provide visual content based on the context of the detected surgical procedure on one or more display devices of the surgical system 100. Figure 1, the auxiliary camera system 108 may include an auxiliary camera 122 ("camera 122") coupled to a camera manipulation system 124. Camera 122 may be implemented by any suitable camera. In some examples, camera 122 may be implemented by multiple imaging devices configured to provide stereoscopic images. Although the auxiliary camera system 108 is depicted and described herein as including one camera 122, it will be appreciated that the auxiliary camera system 108 may include multiple cameras 122 as may serve a particular implementation. For example, if the user control system 104 and surgeon 114-1 are located remotely from the other components of the surgical system 100, additional cameras may be positioned to capture and provide images of the user control system 104 and surgeon 114-1.

[0034] The images captured by the camera 122 (e.g., one or more still images or video images) can be transmitted to and / or displayed by any suitable display device associated with the surgical system 100. For example, the images captured by the camera 122 can be transmitted to and displayed by a display monitor 120, a stereoscopic viewer of the user control system 104, a mobile device communicatively paired with the surgical system 100 during the surgical procedure (e.g., a mobile device used by assistant 114-3), and / or a display device associated with a remote computing device. In some examples where the surgical system 100 includes a dual-user control system 104, such as for training surgeons to use the surgical system 100, the images captured by the camera 122 can be displayed by a display device associated with the second user control system 104 for viewing by another user (e.g., a remote monitor monitoring the surgical procedure and / or training the surgeon). Additionally or alternatively, image data representing the images captured by the camera 122 can be streamed and / or recorded by the remote computing device, enabling playback of the images at a later time (e.g., after the surgical procedure).

[0035] Camera manipulation system 124 may be configured to automatically adjust the position and / or orientation of camera 122 to capture a particular field of view 126 within surgical facility 116. As used herein, the "position" of camera 122 within surgical facility 116 may refer to a particular location of camera 122 within three-dimensional ("3D") space, and the "orientation" of camera 122 may refer to the imaging direction of camera 122 (e.g., field of view 126), which may be any direction within 3D space.

[0036] The camera manipulation system 124 may be implemented by any suitable mechanism(s) configured to adjust the position and / or orientation of the camera 122. For example, Figure 1As shown in FIG, the camera manipulation system 124 may include a camera manipulation arm 128 coupled to the camera 122 via a camera pivot 130. The camera manipulation arm 128 may be configured to adjust the position and / or orientation of the camera 122 through movement of the camera manipulation arm 128. The camera pivot 130 may be configured to adjust the orientation of the camera 122. For example, the camera pivot 130 may rotate the camera 122 within the horizontal plane, such as by panning from left to right (e.g., looking in the imaging direction of the camera 122). Additionally or alternatively, the camera pivot 130 may tilt the camera 122 upward or downward within the vertical plane, such as by adjusting the imaging direction of the camera 122 upward or downward (i.e., at an angle relative to the horizontal plane). The camera manipulation system 124 may include additional or alternative components as may serve a particular embodiment. For example, the camera manipulation system 124 may include various gears, motors, arms, pivots, joints, bearings, and any other electrical and / or mechanical components that may facilitate adjustment of the position and / or orientation of the camera 122.

[0037] The camera manipulation system 124 of the auxiliary camera system 108 can be configured to adjust the position and / or orientation of the camera 122 in any suitable manner. In some examples, the position and / or orientation of the camera manipulation system 124 can be manually adjusted, such as by a user pushing or moving the camera 122 to a desired position or operating one or more electronic controls of the camera manipulation system 124. Additionally or alternatively, the position and / or orientation of the camera 122 can be automatically adjusted (e.g., without user input) based on the context of the surgical procedure (e.g., based on events associated with the surgical procedure), as will be explained in more detail below.

[0038] like Figure 1 As shown in FIG, an auxiliary camera system 108 (e.g., camera manipulation system 124) is coupled to the manipulation system 102. In alternative examples, the auxiliary camera system 108 can be coupled to any other component of the surgical system 100, such as the user control system 104, the auxiliary system 106, or the cart 132 (e.g., instrument cart 132-2), as may be suitable for a particular embodiment. In additional examples, the auxiliary camera system 108 can be separate from the manipulation system 102, the user control system 104, and the auxiliary system 106. For example, the camera manipulation system 124 can be mounted to a wall of an operating room. Alternatively, the auxiliary camera system 108 can be a standalone system within the surgical facility 116, such as Figure 2 As shown in .

[0039] Figure 2 and Figure 1108, except that the auxiliary camera system 108 includes a movable camera cart 202. The camera manipulation system 124 is supported on the camera cart 202, thereby allowing the surgical team member 114 to easily position the camera 122 at a desired location within the surgical facility 116 by moving the camera cart 202. Furthermore, the surgical team member 114 can move the camera cart 202 as needed during the surgical procedure.

[0040] Reference again Figure 1 , the surgical system 100 may also include one or more movable carts 132 for holding certain components of the surgical system 100 and / or supplies to be used during a surgical procedure. For example, one or more computing devices included in the auxiliary system 106 may be housed within the auxiliary cart 132-1. Additionally, surgical instruments that are not coupled to the manipulator arm 118 may be stored on the instrument cart 132-2 for easy access (e.g., by the assistant 114-3) when coupling the surgical instruments to the manipulator arm 118 and / or exchanging surgical instruments during a surgical procedure. For example, the supply cart 132-3 may be used by the anesthesiologist 114-4 to store medications and / or other agents (e.g., fluorescent imaging agents) that may be administered to the patient 110 during a surgical procedure.

[0041] Various components of the surgical system 100 may include one or more displacement sensors, orientation sensors, and / or position sensors (hereinafter referred to as "surgical system sensors") for generating raw (i.e., uncorrected) kinematic information. For example, the user control system 104, the auxiliary system 106, the auxiliary camera system 108, the operating table 112, the manipulator arm 118, the surgical instruments attached to the manipulator arm 118, and the cart 132 may include one or more surgical system sensors. The surgical system 100 (e.g., the auxiliary system 106) may be configured to use the kinematic information to track the various components of the surgical system 100 (e.g., determine the positioning of the various components of the surgical system 100) and / or control the various components of the surgical system 100.

[0042] The components of surgical system 100 may be communicatively coupled to one another in any suitable manner. Figure 1 As shown in FIG, the steering system 102, the user control system 104, the auxiliary system 106, the auxiliary camera system 108, and the cart 118 can be communicatively coupled via a control line 134, which can represent any wired or wireless communication link that can serve a particular embodiment. To this end, the steering system 102, the user control system 104, the auxiliary system 106, the auxiliary camera system 108, and the cart 132 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0043] As mentioned, the position and / or orientation of camera 122 may be automatically adjusted based on the context of the surgical procedure to capture a particular field of view (eg, field of view 126 ) within surgical facility 116 . Figure 3 An exemplary camera control system 300 is shown, which is configured to automatically control certain operations of a camera system associated with a computer-assisted surgery system (e.g., auxiliary camera system 108, an endoscope coupled to manipulator arm 118, multiple cameras associated with the computer-assisted surgery system, etc.) to provide contextual visual content associated with events associated with the computer-assisted surgery. As shown, camera control system 300 may include, but is not limited to, a storage facility 302 and a processing facility 304 selectively and communicatively coupled to each other. Facilities 302 and 304 may each include or be implemented by hardware and / or software components (e.g., a processor, memory, a communication interface, instructions stored in memory for execution by the processor, etc.). In some examples, facilities 302 and 304 may be distributed across multiple devices and / or multiple locations, as may be necessary to facilitate a particular embodiment.

[0044] The storage facility 302 may maintain (e.g., store) executable data used by the processing facility 304 to perform any of the operations described herein. For example, the storage facility 302 may store instructions 306 that may be executed by the processing facility 304 to perform any of the operations described herein. The instructions 306 may be implemented by any suitable application, software, code, and / or other executable data instance.

[0045] The storage facility 302 may also maintain any data received, generated, managed, used, and / or transmitted by the processing facility 304. For example, as will be described in greater detail below, the storage facility 302 may maintain surgical data, user profile data, user input data, and the like.

[0046] The processing facility 304 can be configured to perform (e.g., execute instructions 306 stored in the storage facility 302 to perform) various processing operations associated with automatically adjusting the field of view of a camera to capture a specific field of view at a location associated with an event associated with a surgical procedure. For example, the processing facility 304 can access surgical data for a surgical procedure. The surgical procedure can include the execution of one or more operations performed by a computer-assisted surgical system (e.g., surgical system 100). The processing facility 304 can identify an event associated with the surgical procedure, such as an event that has occurred or an event that is likely to occur, based on the surgical data. Based on the surgical data, the processing facility 304 can determine a location associated with the event and, in response to the determination of the location of the event, can direct automatic adjustment of the field of view of a camera (e.g., camera 122) to capture a specific field of view at the location associated with the event. These and other operations that can be performed by the processing facility 304 are described in more detail herein.

[0047] In some examples, camera control system 300 is implemented entirely by the computer-assisted surgery system itself. For example, camera control system 300 can be implemented by one or more computing devices included in surgical system 100 (e.g., one or more computing devices included in manipulation system 102, user control system 104, auxiliary system 106, and / or auxiliary camera system 108).

[0048] Figure 4 Another exemplary embodiment 400 of the camera control system 300 is shown. In embodiment 400, a remote computing system 402 can be communicatively coupled to the surgical system 100 via a network 404. The remote computing system 402 can include one or more computing devices (e.g., servers) configured to perform any of the operations described herein. In some examples, the camera control system 300 can be implemented entirely by the remote computing system 402. Alternatively, the camera control system 300 can be implemented by both the remote computing system 402 and the surgical system 100.

[0049] The network 404 can be a local area network, a wireless network (e.g., Wi-Fi), a wide area network, the Internet, a cellular data network, and / or any other suitable network. Data can flow between components connected to the network 404 using any communication technology, device, medium, and protocol as may serve a particular embodiment.

[0050] Figure 5 Another exemplary embodiment 500 of the camera control system 300 is shown. Figure 5 Similar to Figure 4, except that the user device 502 can be communicatively paired with the surgical system 100 and connected to the network 404. As shown, the user device 502 can communicate with the remote computing system 402 and / or the surgical system 100 via the network 404. Additionally or alternatively, the user device 502 can communicate with the surgical system 100 via a direct connection 504 (e.g., a direct wired connection and / or a direct wireless connection, such as a Bluetooth connection, a near field communication connection, etc.). In some examples, the camera control system 300 can be implemented by the remote computing system 402, the surgical system 100, and / or the user device 502.

[0051] User device 502 may be any suitable computing device configured to receive and transmit user input and / or present visual content. For example, user device 502 may be, but is not limited to, a mobile device (e.g., a mobile phone, a handheld device, a tablet computing device, a laptop computer, a personal computer, etc.), a wearable device (e.g., a smartwatch device, an activity tracker, a head-mounted display device, a virtual or augmented reality device, etc.), and / or a display device (e.g., a television, a projector, a monitor, a touch screen display device, etc.).

[0052] As shown, a user 506 (e.g., surgical team member 114) can use or otherwise access a user device 502. In some examples, the user 506 may have to log into the user device 502 or an application executed by the user device 502 in order to use and / or interact with the user device 502. In some examples, the application executed by the user device 502 can be used by the user 506 to provide user input regarding the workflow segmentation of the surgical procedure and to view images captured by a camera (e.g., camera 122). For example, the user 506 can provide input regarding tasks completed by the user 506 or by another surgical team member (e.g., one or more operations performed by a surgical team member or a computer-assisted surgical system), the stage of the surgical procedure, the patient's condition, malfunctions of components of the surgical system, changes in surgical team personnel, and any other information as may be appropriate for a particular embodiment. As will be explained in more detail below, such user input can be utilized by the camera control system 300 to more accurately identify surgical events that have occurred or may occur during the surgical procedure.

[0053] Although Figure 5 Only one user device 502 is shown communicatively paired with the surgical system 100 and connected to the network 404, but any number of user devices 502 may be paired with the surgical system 100 and / or connected to the network 404. For example, each surgical team member 114 may utilize a user device 502 during a surgical procedure to provide input regarding the workflow of the surgical procedure and to view images captured by a camera (e.g., camera 122).

[0054] Various operations and examples of these operations that may be performed by the camera control system 300 (eg, by the processing facility 304 of the camera control system 300) will now be described. It will be appreciated that the operations and examples described herein are merely illustrative of the many different types of operations that may be performed by the camera control system 300.

[0055] The camera control system 300 can direct automatic adjustment of a camera's field of view (e.g., field of view 126 of camera 122) to capture a specific field of view of a location associated with a surgical event. To this end, the camera control system 300 can access surgical data for the surgical procedure and, based on the surgical data, identify an event associated with the surgical procedure. Various examples of these operations will now be provided.

[0056] In some examples, the surgical data accessed by the camera control system 300 can be generated during the surgical procedure and can be based on one or more operations performed by the computer-assisted surgery system (e.g., surgical system 100) during the surgical procedure. The operations performed by the computer-assisted surgery system can include any mechanical, electrical, hardware, and / or software-based operations as may be suitable for a particular embodiment. The surgical data can be generated by the computer-assisted surgery system (e.g., by one or more components within the surgical system 100), by one or more components coupled to the computer-assisted surgery system during the surgical procedure (e.g., one or more surgical instruments coupled to the manipulator arm 118), by a user device (e.g., user device 502) that is communicatively paired with the computer-assisted surgery system during the surgical procedure, and / or by any other device associated with the computer-assisted surgery system as may be suitable for a particular embodiment. In scenarios where the camera control system 300 is implemented entirely by the remote computing system 402, the surgical data can additionally or alternatively be generated by the remote computing system 402 while, for example, the remote computing system 402 tracks the operations performed by the surgical system 100.

[0057] The surgical data generated during a surgical procedure may include various types of data. For example, the surgical data generated during a surgical procedure may include kinematic data, image data, sensor data, surgical instrument data, and / or any other type of data as may serve a particular embodiment.

[0058] The kinematic data may represent the position, pose, and / or orientation of components within the computer-assisted surgery system and / or components coupled to the computer-assisted surgery system. For example, the kinematic data may represent the position, pose, and / or orientation of the manipulator arm 118 and / or a surgical instrument coupled to the manipulator arm 118. As another example, the kinematic data may represent the position of the manipulation system 102, the user control system 104, the auxiliary camera system 108, the operating table 112, and / or the cart 132.

[0059] The image data may represent one or more images captured by an imaging device coupled to the computer-assisted surgery system. For example, the image data may represent one or more images captured by an imaging device (e.g., a stereo endoscope) coupled to the manipulator arm 118. The one or more images may constitute one or more still images and / or videos captured by the imaging device. In some examples, the camera control system 300 may access the image data by receiving (e.g., over a network) images output by the imaging device. In additional or alternative examples, the image data may include image data generated by an imaging device (e.g., camera 122) outside the patient's body.

[0060] Sensor data may include any data generated by surgical system sensors included in or associated with the computer-assisted surgery system. Sensor data may represent any sensed parameter that may serve a particular embodiment. For example, sensor data may indicate whether the operating table 112 is moving or whether the surgeon 114-1 is actively interacting with the user control system 104.

[0061] Surgical instrument data may include any data generated by a surgical instrument and may represent an identification ("ID") of the surgical instrument, an operating state of the surgical instrument (e.g., open, closed, powered, idle, etc.), fault codes of the surgical instrument, etc.

[0062] In some examples, the camera control system 300 may additionally or alternatively access surgical data generated by the computer-assisted surgery system during one or more other surgical procedures prior to, for example, the current surgical procedure. For example, the camera control system 300 may generate surgical data during a first surgical procedure in which the computer-assisted surgery system is used to perform a first surgical procedure with respect to a first patient. The camera control system 300 may also generate additional surgical data during a second surgical procedure in which the computer-assisted surgery system is used to perform a second surgical procedure with respect to the patient or another patient. During the second surgical procedure, the camera control system 300 may access both the surgical data and the additional surgical data. The surgical data generated prior to the current surgical procedure may be referred to herein as "historical surgical data." As will be described below, the historical surgical data may allow the camera control system 300 to more effectively identify and / or predict events that may occur during the second surgical procedure.

[0063] The camera control system 300 may additionally or alternatively access surgical data based on operations performed by one or more computer-assisted surgery systems other than the computer-assisted surgery system being used during a particular surgical procedure. For example, the camera control system 300 may access surgical data generated during multiple different computer-assisted surgery procedures located within a particular medical center, hospital network, and / or any other grouping. This type of surgical data may be referred to herein as "global surgical data" and, as will be described below, may allow the camera control system 300 to more effectively identify and / or predict events that may occur during a particular surgical procedure in which a particular computer-assisted surgery system included in the grouping is being used to perform the surgical procedure.

[0064] In some examples, the camera control system 300 may provide a user interface configured to allow a user to define specific groupings of computer-assisted surgeries and / or computer-assisted surgery systems, wherein surgical data may be accessed by the camera control system 300 .

[0065] The camera control system 300 can identify events associated with computer-assisted surgery ("surgical events") based on surgical data, historical surgical data, and / or global surgical data for a surgical procedure. A surgical event can include any distinct operation or action that occurs or may occur with respect to the computer-assisted surgery system during a surgical procedure. A surgical event can occur during the preoperative phase, the surgical phase, and / or the postoperative phase of a surgical procedure.

[0066] For example, a surgical event may include any operations or actions associated with various preoperative phase operations. Exemplary preoperative phase operations may include, but are not limited to, patient admission (e.g., admitting the patient to the medical facility, receiving patient documentation, etc.), preparing the operating room, sterilizing surgical instruments, testing the computer-assisted surgery system and equipment, blanketing the computer-assisted surgery system (i.e., covering one or more components of the computer-assisted surgery system (e.g., manipulator arms 118) with a sterile or protective covering), preparing the patient for the surgical procedure (e.g., checking patient vital signs, providing intravenous fluids, administering anesthesia to the patient, bringing the patient to the operating room), and targeting the computer-assisted surgery system relative to the patient (e.g., positioning the manipulator system 102 at the patient's bedside and positioning or configuring one or more manipulator arms 118).

[0067] A surgical event may additionally or alternatively include any operations or actions associated with various surgical phase operations. Exemplary surgical phase operations may include, but are not limited to, opening a surgical area associated with a patient (e.g., by making an incision in tissue external to the patient), inserting surgical instruments into the patient, performing a surgical operation on the patient's tissue (e.g., by cutting the tissue, repairing the tissue, suturing the tissue, cauterizing the tissue, etc.), and closing the surgical area associated with the patient (e.g., removing the surgical instruments from the patient, suturing the incision site, dressing any wounds, etc.).

[0068] A surgical event may additionally or alternatively include any operations or actions associated with various postoperative phase operations. Exemplary postoperative phase operations may include, but are not limited to, removing the computer-assisted surgery system from the patient (e.g., removing the manipulation system 102 from the patient's bedside), patient care and recovery operations (e.g., removing the patient from the operating room, monitoring the patient as they recover from the surgical procedure, etc.), cleaning the operating room, cleaning the computer-assisted surgery system and / or surgical instruments, receiving report documentation by surgical team members, and patient discharge operations.

[0069] Camera control system 300 may identify surgical events based on the surgical data in any suitable manner. Figure 6 6 shows an exemplary manner in which the camera control system 300 can identify surgical events based on surgical data. As shown, the camera control system 300 can apply surgical data 602 as input to an event detection heuristic 604. The event detection heuristic 604 can analyze the surgical data 602 and output various instances of event data 606 (e.g., event data 606-1 through event data 606-N). Each instance of event data 606 can represent a specific surgical event identified by the event detection heuristic 604.

[0070] Event detection heuristics 604 may include any suitable heuristics, procedures, and / or operations that may be performed or executed by camera control system 300 and that may be configured to identify events based on surgical data 602. To illustrate, event detection heuristics 604 may detect indicators and / or patterns in the surgical data that indicate the occurrence of a particular surgical event.

[0071] For example, kinematic data generated during a particular portion of a surgical procedure may indicate movement of a surgical instrument in a suturing mode. Additionally, surgical instrument data may indicate that the surgical instrument used during the same portion of the surgical procedure was a needle driver. Based on this kinematic data and the surgical instrument data, the camera control system 300 may determine that a suturing event is occurring or has occurred.

[0072] As another example, image data representing an image generated by an endoscope may indicate that a particular surgical instrument has remained outside the field of view of the endoscope for a predetermined period of time. Such image data may indicate an idle state event (e.g., the surgical instrument is in an idle state).

[0073] In some examples, surgical data 602 may include historical surgical data, as described above. In these examples, one of the event data instances 606 output by event detection heuristic 604 may represent a surgical event that camera control system 300 predicts will occur based on the historical surgical data. For example, the historical surgical data may include surgical data generated during multiple surgical procedures in which the same type of surgical procedure was performed using a computer-assisted surgery system. Based on this historical surgical data, event detection heuristic 604 may predict that a second event will occur after a first event.

[0074] In some examples, surgical data 602 may include global surgical data, as described above. In these examples, one of surgical event data instances 606 output by event detection heuristic 604 may represent a surgical event determined to occur based on the global surgical data. For example, the global surgical data may indicate that a particular kinematic data value for a particular surgical tool indicates that the surgical tool is within a predetermined distance from patient tissue. When the actual kinematic data for the surgical tool being used during the surgical procedure is equal to or less than that value, event detection heuristic 604 may detect a surgical event indicating that the surgical tool is, in fact, within the predetermined distance from patient tissue.

[0075] Event detection heuristics 604 may receive additional or alternative types of input as may serve a particular implementation. For example, Figure 7 Similar to Figure 6, but illustrates that event detection heuristic 604 can accept user profile data 702 (e.g., data representing a user profile of one or more surgical team members involved in a surgical procedure) as additional input. In this configuration, event detection heuristic 604 can detect surgical events based on both surgical data 602 and user profile data 702.

[0076] For illustration, user profile data 702 may include data representing a user profile of a surgeon involved in a surgical procedure (e.g., surgeon 114-1). The user profile for the surgeon, combined with the surgical procedure data, may instruct the surgeon to perform various operations in a specific order unique to the surgeon. Thus, event detection heuristics 604 may detect that a specific surgical procedure event is about to occur according to a specific order.

[0077] Figure 8 Another example of receiving additional or alternative types of input is shown. Figure 8 Similar to Figure 6 , but illustrates that the event detection heuristic 604 can accept user input data 802 as additional input. The user input data 802 can represent information entered by a user through a computing device included in the computer-assisted surgery system (e.g., through the user control system 104 or the auxiliary system 106) or through a computing device (e.g., the user device 502) that is communicatively paired with the computer-assisted surgery system. In this configuration, the event detection heuristic 604 can detect surgical events based on both the surgical data 602 and the user input data 802. The user input data 802 can include, for example, information entered through an application executed by a user device associated with a surgical team member.

[0078] To illustrate, user 506 (e.g., anesthesiologist 114-4) may input information indicating that patient 110 is fully sedated via an application currently executed by user device 502. This information, combined with surgical data, may indicate that the preoperative phase of the surgical procedure has concluded. Thus, event detection heuristic 604 may detect that a particular surgical event may have occurred, such as the opening of an incision site on patient 110. Consequently, camera control system 300 may direct automatic adjustment of camera 122 to capture a particular field of view of a location associated with the next event, such as instrument cart 132-2.

[0079] In some examples, event detection heuristics 604 can implement a machine learning model. The machine learning model can use historical surgical data, global surgical data, user profile data, user input data, or any combination or subcombination thereof to identify one or more unique patterns of surgical system operation and associate surgical events with the detected patterns of surgical system operation. As camera control system 300 collects more data, event data 606 output by event detection heuristics 604 can be updated or corrected as needed.

[0080] When the camera control system 300 identifies a surgical event, the camera control system 300 may determine a location within the surgical facility associated with the identified surgical event (the "event location"). The camera control system 300 may then direct automatic adjustments made by a camera system associated with the computer-assisted surgery system (e.g., the auxiliary camera system 108) to automatically adjust the camera's field of view (e.g., the field of view 126 of the camera 122) to capture a specific field of view of the event location.

[0081] The camera control system 300 may determine the event location in any suitable manner. In some examples, the camera control system 300 may determine the event location by determining the location of a component associated with the identified surgical event ("event component"). To this end, in some embodiments, the camera control system 300 may access an event location table to identify the component associated with the identified surgical event. For illustration, Figure 9 An exemplary event location table 900 that may be maintained or otherwise accessed by the camera control system 300 is shown. As shown in column 902, the table 900 may include multiple entries representing various surgical events that may occur during a surgical procedure (e.g., manipulator arm collision events, instrument change events, bed movement events, manipulator arm error events, master controller collision events, and manipulator arm no signal events). As shown in column 904, the table 900 may also list the components associated with each surgical event (e.g., a specific manipulator arm, operating table, user control system, and auxiliary cart).

[0082] Once the event component has been identified, the camera control system 300 may determine the location of the event component ("component location") in any suitable manner. In some examples, the camera control system 300 may determine the component location based on surgical data generated during the surgical procedure. For example, the camera control system 300 may be configured to determine the component location for the event component based on kinematic data generated during the surgical procedure. Additionally or alternatively, the event position table may list component location data representing the current position of each component listed in the event position table. The component location data may be populated at the start of the surgical procedure and updated throughout the surgical procedure based on surgical data (e.g., kinematic data) generated during the surgical procedure.

[0083] In addition to or in lieu of accessing kinematic data, the camera control system 300 can determine component positions based on component tracking. In some examples, the camera control system 300 can utilize marker-based computer vision tracking. To this end, unique markers (e.g., bar codes, colors, patterns, shapes, etc.) can be integrated into or otherwise attached to various components of the computer-assisted surgical system (e.g., manipulator arm 118, surgical instruments, cart 132, etc.), and an event position table can list marker data representing the unique marker associated with each component. The camera control system 300 can access the event position table to identify a specific marker data instance associated with the event component. Based on the identified marker data instance, the camera control system 300 can then determine the component position by detecting the marker associated with the event component in the image captured by the camera system.

[0084] In an additional or alternative example, the camera control system 300 can utilize signal-based tracking to determine component locations. To this end, emitters can be integrated into or otherwise attached to various components of the computer-assisted surgery system (e.g., manipulator arm 118, surgical instruments, cart 132, etc.), and the camera system can include sensors configured to detect signals emitted from the components. The emitters can be configured to emit any suitable signal, such as an infrared ("IR") signal. Each emitter can be configured to emit a unique signal based on, for example, a unique wavelength, a unique flashing pattern, a unique frequency, etc. An event location table can list signal data representing the unique signal associated with each component. The camera control system 300 can access the event location table to identify a specific signal data instance associated with an event component. The camera control system 300 can then direct the event component to emit its specific signal. Alternatively, the component can periodically emit its associated signal. The camera control system 300 can then identify the component location as the source of the emitted signal that matches the identified signal data instance associated with the event component.

[0085] Once the camera control system 300 has determined the event location, the camera control system 300 can direct automatic adjustment of the camera's field of view to capture a specific field of view at the event location. In some examples, the camera control system 300 can cause a camera manipulation system to which the camera is coupled to adjust the camera's position and / or orientation to capture a specific field of view at the event location. In alternative examples where the camera system is implemented by a separate computing device, the camera control system 300 can transmit data representing the event location and commands to adjust the camera's position and / or orientation to capture a specific field of view at the event location to the camera system.

[0086] In some examples where the camera control system determines the event location based on marker-based tracking or signal-based tracking, the camera control system 300 can direct the camera toward the marker or emitted signal associated with the event location.

[0087] In additional or alternative examples, the amount and direction of adjustment of the position and / or orientation of the camera can be determined based on the spatial relationship between the camera and the event location. In some examples where the camera system is physically coupled (e.g., mounted) directly to a component of the surgical system, the spatial relationship between the camera and the event location can be determined based on surgical data (e.g., kinematic data, sensor data, etc.). Thus, adjustment of the field of view of the camera can be based on the surgical data. For example, the camera control system 300 can determine the position and orientation of the camera relative to the event location based on kinematic data and sensor data associated with the camera system. Based on the determined position and orientation of the camera, the camera control system 300 can direct automatic adjustment of the camera to capture a particular field of view of the event location.

[0088] In other examples where the camera system is not physically coupled to components of the surgical system, the spatial relationship between the camera and the event location may not be readily available or determined solely from surgical data. In such examples, to facilitate automatic adjustment of the camera's field of view to capture a specific field of view of the event location, images captured by the camera system's cameras and 3D position data tracked by the computer-assisted surgery system can be registered to a common 3D space. Additionally or alternatively, the camera system (e.g., camera 122 and / or camera manipulation system 124) can be registered to the common 3D space. As mentioned above, the computer-assisted surgery system can use kinematic information generated from the surgical system's sensors to track various components of the surgical system within the 3D space (e.g., determine their positioning). Registration of the images captured by the camera to the same 3D space used by the computer-assisted surgery system can allow the camera control system 300 to use kinematic information associated with the computer-assisted surgery system including the camera system to determine the spatial relationship between the camera and the event location. Such spatial relationship information can be utilized by the camera control system 300 to determine the direction and amount of adjustment of the camera's positioning and / or orientation to capture the field of view of the event location. Even where tracking-based techniques are used to determine component position (and therefore event position), registration may enable estimation of camera location and / or orientation based on surgical data and thereby accelerate tracker-based determination of component position.

[0089] The camera control system 300 can be configured to perform a registration process once after initial registration (e.g., as part of a calibration, setup process, or other initial registration) and / or periodically or continuously to refine the initial registration (e.g., to account for changes in physical positioning, which may have various kinematic errors) and / or to account for positional adjustments of surgical system components. Registration can be performed using any suitable registration technique, including but not limited to vision-based registration, model-based registration, and marker-based registration.

[0090] To illustrate, images captured by a camera can be used in the registration process. For example, the camera control system 300 can determine the position of components included in the computer-assisted surgical system based on the images captured by the camera and using a trained neural network. In some examples, this can include the camera control system 300 using the trained neural network to identify an object of interest shown in the image; associating the object of interest with a component of the surgical system in any suitable manner (e.g., object recognition); and determining the position of the object (e.g., based on the position of the image).

[0091] The camera control system 300 can then perform an optimization to fit the model of the surgical system component to the determined position of the object (e.g., the position of the object in the image). For components positioned on the floor, such as the cart 132, the camera control system 300 can be configured to constrain the optimization to search only for solutions that are rotational and translational about the plane of the floor on which the component is placed. This constrained optimization can provide faster and / or more accurate results than traditional optimization performed in all six degrees of freedom.

[0092] The camera control system 300 may use a registration process to determine missing links in a kinematic chain connecting a camera that is not physically coupled to a component of the computer-assisted surgery system. The camera control system 300 may represent the missing links as transformations that define an estimated spatial relationship between the camera and the component of the surgical system. The transformations may define rotations and translations that may be applied to transform data points from the camera's reference frame to the computer-assisted surgery system's reference frame (e.g., transforming coordinate points from the camera's coordinate system to the computer-assisted surgery system's coordinate system), and vice versa.

[0093] The camera control system 300 can use the missing links to complete the kinematic chain connecting the camera and the surgical system, so that the complete kinematic chain is known and accessible to the camera control system 300. The camera control system 300 can then use the kinematics of the camera system to track the positioning of the camera and determine the appropriate position and / or orientation adjustment of the camera to capture the field of view of the event location.

[0094] As mentioned, the images captured by the camera can be transmitted to and displayed by a display device associated with the computer-assisted surgery system and / or a display device associated with the remote computing device. In this way, surgical team members, remote monitors, remote technicians, etc. can view background visual content associated with the surgical event and use the background visual content, for example, to perform one or more tasks during the surgical procedure (e.g., to perform the surgical procedure, to monitor or train surgical team members, to resolve technical issues, etc.). An example of this process will now be described.

[0095] For example, if the camera control system 300 detects that the manipulator arm 118-1 is colliding with the manipulator arm 118-2 (labeled as "MA1_Collide_MA2" in table 900), the camera control system 300 can determine that the location of the detected event is the location of the manipulator arm 118-1 (labeled as "MA1" in table 900). Based on this determination, the camera control system 300 can utilize the surgical data (e.g., kinematic data) to determine the location of the manipulator arm 118-1 within the surgical facility. The camera control system 300 can then direct the camera manipulation system 124 to adjust the positioning and / or orientation of the camera 122 to capture the field of view of the manipulator arm 118-1. The image captured by the camera 122 can then be displayed on a display device associated with the surgical team member 114 (e.g., surgeon 114-1), thereby allowing the surgical team member 114 to quickly identify why the manipulator arm 118-1 is not moving.

[0096] As another example, if the camera control system 300 determines that no signal was detected from a surgical instrument coupled to the fourth manipulator arm 118-4 (labeled "MA4_NoSignal" in table 900), the camera control system 300 may determine that the location of the detected event is a location on the fourth manipulator arm 118-4 where the surgical instrument should be connected (labeled "MA4" in table 900). Based on this determination, the camera control system 300 may utilize surgical data (e.g., kinematic data) to determine the location of the fourth manipulator arm 118-4 within the surgical facility. The camera control system 300 may then direct the camera manipulation system 124 to adjust the positioning and / or orientation of the camera 122 to capture the field of view of the fourth manipulator arm 118-4. The image captured by the camera 122 may then be displayed on a display device associated with the surgical team member 114 (e.g., a user device used by the assistant 114-3), thereby allowing the surgical team member 114 to quickly determine whether the surgical instrument is properly connected to the fourth manipulator arm 118-4.

[0097] In some examples, multiple surgical system components can be associated with a particular event. For example, Figure 10An exemplary event location table 1000 that may be maintained or otherwise accessed by the camera control system 300 is shown. As shown in column 1002, the table 1000 may include multiple entries representing various events that may occur during a surgical procedure. As shown in column 1004, the table 1000 may list the primary component associated with each event. As shown in column 1006, the table 1000 may list the secondary component associated with each event. The camera control system 300 may determine the event location as the location of the primary component associated with the detected event. The camera control system 300 may then switch the event location to the location of the secondary component in response to a location switch event.

[0098] In some examples, the position switching event may include a predetermined time period (e.g., 20 seconds) having passed since the particular view of the primary component was first displayed. In some examples, the user may specify the duration of the predetermined time period.

[0099] Alternatively, the position switching event may be the reception by the camera control system 300 of user input directing the camera control system 300 to switch to the secondary position. The user input may be received by any suitable computing device associated with the computer-assisted surgery system (e.g., the user device 502, the remote computing system 402, or a computing device included in the user control system 104, the auxiliary system 106, or the camera system 108).

[0100] To illustrate, if the camera control system 300 detects that the surgical instrument coupled to the manipulator arm 118-3 is not moving when being operated by the surgeon 114-1 (labeled "MA3_NoMovement" in table 1000), the camera control system 300 may determine that the primary location of the detected event is the position of the manipulator arm 118-3 (labeled "MA3" in table 1000). Based on this determination, the camera control system 300 may utilize surgical data (e.g., kinematic data) to determine the position of the manipulator arm 118-3 within the surgical facility. The camera control system 300 may then direct the auxiliary camera system 108 to adjust the field of view of the camera 122 to capture a specific field of view of the manipulator arm 118-3. When the image captured by the camera 122 is displayed on a display associated with the user control system (e.g., on a touch screen or on the user device 502), the surgeon may quickly determine whether the surgical instrument is not properly coupled to the manipulator arm 118-3 or whether the manipulator arm 118-3 is colliding with another manipulator arm. The surgeon may then provide input via the user control system (e.g., via a touchpad or foot pedal on the user control system 104) or the user device 502 to view the secondary location. In response to this user input, the camera control system 300 may direct the auxiliary camera system 108 to adjust the field of view of the camera 122 to capture a particular field of view of the auxiliary cart 132-1. This may allow the surgeon to determine whether the cables of the surgical instruments are properly connected to the auxiliary cart 132-1.

[0101] In some examples, the user may be able to manually adjust the camera's field of view after the camera has been adjusted to capture the field of view of the event location. Manual adjustments may include positioning and / or orientation adjustments, as well as optical adjustments (e.g., zoom and focus adjustments).

[0102] In some examples, the camera control system 300 can be configured to provide contextual information in addition to the images captured by the camera for display by a display device associated with computer-assisted surgery. For example, the camera control system 300 can identify contextual content associated with an identified surgical event and / or an identified event location. The contextual content can be stored, for example, in table 900 or table 1000. The contextual content can include, for example, a message to be displayed with the images captured by the camera.

[0103] For illustration, the camera control system 300 may determine, based on surgical data, user profile data, and / or user input data, that the surgical procedure has progressed to the stapling stage. Accordingly, the camera control system 300 may direct the automatic adjustment of the camera to capture a particular field of view of the manipulator arm and may direct the display device to display a message along with the image of the manipulator arm, such as: "Replace this surgical instrument with a stapler instrument."

[0104] Additionally or alternatively, the camera control system 300 can be configured to provide contextual information, such as by directing lighting (e.g., task lights, laser lights, etc.) to illuminate the event location, rather than through the display device. For example, the camera control system 300 can direct the adjustment of a laser device in computer-assisted surgery to illuminate a manipulator arm to which a surgical instrument should be coupled. Adjustment of the lighting device can be accomplished in a manner similar to the adjustment of the camera described herein.

[0105] Figure 11 An exemplary camera control method 1100 is shown. Figure 11 Exemplary operations according to one embodiment are shown, but other embodiments may omit, add to, reorder, combine, and / or modify Figure 11 Follow any of the steps shown in . Figure 11 One or more of the operations shown in may be performed by camera control system 300, any components included therein, and / or any implementation thereof.

[0106] In step 1102, the camera control system accesses surgical data for a surgical procedure. The surgical procedure includes the performance of one or more operations performed by a computer-assisted surgery system. Step 1102 can be performed in any manner described herein.

[0107] In step 1104, the camera control system identifies events associated with the surgical procedure based on the surgical procedure data. Step 1104 can be performed in any manner described herein.

[0108] In step 1106, the camera control system determines a location associated with the event based on the surgical data. Step 1106 can be performed in any manner described herein.

[0109] In step 1108, the camera control system directs automatic adjustment of the camera's field of view to capture a particular field of view of the location associated with the event in response to determining the location of the event. Step 1108 may be performed in any manner described herein.

[0110] In some examples, a non-transitory computer-readable medium storing computer-readable instructions can be provided according to the principles described herein. When executed by a processor of a computing device, the instructions can direct the processor and / or computing device to perform one or more operations, including one or more operations described herein. Any of various known computer-readable media can be used to store and / or transmit such instructions.

[0111] As referred to herein, non-transitory computer-readable media may include any non-transitory storage medium that participates in providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by a processor of a computing device). For example, non-transitory computer-readable media may include, but are not limited to, any combination of non-volatile storage media and / or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tape, etc.), ferroelectric random access memory ("RAM"), and optical disks (e.g., optical discs, digital video discs, Blu-ray discs, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0112] Figure 12 An exemplary computing device 1200 is shown that may be specifically configured to perform one or more of the processes described herein. Figure 12 As shown in FIG, computing device 1200 may include a communication interface 1202, a processor 1204, a storage device 1206, and an input / output ("I / O") module 1208 communicatively coupled to one another via a communication infrastructure 1210. Figure 12 An exemplary computing device 1200 is shown in FIG. Figure 12 The components shown in the drawings are not intended to be limiting. Additional or alternative components may be used in other embodiments. Figure 12 Components of computing device 1200 shown in .

[0113] The communication interface 1202 can be configured to communicate with one or more computing devices. Examples of the communication interface 1202 include, but are not limited to, a wired network interface (e.g., a network interface card), a wireless network interface (e.g., a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0114] The processor 1204 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the performance of one or more of the instructions, processes, and / or operations described herein. The processor 1204 may perform operations by executing computer-executable instructions 1212 (e.g., applications, software, code, and / or other executable data instances) stored in the storage device 1206.

[0115] Storage 1206 may include one or more data storage media, devices, or configurations, and may employ any type, form, and combination of data storage media and / or devices. For example, storage 1206 may include, but is not limited to, any combination of non-volatile media and / or volatile media as described herein. Electronic data, including data described herein, may be temporarily and / or permanently stored in storage 1206. For example, data representing computer-executable instructions 1212 configured to direct processor 1204 to perform any of the operations described herein may be stored within storage 1206. In some examples, the data may be arranged in one or more databases residing within storage 1206.

[0116] The I / O modules 1208 may include one or more I / O modules configured to receive user input and provide user output. One or more I / O modules may be used to receive input for a single virtual experience. The I / O modules 1208 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I / O modules 1208 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0117] The I / O module 1208 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In certain embodiments, the I / O module 1208 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content as may serve a particular embodiment.

[0118] In some examples, any of the systems, computing devices, and / or other components described herein may be implemented by computing device 1200. For example, processing facility 304 may be implemented by processor 1204, and storage facility 302 may be implemented by storage device 1206.

[0119] In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and variations may be made thereto, and that additional embodiments may be implemented, without departing from the scope of the present invention as set forth in the appended claims. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. Accordingly, the description and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A system, comprising: a memory for storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to: accessing surgical session data for a surgical session including performance of one or more operations performed by a computer-assisted surgery system; identifying, based on the surgical session data, an event associated with the surgical session; determining a location associated with the event; as well as In response to the determination of the location of the event, automatic adjustment of a field of view of a camera is directed to capture a particular field of view of the location associated with the event.

2. The system of claim 1 , wherein determining the location associated with the event comprises: responsive to identifying the event, identifying a component of the computer-assisted surgery system associated with the event; as well as A location of the component associated with the event is determined. 3 . The system of claim 2 , wherein the determination of the position of the component associated with the event is based on kinematic data representing the positioning of the component associated with the event.

4. The system of claim 2, wherein the determination of the position of the component associated with the event is based on at least one of a marker provided on the component associated with the event and a signal emitted from the component associated with the event.

5. The system of claim 1 , wherein directing the automatic adjustment of the field of view of the camera comprises directing adjustment of at least one of an orientation of the camera and a positioning of the camera to capture a particular field of view of the location associated with the event.

6. The system of any one of claims 1-5, wherein the processor is further configured to execute the instructions to register the camera's image and three-dimensional ("3D") positioning data tracked by the computer-assisted surgery system into a common 3D space.

7. The system of any one of claims 1-5, wherein the camera is coupled to a component of the computer-assisted surgery system.

8. The system of any one of claims 1-5, wherein the surgical session data comprises data representing one or more operations performed by the computer-assisted surgery system during the surgical session.

9. The system of any one of claims 1-5, wherein the surgical session data comprises kinematic data representing at least one of a position, a pose, and an orientation of a component of the computer-assisted surgery system.

10. The system of any one of claims 1-5, wherein the surgical session data comprises image data representing one or more images captured by at least one of the camera and an imaging device coupled to a manipulator arm of the computer-assisted surgery system.

11. The system of any one of claims 1-5, wherein the surgical session data comprises data generated based on user input received during the surgical session via a user device communicatively paired with the computer-assisted surgery system during the surgical session.

12. The system of claim 11, wherein the user input specifies tasks to be performed by the computer-assisted surgery system or a surgical team member during the surgical session.

13. The system of any one of claims 1-5, wherein the processor is further configured to execute the instructions to: Visit at least one of the following: historical surgical session data generated during one or more additional surgical sessions preceding the surgical session, and global surgical session data generated based on operations performed by one or more computer-assisted surgical systems other than the computer-assisted surgical system; as well as applying at least one of the historical surgical session data and the global surgical session data to a machine learning model executed by at least one physical computing device; Wherein the machine learning model uses at least one of the historical surgical session data and the global surgical session data to associate patterns of surgical system operation with multiple events.

14. A method comprising: accessing, by the camera control system, surgical session data for a surgical session including performance of one or more operations performed by a computer-assisted surgery system; identifying, by the camera control system, an event associated with the surgical session based on the surgical session data; determining, by the camera control system, a location associated with the event; as well as Automatic adjustment of a camera's field of view is directed by the camera control system in response to the determination of the location of the event to capture a particular field of view of the location associated with the event.

15. The method of claim 14, wherein determining the location associated with the event comprises: In response to identifying the event, identifying a component of the computer-assisted surgery system associated with the event; as well as A location of the component associated with the event is determined.

16. The method of claim 15, wherein the determination of the position of the component associated with the event is based on kinematic data representing the positioning of the component associated with the event.

17. The method of claim 15, wherein the determination of the position of the component associated with the event is based on at least one of a marker provided on the component associated with the event and a signal emitted from the component associated with the event.

18. The method of claim 14, wherein directing the automatic adjustment of the field of view of the camera comprises directing adjustment of at least one of an orientation of the camera and a positioning of the camera to capture a particular field of view of the location associated with the event.

19. The method according to any one of claims 14 to 18, further comprising: The camera's image and three-dimensional ("3D") positioning data tracked by the computer-assisted surgery system are registered to a common 3D space by the camera control system.

20. A system comprising: a memory for storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to: receiving, from a user device communicatively paired with a computer-assisted surgery system during a surgical session, user input indicating workflow segmentation of the surgical session; identifying an event associated with the surgical session based at least in part on the user input; determining a location associated with the identified event; as well as In response to the determination of the location of the event, automatic adjustment of a field of view of a camera is directed to capture a particular field of view of the location associated with the event.

Images