Camera control systems and methods for computer-assisted surgical system

Abstract

To provide camera control systems and methods for a computer-assisted surgical system.SOLUTION: A camera control system may access surgical session data for a surgical session, the surgical session including performance of one or more operations by a computer-assisted surgical system. The camera control system may identify, based on the surgical session data, an event associated with the surgical session, and may determine, based on the surgical session data, a location associated with the event. In response to the determination of the location of the event, the camera control system may direct an automatic adjustment of a view of a camera to capture a specific view of the location associated with the event.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Related Applications This application is related to the application filed on January 31, 2019, entitled "CAMERA CONTROL SYSTEMS AND METHODS FOR A COMPUTER-ASSISTED SURGICAL This application claims priority to U.S. Provisional Patent Application No. 62 / 799,729, entitled "DUAL SYSTEM," the contents of which are incorporated herein by reference in their entirety.

[0002] This application relates to a camera control system and method for a computer-assisted surgery system. [Background technology]

[0003] During a computer-assisted surgical procedure, such as a minimally invasive surgical procedure using a computer-assisted surgery system, a camera can capture images of a scene in the computer-assisted surgery system that is external to the patient. The computer-assisted surgery system can display the captured images to medical personnel (e.g., a surgeon and / or other members of the surgical team) to provide visualization of the computer-assisted surgery system. The visualized images assist the medical personnel in performing the surgical procedure. However, a user typically must manually position and move the camera to capture the desired view, which is time-consuming and interrupts the flow of the surgical procedure. As a result, the camera may not be fully utilized during the surgical procedure. Summary of the Invention

[0004] An exemplary system may include a memory storing instructions and a processor communicatively coupled to the memory, the processor configured to execute the instructions to access surgical session data relating to a surgical session, the surgical session including the performance of one or more actions by a computer-assisted surgery system; identify an event associated with the surgical session based on the surgical session data; determine a location associated with the event based on the surgical session data; and, in response to determining the event location, direct an automatic adjustment of a camera view to capture a particular view of the location associated with the event.

[0005] An exemplary method may include: a camera control system accessing surgical session data relating to a surgical session, the surgical session including the performance of one or more actions by a computer-assisted surgical system; the camera control system identifying an event associated with the surgical session based on the surgical session data; the camera control system determining a location associated with the event based on the surgical session data; and the camera control system, in response to determining the event location, directing automatic adjustment of a camera view to capture a particular view of the location associated with the event.

[0006] An exemplary system includes a memory storing instructions and a processor communicatively coupled to the memory, the processor configured to execute the instructions to: receive, during a surgical session, user input from a user device communicatively paired with the computer-assisted surgery system indicating a workflow segmentation of the surgical session; identify an event associated with the surgical session based at least in part on the user input; determine a location associated with the identified event based on surgical session data related to the surgical session; and, in response to determining the event location, direct an automatic adjustment of a camera view to capture a particular view of the location associated with the event. [Brief explanation of the drawings]

[0007] The accompanying drawings illustrate various embodiments and are a part of this specification. The illustrated embodiments are merely examples and are not intended to limit the scope of the present disclosure. Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. [Figure 1] FIG. 1 illustrates an exemplary computer-assisted surgery system according to principles described herein. [Figure 2] FIG. 1 illustrates another exemplary computer-assisted surgery system according to principles described herein. [Figure 3] FIG. 1 illustrates an exemplary camera control system according to principles described herein. [Figure 4] 4 illustrates an exemplary implementation of the camera control system shown in FIG. 3 in accordance with principles described herein. [Figure 5] 4 illustrates another exemplary implementation of the camera control system shown in FIG. 3 in accordance with principles described herein. [Figure 6] 1 illustrates an exemplary method by which events may be identified based on surgical session data and / or additional data in accordance with principles described herein. [Figure 7] 1 illustrates an exemplary method by which events may be identified based on surgical session data and / or additional data in accordance with principles described herein. [Figure 8] 1 illustrates an exemplary method by which events may be identified based on surgical session data and / or additional data in accordance with principles described herein. [Figure 9] FIG. 1 illustrates an exemplary table of event locations according to principles described herein. [Figure 10] FIG. 1 illustrates an exemplary table of event locations according to principles described herein. [Figure 11] 1 illustrates an exemplary camera control method according to principles described herein. [Figure 12]FIG. 1 illustrates an exemplary computing system according to principles described herein. DETAILED DESCRIPTION OF THE INVENTION

[0008] Described herein are camera control systems and methods for computer-assisted surgery systems. As described in more detail below, an exemplary camera control system can access surgical session data relating to a surgical session, the surgical session including the performance of one or more actions by the computer-assisted surgery system. The camera control system can identify an event associated with the surgical session based on the surgical session data, and can determine a location associated with the event based on the surgical session data. In response to determining the location of the event, the camera control system can direct automatic adjustment of the view of a camera to capture a particular view of the location associated with the event. In some examples, images captured by the camera can be displayed by a display device associated with the computer-assisted surgery system.

[0009] By way of example, during a minimally invasive surgical procedure performed in a surgical suite equipped with a computer-assisted surgery system, a camera control system may detect that a first manipulator arm, to which a surgical instrument is attached, is colliding with a second manipulator arm, thereby preventing movement of the tip of the surgical instrument coupled to the first manipulator arm. As a result, the camera control system may determine that the collision event location is the position of the first manipulator arm and direct automatic adjustment of a camera positioned within the surgical suite to capture a particular view of the first manipulator arm. Video from the camera may be displayed by a display device associated with a user control system used by the surgeon. In this way, the surgeon may quickly understand why the tip of the surgical instrument is stuck and quickly identify corrective measures without having to leave his or her position at the user control system.

[0010] The camera control systems and methods described herein may provide various advantages. For example, the systems and methods described herein may automatically direct the adjustment of cameras positioned within a surgical facility to capture images based on the detected context of the surgical session. Automatic camera adjustment may increase the efficiency of a surgical session by eliminating interruptions caused by manually adjusting cameras. Furthermore, the systems and methods described herein may provide relevant contextual information to surgical team members in real time during a surgical procedure, which may result in more effective and efficient collaboration and coordination among surgical team members and enable surgical team members to troubleshoot problems quickly and efficiently. Furthermore, the systems and methods may predict events that may occur during a surgical session and direct the presentation of contextual visual content related to such events, thereby enabling surgical team members to prepare for and / or resolve such events before they occur.

[0011] Numerous technical computing advantages may also be realized by the systems and methods described herein. For example, the systems and methods described herein may be configured to access, transform, and process data from different computing systems in a manner that enables the systems and methods to provide timely (e.g., real-time) information to various users via a variety of computing platforms. To this end, the systems and methods described herein may seamlessly integrate one or more special-purpose computing devices to process various types of data (e.g., by applying kinematic data, image data, sensor data, and / or surgical instrument data to one or more machine learning models) to identify events that occur or may occur during a surgical session and / or determine contextual information related to the events. Furthermore, the systems and methods described herein may utilize historical surgical session data generated during surgical sessions preceding a current surgical session to determine the context of the surgical session with reference to other, previous surgical sessions. In this manner, the systems and methods described herein may perform operations that humans alone cannot perform. Furthermore, the systems and methods described herein may improve the operation of computer-assisted surgical systems by increasing their efficiency, accuracy, and effectiveness.

[0012] Various embodiments will now be described in more detail with reference to the figures. The systems and methods described herein may provide one or more of the advantages set forth above and / or various additional and / or alternative advantages as set forth herein.

[0013] In some embodiments, the camera control system can operate as part of or in combination with a computer-assisted surgery system. As such, an exemplary computer-assisted surgery system is now described. The exemplary computer-assisted surgery system described is illustrative and not limiting. The camera control system can operate as part of or in combination with the computer-assisted surgery systems described herein and / or in combination with other suitable computer-assisted surgery systems.

[0014] 1 illustrates an exemplary computer-assisted surgery system 100 (surgical system 100). As shown, surgical system 100 may include an operating system 102, a user control system 104, an auxiliary system 106, and an auxiliary camera system 108 that are communicatively coupled to one another. In some examples, the camera control system described herein may be implemented by one or more of these components.

[0015] The surgical system 100 may be utilized by a surgical team to perform a computer-assisted surgical procedure on a patient 110 positioned on an operating table 112. As shown, the surgical team may include a surgeon 114-1, a nurse 114-2, an assistant 114-3, and an anesthesiologist 114-4, all of which may be collectively referred to as "surgical team members 114." Additional or alternative surgical team members may be present during a surgical session as may be useful in a particular embodiment. The surgical system 100 and the surgical team members 114 are located in a surgical suite 116. As used herein, "surgical suite" may include any area in which the surgical system 100 and the surgical team members 114 are located when performing a computer-assisted surgical procedure. For example, the surgical suite may include an operating room, a training facility, a specific area within a room, etc. In some examples, the surgeon 114-1 may be located in a room or location separate from the room or location in which the manipulation system 102, auxiliary systems 106, and / or other surgical team members 114 are located during the surgical procedure. In these examples, the surgical facility may include either or both locations.

[0016] While FIG. 1 depicts a minimally invasive surgical procedure in progress in a surgical suite 116, surgical system 100 may similarly be used to perform open surgical procedures or other types of surgical procedures, which may similarly benefit from the precision and convenience of surgical system 100. It should be understood that a surgical session in which surgical system 100 may be used may include not only the operative phase of a surgical procedure, as shown in FIG. 1 , but also pre-operative, post-operative, and / or other appropriate phases of a surgical procedure. A surgical procedure may include any procedure performed on a patient using manual and / or instrumented techniques to investigate, diagnose, or treat a patient's condition. Additionally, a surgical procedure may include any procedure not performed on a live patient, such as calibration procedures, training procedures, and experimental or research procedures.

[0017] 1, manipulation system 102 may include multiple manipulator arms 118 (e.g., manipulator arms 118-1 through 118-4) to which multiple surgical instruments (not shown) may be coupled. Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool having tissue interaction capabilities), medical tool, monitoring tool (e.g., an endoscope), sensing instrument (e.g., a force-sensing surgical instrument), diagnostic instrument, etc. that may be used in a computer-assisted surgical procedure (e.g., by being at least partially inserted into patient 110 and manipulated to perform a computer-assisted surgical procedure on patient 110). While manipulation system 102 is depicted and described herein as including four manipulator arms 118, it will be appreciated that manipulation system 102 may include only a single manipulator arm 118 or any other number of manipulator arms that may be useful in a particular implementation.

[0018] Each surgical instrument coupled to manipulator arm 118 may be positioned in a surgical field associated with a patient. A "surgical field" may, in certain instances, be located entirely within a patient and may include an area in or near a patient where a surgical procedure is to be performed, is being performed, or has been performed. For example, in the case of a minimally invasive surgical procedure performed on tissue internal to a patient, the surgical field may include the tissue, the anatomical structures underlying the tissue, as well as the space around the tissue where, for example, surgical instruments being used to perform the surgical procedure are located. In other instances, the surgical field may be at least partially located outside of or near the patient where a surgical procedure is to be performed, is being performed, or has been performed on the patient. For example, surgical system 100 may be used to perform an open surgical procedure such that a portion of the surgical field (e.g., the tissue being operated on) is internal to the patient, while another portion of the surgical field (e.g., the space around the tissue where one or more surgical instruments may be positioned) is external to the patient. A surgical instrument may be referred to as being positioned in or within the surgical field when at least a portion of the surgical instrument (e.g., a distal portion of the surgical instrument) is located within the surgical field.

[0019] The user control system 104 may be configured to facilitate control by the surgeon 114-1 of the manipulator arm 118 and the surgical instruments coupled to the manipulator arm 118. For example, the surgeon 114-1 may interact with the user control system 104 to remotely move or manipulate the manipulator arm 118 and the surgical instruments. To this end, the user control system 104 may provide the surgeon 114-1 with images (e.g., high-resolution stereo images) of the surgical area associated with the patient 110 captured by an imaging device (e.g., an endoscope). In particular examples, the user control system 104 may include a stereoscopic viewer having two displays that enable the surgeon 114-1 to view stereo images of the surgical area associated with the patient 110 and generated by a stereoscopic imaging system. The surgeon 114-1 may use the images to perform one or more procedures using one or more surgical instruments coupled to the manipulator arm 118.

[0020] To facilitate control of surgical instruments, user control system 104 may include a set of master controls (not shown). These master controls may be operated by surgeon 114-1 to control the movement of surgical instruments (e.g., by utilizing robotic and / or teleoperation technology). The master controls may be configured to detect a wide variety of hand, wrist, and finger movements by surgeon 114-1. In this manner, surgeon 114-1 may intuitively perform a surgical procedure using one or more surgical instruments.

[0021] User control system 104 may be further configured to facilitate control of other components of surgical system 100 by surgeon 114-1. For example, surgeon 114-1 may interact with user control system 104 to change the configuration or operating mode of surgical system 100, change the display mode of surgical system 100, generate additional control signals used to control surgical instruments attached to manipulator arm 118, facilitate switching of control from one surgical instrument to another, or perform any other suitable operation. To this end, user control system 104 may also include one or more input devices (e.g., foot pedals, buttons, switches, etc.) configured to receive input from surgeon 114-1.

[0022] The auxiliary system 106 may include one or more computing devices configured to perform primary processing operations for the surgical system 100. The one or more computing devices included in the auxiliary system 106 may control and / or coordinate operations performed by various other components of the surgical system 100 (e.g., the manipulation system 102, the surgical instruments attached to the manipulator arms 118, and / or the user control system 104). For example, a computing device included in the user control system 104 may send instructions to the manipulation system 102 via one or more computing devices included in the auxiliary system 106. As another example, the auxiliary system 106 may receive and process image data from the manipulation system 102 representing images captured by an imaging device attached to one of the manipulator arms 118.

[0023] In some examples, the auxiliary system 106 may be configured to present visual content to surgical team members 114 who may not have access to the images provided to the surgeon 114-1 by the user control system 104. To this end, the auxiliary system 106 may include a display monitor 120 configured to display, in one or more user interfaces, images of the surgical field (e.g., 2D images), information related to the patient 110 and / or the surgical procedure, and / or any other visual content that may be useful in a particular implementation. For example, the display monitor 120 may display an image of the surgical field along with additional content (e.g., graphical content, contextual information, etc.) displayed simultaneously with the image. In some embodiments, the display monitor 120 is implemented by a touchscreen display with which the surgical team members 114 can interact (e.g., via touch gestures) to provide user input to the surgical system 100.

[0024] The auxiliary camera system 108 may be configured to capture and provide visual content based on the detected context of the surgical procedure on one or more display devices of the surgical system 100. As shown in FIG. 1 , the auxiliary camera system 108 may include an auxiliary camera 122 (camera 122) coupled to a camera manipulation system 124. The camera 122 may be implemented by any suitable camera. In some examples, the camera 122 may be implemented by multiple imaging devices configured to provide stereoscopic images. While the auxiliary camera system 108 is depicted and described herein as including one camera 122, it will be appreciated that the auxiliary camera system 108 may include multiple cameras 122, as may be useful in certain implementations. For example, if the user control system 104 and the surgeon 114-1 are located remotely from other components of the surgical system 100, additional cameras may be positioned to capture and provide images of the user control system 104 and the surgeon 114-1.

[0025] Images (e.g., one or more still images or video images) captured by camera 122 may be transmitted to and / or displayed on any suitable display device associated with surgical system 100. For example, images captured by camera 122 may be transmitted to and / or displayed by display monitor 120, a stereoscopic viewer of user control system 104, a mobile device communicatively paired with surgical system 100 during a surgical session (e.g., a mobile device used by assistant 114-3), and / or a display device associated with a remote computing device. In some examples where surgical system 100 includes a dual user control system 104, such as for training surgeons who will use surgical system 100, images captured by camera 122 are displayed on a display device associated with the second user control system 104 for viewing by another user (e.g., a remote supervisor monitoring the surgical session and / or training the surgeon). Additionally or alternatively, image data representing images captured by camera 122 may be streamed and / or recorded by a remote computing device, thereby enabling playback of the images at a later time (e.g., after the surgical session).

[0026] The camera manipulation system 124 may be configured to automatically adjust the position and / or orientation of the camera 122 to capture a particular view 126 within the surgical suite 116. As used herein, the "position" of the camera 122 within the surgical suite 116 may refer to a particular location of the camera 122 within three-dimensional (3D) space, and the "orientation" of the camera 122 may refer to the imaging direction (e.g., view 126) of the camera 122, which may be any direction within the 3D space.

[0027] The camera manipulation system 124 may be implemented by any suitable mechanism configured to adjust the position and / or orientation of the camera 122. For example, as shown in FIG. 1 , the camera manipulation system 124 may include a camera manipulator arm 128 to which the camera 122 is coupled by a camera pivot 130. The camera manipulator arm 128 may be configured to adjust the position and / or orientation of the camera 122 through movement of the camera manipulator arm 128. The camera pivot 130 may be configured to adjust the orientation of the camera 122. For example, the camera pivot 130 may rotate the camera 122 in a horizontal plane, such as by panning from left to right (as viewed from the imaging direction of the camera 122). Additionally or alternatively, the camera pivot 130 may tilt the camera 122 up or down in a vertical plane, such as by adjusting the imaging direction of the camera 122 up or down (i.e., at an angle relative to the horizontal plane). The camera manipulation system 124 may include additional or alternative components as may be useful in particular implementations. For example, the camera manipulation system 124 may include various gears, motors, arms, pivots, joints, bearings, and any other electrical and / or mechanical components that may facilitate adjusting the position and / or orientation of the camera 122.

[0028] The camera manipulation system 124 of the auxiliary camera system 108 may be configured to adjust the position and / or orientation of the camera 122 in any suitable manner. In some examples, the position and / or orientation of the camera manipulation system 124 may be adjusted manually, such as by a user pushing or moving the camera 122 to a desired position or by manipulating one or more electronic controls of the camera manipulation system 124. Additionally or alternatively, the position and / or orientation of the camera 122 may be adjusted automatically (e.g., without user input) based on the context of the surgical session (e.g., based on events related to the surgical session), as described in more detail below.

[0029] As shown in FIG. 1 , the auxiliary camera system 108 (e.g., camera operation system 124) is coupled to the operation system 102. In alternative examples, the auxiliary camera system 108 may be coupled to any other component of the surgical system 100 suitable for a particular implementation, such as the user control system 104, the auxiliary system 106, or a cart 132 (e.g., instrument cart 132-2). In additional examples, the auxiliary camera system 108 may be separate from the operation system 102, the user control system 104, and the auxiliary system 106. For example, the camera operation system 124 may be mounted on a wall of the operating room. Alternatively, the auxiliary camera system 108 may be a standalone system within the surgical suite 116, as shown in FIG. 2 .

[0030] Figure 2 is the same as Figure 1 except that the auxiliary camera system 108 includes a movable camera cart 202. A camera manipulation system 124 is supported on the camera cart 202, thereby allowing the surgical team member 114 to easily position the camera 122 at a desired location within the surgical suite 116 by moving the camera cart 202. Additionally, the surgical team member 114 can move the camera cart 202 as needed during the surgical procedure.

[0031] Referring again to FIG. 1 , the surgical system 100 may include one or more mobile carts 132 for holding certain components of the surgical system 100 and / or supplies used during a surgical procedure. For example, one or more computing devices included in the auxiliary system 106 may be housed within the auxiliary cart 132-1. Additionally, surgical instruments not coupled to the manipulator arm 118 may be stored in the instrument cart 132-2 for easy access (e.g., by an assistant 114-3) when coupling surgical instruments to the manipulator arm 118 and / or when replacing surgical instruments during a surgical procedure. The supply cart 132-3 may be used, for example, to store drugs and / or other agents (e.g., fluorescent contrast agents) that may be administered to the patient 110 during a surgical procedure by the anesthesiologist 114-4.

[0032] Various components of surgical system 100 may include one or more displacement transducers, orientation sensors, and / or position sensors (hereinafter "surgical system sensors") that are used to generate raw (i.e., uncorrected) kinematic information. For example, user control system 104, auxiliary system 106, auxiliary camera system 108, surgical table 112, manipulator arm 118, surgical instruments attached to manipulator arm 118, and cart 132 may include one or more surgical system sensors. Surgical system 100 (e.g., auxiliary system 106) may be configured to use the kinematic information to track (e.g., determine the position of) and / or control various components of surgical system 100.

[0033] The components of surgical system 100 may be communicatively coupled to one another in any suitable manner. For example, as shown in FIG. 1 , manipulation system 102, user control system 104, auxiliary system 106, auxiliary camera system 108, and cart 118 may be communicatively coupled via control lines 134, which may represent any wired or wireless communication link that may be useful in a particular implementation. To this end, manipulation system 102, user control system 104, auxiliary system 106, auxiliary camera system 108, and cart 132 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0034] As previously discussed, the position and / or orientation of camera 122 may be automatically adjusted based on the context of the surgical session to capture a particular view (e.g., view 126) within surgical suite 116. Figure 3 illustrates an exemplary camera control system 300 configured to automatically control certain operations of camera systems associated with a computer-assisted surgery session (e.g., auxiliary camera system 108, an endoscope coupled to manipulator arm 118, multiple cameras associated with a computer-assisted surgery system, etc.) to provide contextual visual content related to events associated with the computer-assisted surgery system. As shown, camera control system 300 includes, but is not limited to, a storage facility 302 and a processing unit 304 selectively and communicatively coupled to each other. Devices 302 and 304 may include a device (or facility) 304. Devices 302 and 304 may each include or be implemented by hardware and / or software components (e.g., a processor, memory, a communication interface, instructions stored in the memory for execution by the processor, etc.). In some examples, devices 302 and 304 may be distributed across multiple devices and / or multiple locations as may be useful for a particular implementation.

[0035] The storage device 302 may maintain (e.g., store) executable data used by the processing device 304 to perform any of the operations described herein. For example, the storage device 302 may store instructions 306 that may be executed by the processing device 304 to perform any of the operations described herein. The instructions 306 may be implemented by any suitable application, software, code, and / or other executable data instance.

[0036] The storage device 302 may also maintain any data received, generated, managed, used, and / or transmitted by the processing device 304. For example, as described in more detail below, the storage device 302 may maintain surgical session data, user profile data, user input data, etc.

[0037] The processing unit 304 may be configured to perform various processing operations (e.g., execute instructions 306 stored in the memory device 302 to perform the processing operations) related to automatically adjusting the view of a camera to capture a particular view of a location associated with an event related to a surgical session. For example, the processing unit 304 may access surgical session data related to the surgical session. The surgical session may include the performance of one or more actions by a computer-assisted surgical system (e.g., surgical system 100). The processing unit 304 may identify events related to the surgical session, such as events that have occurred or events that may occur, based on the surgical session data. Based on the surgical session data, the processing unit 304 may determine a location associated with the event and, in response to determining the event location, may direct an automatic adjustment of the view of a camera (e.g., camera 122) to capture a particular view of the location associated with the event. These and other operations that may be performed by the processing unit 304 are described in more detail herein.

[0038] In some examples, the camera control system 300 is implemented entirely by the computer-assisted surgery system itself. For example, the camera control system 300 may be implemented by one or more computing devices included in the surgical system 100 (e.g., in one or more computing devices included in the operation system 102, the user control system 104, the auxiliary system 106, and / or the auxiliary camera system 108).

[0039] 4 illustrates another exemplary implementation 400 of the camera control system 300. In the implementation 400, a remote computing system 402 may be communicatively coupled to the surgical system 100 via a network 404. The remote computing system 402 may include one or more computing devices (e.g., servers) configured to perform any of the operations described herein. In some examples, the camera control system 300 may be implemented entirely by the remote computing system 402. Alternatively, the camera control system 300 may be implemented by both the remote computing system 402 and the surgical system 100.

[0040] Network 404 may be a local area network, a wireless network (e.g., Wi-Fi), a wide area network, the Internet, a cellular data network, and / or any other suitable network. Data may flow between components connected to network 404 using any communications technologies, devices, media, and protocols as may be useful in a particular implementation.

[0041] FIG. 5 illustrates another exemplary implementation 500 of the camera control system 300. FIG. 5 is similar to FIG. 4, except that a user device 502 may be communicatively paired with the surgical system 100 and connected to the network 404. As shown, the user device 502 may communicate with the remote computing system 402 and / or the surgical system 100 via the network 404. Additionally or alternatively, the user device 502 may communicate with the surgical system 100 via a direct connection 504 (e.g., a direct wired connection and / or a direct wireless connection, such as a Bluetooth connection, a near field communication connection, etc.). In some examples, the camera control system 300 may be implemented by the remote computing system 402, the surgical system 100, and / or the user device 502.

[0042] User device 502 may be any suitable computing device configured to receive and send user input and / or present visual content. For example, user device 502 may be a mobile device (e.g., a mobile phone, a handheld device, a tablet computing device, a laptop computer, a personal computer, etc.), a wearable device (e.g., a smartwatch device, an activity tracker, a head-mounted display, a virtual or augmented reality device, etc.), and / or a display device (e.g., a television, a projector, a monitor, a touchscreen display, etc., without limitation).

[0043] As shown, a user 506 (e.g., a surgical team member 114) can use or otherwise access the user device 502. In some examples, the user 506 may have to log in to the user device 502 or to an application executed by the user device 502 in order to use and / or interact with the user device 502. In some examples, the application executed by the user device 502 may be used by the user 506 to provide user input regarding workflow segmentation of a surgical session and to display images captured by a camera (e.g., camera 122). For example, the user 506 may provide input regarding tasks (e.g., one or more actions performed by a surgical team member or a computer-assisted surgery system) to be completed by the user 506 or another surgical team member, the stage of the surgical procedure, the patient's condition, a component failure of the surgical system, a change in surgical team personnel, and any other information as appropriate for a particular implementation. As described in more detail below, such user input may be utilized by the camera control system 300 to more accurately identify surgical session events that occur or may occur during a surgical session.

[0044] 5 shows only one user device 502 communicatively paired with the surgical system 100 and connected to the network 404, any number of user devices 502 may be paired with the surgical system 100 and / or connected to the network 404. For example, each surgical team member 114 may utilize a user device 502 during a surgical session to provide input regarding the workflow of the surgical session and to display images captured by a camera (e.g., camera 122).

[0045] Various operations, and examples of these operations, that may be performed by camera control system 300 (e.g., by processing unit 304 of camera control system 300) will now be described. It will be recognized that the operations and examples described herein are merely illustrative of the many different types of operations that may be performed by camera control system 300.

[0046] The camera control system 300 can direct the automatic adjustment of a camera's view (e.g., view 126 of camera 122) to capture a particular view of a location associated with a surgical session event. To this end, the camera control system 300 can access surgical session data related to the surgical session and identify events associated with the surgical session based on the surgical session data. Various examples of these operations are now provided.

[0047] In some examples, the surgical session data accessed by camera control system 300 may be generated during a surgical session and may be based on one or more operations performed by a computer-assisted surgery system (e.g., surgical system 100) during the surgical session. The operations performed by the computer-assisted surgery system may include any mechanical, electrical, hardware, and / or software-based operations that may be useful in a particular implementation. The surgical session data may be generated by the computer-assisted surgery system (e.g., by one or more components within surgical system 100), by one or more components coupled to the computer-assisted surgery system during the surgical session (e.g., one or more surgical instruments coupled to manipulator arm 118), by a user device communicatively paired with the computer-assisted surgery system during the surgical session (e.g., user device 502), and / or by any other device associated with the computer-assisted surgery system that may be useful in a particular implementation. In a scenario in which the camera control system 300 is implemented entirely by the remote computing system 402, surgical session data may additionally or alternatively be generated by the remote computing system 402, while, for example, the remote computing system 402 tracks operations performed by the surgical system 100.

[0048] The surgical session data generated during a surgical session may include various types of data. For example, the surgical session data generated during a surgical session may include kinematic data, image data, sensor data, surgical instrument data, and / or any other type of data that may be useful in a particular implementation.

[0049] The kinematic data may represent the position, pose, and / or orientation of components within and / or coupled to the computer-assisted surgery system. For example, the kinematic data may represent the position, pose, and / or orientation of the manipulator arm 118 and / or a surgical instrument coupled to the manipulator arm 118. As another example, the kinematic data may represent the position of the manipulation system 102, the user control system 104, the auxiliary camera system 108, the surgical table 112, and / or the cart 132.

[0050] The image data may represent one or more images captured by an imaging device coupled to the computer-assisted surgery system. For example, the image data may represent one or more images captured by an imaging device (e.g., a stereoscopic endoscope) coupled to the manipulator arm 118. The one or more images may constitute one or more still images and / or videos captured by the imaging device. In some examples, the camera control system 300 may access the image data by receiving (e.g., over a network) images output by the imaging device. In additional or alternative examples, the image data may include image data generated by an imaging device external to the patient, such as the camera 122.

[0051] The sensor data may include any data generated by surgical system sensors included in or associated with the computer-assisted surgery system. The sensor data may represent any sensed parameter that may be useful in a particular implementation. For example, the sensor data may indicate whether the operating table 112 is moving or whether the surgeon 114-1 is actively interacting with the user control system 104.

[0052] The surgical instrument data may include any data generated by the surgical instrument and may represent the identification (ID) of the surgical instrument, the operational state of the surgical instrument (e.g., open, closed, charging, idle, etc.), a fault code for the surgical instrument, etc.

[0053] In some examples, the camera control system 300 may additionally or alternatively access surgical session data generated by the computer-assisted surgery system during one or more other surgical sessions, e.g., prior to the current surgical session. For example, the camera control system 300 may generate surgical session data during a first surgical session in which a first surgical procedure is performed on a first patient using the computer-assisted surgery system. The camera control system 300 may also generate additional surgical session data during a second surgical session in which a second surgical procedure is performed on that patient or another patient using the computer-assisted surgery system. During the second surgical session, the camera control system 300 may access both the surgical session data and the additional surgical session data. Surgical session data generated prior to the current surgical session may be referred to herein as “historical surgical session data.” As described below, the historical surgical session data may enable the camera control system 300 to more effectively identify and / or predict events that may occur during the second surgical session.

[0054] The camera control system 300 may additionally or alternatively access surgical session data based on operations performed by one or more computer-assisted surgery systems other than the computer-assisted surgery system being used during a particular surgical session. For example, the camera control system 300 may access surgical session data generated during multiple separate computer-assisted surgery sessions within a particular medical center, hospital network, and / or any other group. This type of surgical session data is referred to herein as "global surgical session data" and, as described below, may enable the camera control system 300 to more effectively identify and / or predict events that may occur during a particular surgical session in which a surgical procedure is performed using a particular computer-assisted surgery system included in the group.

[0055] In some examples, the camera control system 300 may provide a user interface configured to enable a user to define a particular group of computer-assisted surgery sessions and / or computer-assisted surgery systems from which the camera control system 300 may access surgical session data.

[0056] The camera control system 300 can identify events associated with a computer-assisted surgery session (surgical session events) based on surgical session data related to the surgical session, historical surgical session data, and / or global surgical session data. A surgical session event can include any distinct action or behavior that occurs or can occur with respect to the computer-assisted surgery system during a surgical session. A surgical session event can occur during a pre-operative phase, an operative phase, and / or a post-operative phase of a surgical procedure.

[0057] For example, a surgical session event may include any action or behavior associated with various pre-operative activities. Exemplary pre-operative activities may include, but are not limited to, admitting a patient (e.g., admitting a patient to a medical facility, receiving a patient's relevant documentation, etc.), preparing an operating room, sterilizing surgical instruments, testing a computer-assisted surgery system and equipment, draping a computer-assisted surgery system (i.e., covering one or more components of a computer-assisted surgery system, such as the manipulator arm 118, with a sterilized or protective covering), preparing a patient for a surgical procedure (e.g., checking a patient's vital signs, providing intravenous fluids, administering anesthesia to the patient, transporting the patient to the operating room), and targeting a computer-assisted surgery system to a patient (e.g., positioning the manipulation system 102 at the patient's bedside and positioning or configuring one or more manipulator arms 118).

[0058] Surgical session events may additionally or alternatively include any actions or behaviors associated with various surgical phase operations. Exemplary surgical phase operations may include, but are not limited to, opening a surgical area associated with a patient (e.g., by incising tissue external to the patient), inserting surgical instruments into the patient, performing a surgical operation on the patient's tissue (e.g., by cutting tissue, repairing tissue, suturing tissue, cauterizing tissue, etc.), and closing a surgical area associated with the patient (e.g., removing surgical instruments from the patient, suturing closed incisions, dressing any wounds).

[0059] Surgical session events may additionally or alternatively include any actions or behaviors associated with various post-operative phase activities. Exemplary post-operative phase activities may include, but are not limited to, removing the computer-assisted surgery system from the patient (e.g., removing the operating system 102 from the patient's bedside), patient care and recovery activities (e.g., transporting the patient from the operating room, monitoring the patient while the patient recovers from the surgical procedure, etc.), cleaning the operating room, cleaning the computer-assisted surgery system and / or surgical instruments, receipt of debriefing documents by members of the surgical team, and patient discharge activities.

[0060] The camera control system 300 may identify surgical session events based on the surgical session data in any suitable manner. FIG. 6 illustrates an exemplary manner in which the camera control system 300 may identify surgical session events based on the surgical session data. As shown, the camera control system 300 may apply surgical session data 602 as input to an event detection heuristic 604. The event detection heuristic 604 may analyze the surgical session data 602 and output various instances of event data 606 (e.g., event data 606-1 through event data 606-N). Each instance of the event data 606 may represent a particular surgical session event identified by the event detection heuristic 604.

[0061] Event detection heuristics 604 may include any suitable heuristics, processes, and / or operations that may be implemented or executed by camera control system 300 and that may be configured to identify events based on surgical session data 602. By way of example, heuristics 604 may detect indicators and / or patterns within the surgical session data that indicate the occurrence of a particular surgical session event.

[0062] For example, kinematic data generated during a particular portion of a surgical session may indicate a surgical instrument moving in a suturing pattern. Further, the surgical instrument data may indicate that the surgical instrument used during the same portion of the surgical session is a needle driver. Based on this kinematic data and the surgical instrument data, the camera control system 300 can determine that a suturing event is occurring or has occurred.

[0063] As another example, image data representing images generated by an endoscope may indicate that a particular surgical instrument remains out of the field of view of the endoscope for a predetermined period of time, and such image data may indicate an idle event (e.g., the surgical instrument being idle).

[0064] In some examples, the surgical session data 602 may include historical surgical session data, as described above. In these examples, one of the event data instances 606 output by the event detection heuristics 604 may represent a surgical session event that the camera control system 300 predicts will occur based on the historical surgical session data. For example, the historical surgical session data may include surgical session data generated during multiple surgical sessions using a computer-assisted surgery system to perform the same type of surgical procedure. Based on this historical surgical session data, the event detection heuristics 604 may predict that a particular second event will occur following the occurrence of a particular first event.

[0065] In some examples, the surgical session data 602 may include global surgical session data, as described above. In these examples, one of the surgical event data instances 606 output by the event detection heuristics 604 may represent a surgical session event determined to occur based on the global surgical session data. For example, the global surgical session data may indicate that a particular kinematic data value for a particular surgical tool may indicate that the surgical tool is located within a predetermined distance from the patient's tissue. If the actual kinematic data of a surgical tool being used during the surgical session is less than or equal to this value, the event detection heuristics 604 may detect a surgical session event indicating that the surgical tool is actually located within the predetermined distance from the patient's tissue.

[0066] Event detection heuristics 604 can accept additional or alternative types of input as may be useful in a particular implementation. For example, Figure 7 is similar to Figure 6, but shows that event detection heuristics 604 can accept user profile data 702 (e.g., data representing the user profiles of one or more surgical team members involved in the surgical session) as additional input. In this configuration, event detection heuristics 604 can detect surgical session events based on both surgical session data 602 and user profile data 702.

[0067] To illustrate, user profile data 702 may include data representing a user profile of a surgeon (e.g., surgeon 114-1) involved in a surgical session. The surgeon's user profile, combined with the surgical session data, may indicate that the surgeon performs various operations in a particular order that is unique to the surgeon. Thus, event detection heuristics 604 may detect that particular surgical session events are occurring according to a particular order.

[0068] FIG. 8 illustrates another example of receiving additional or alternative types of input. FIG. 8 is similar to FIG. 6, but illustrates that the event detection heuristic 604 can accept user input data 802 as an additional input. The user input data 802 can represent information entered by a user through a computing device included in the computer-assisted surgery system (e.g., via the user control system 104 or the auxiliary system 106) or communicatively paired with the computer-assisted surgery system (e.g., via a user device 502). In this configuration, the event detection heuristic 604 can detect a surgical session event based on both the surgical session data 602 and the user input data 802. The user input data 802 can include, for example, information entered by an application executed by a user device associated with a member of the surgical team.

[0069] To illustrate, a user 506 (e.g., anesthesiologist 114-4) may input information indicating that patient 110 is fully sedated via an application currently running on user device 502. This information may be combined with surgical session data to indicate that the pre-operative phase of the surgical session is complete. Accordingly, event detection heuristics 604 may detect that a particular surgical session event, such as opening an incision site on patient 110, is likely to occur. Camera control system 300 may then direct automatic adjustment of camera 122 to capture a particular view of a location associated with the upcoming event, such as instrument cart 132-2.

[0070] In some examples, the event detection heuristics 604 may implement a machine learning model that may use historical surgical session data, global surgical session data, user profile data, user input data, or any combination or subcombination thereof to identify one or more unique patterns of surgical system behavior and associate surgical session events with the detected patterns of surgical system behavior. As the camera control system 300 collects more data, the event data 606 output by the event detection heuristics 604 may be updated or revised as needed.

[0071] When the camera control system 300 identifies a surgical session event, the camera control system 300 can determine a location within the surgical facility associated with the identified surgical session event (the event location). The camera control system 300 can then direct an automatic adjustment by a camera system (e.g., the auxiliary camera system 108) associated with the computer-assisted surgery system to automatically adjust the camera's view (e.g., the view 126 of the camera 122) to capture a particular view of the event location.

[0072] The camera control system 300 may determine the event location in any suitable manner. In some examples, the camera control system 300 may determine the event location by determining the location of a component (event component) associated with an identified surgical session event. To this end, in certain embodiments, the camera control system 300 may access an event location table to identify the component associated with the identified surgical session event. By way of example, FIG. 9 shows an exemplary event location table 900 that may be maintained or otherwise accessed by the camera control system 300. As shown in column 902, table 900 may include multiple entries representing various surgical session events that may occur during a surgical session (e.g., a manipulator arm collision event, an instrument change event, a bed movement event, a manipulator arm error event, a master controller collision event, and a manipulator arm no signal event). As shown in column 904, table 900 may also list the component (e.g., a particular manipulator arm, operating table, user control system, and auxiliary cart) associated with each surgical session event.

[0073] Once an event component is identified, the camera control system 300 may determine the location of the event component (component position) in any suitable manner. In some examples, the camera control system 300 may determine the component position based on surgical session data generated during the surgical session. For example, the camera control system 300 may be configured to determine the component position based on kinematic data generated during the surgical session for the event component. Additionally or alternatively, the event position table may list component position data representing the current position of each component listed in the event position table. Component position data may be entered at the start of the surgical session and updated throughout the surgical session based on surgical session data (e.g., kinematic data) generated during the surgical session.

[0074] In addition to or instead of accessing the kinematic data, the camera control system 300 can determine the location of a component based on tracking of the component. In some examples, the camera control system 300 can utilize marker-based computer vision tracking. To this end, unique markers (e.g., barcodes, colors, patterns, shapes, etc.) can be integrated into or otherwise attached to various components of the computer-assisted surgery system (e.g., the manipulator arm 118, the surgical instrument, the cart 132, etc.), and an event location table can list marker data representing the unique marker associated with each component. The camera control system 300 can access the event location table to identify a specific marker data instance associated with the event component. Then, based on the identified marker data instance, the camera control system 300 can determine the location of the component by detecting the marker associated with the event component in images captured by the camera system.

[0075] In an additional or alternative example, the camera control system 300 can utilize signal-based tracking to determine the location of a component. To this end, emitters may be integrated into or otherwise attached to various components of the computer-assisted surgery system (e.g., the manipulator arm 118, the surgical instrument, the cart 132, etc.), and the camera system may include sensors configured to detect signals emanating from the components. The emitters may be configured to emit any suitable signal, such as an infrared (IR) signal. Each emitter may be configured to emit a unique signal based, for example, on a unique wavelength, a unique blinking pattern, a unique frequency, etc. An event location table may list signal data representing the unique signal associated with each component. The camera control system 300 can access the event location table to identify a specific signal data instance associated with the event component. The camera control system 300 can then instruct the event component to emit that specific signal. Alternatively, the component may periodically emit the associated signal. The camera control system 300 may then identify the location of the component that matches the identified signal data instance associated with the event component as the source of the emitted signal.

[0076] Once the camera control system 300 determines the event location, the camera control system 300 can instruct the camera to automatically adjust the view of the camera to capture a particular view of the event location. In some examples, the camera control system 300 can cause a camera operating system to which the camera is coupled to adjust the position and / or orientation of the camera to capture a particular view of the event location. In alternative examples where the camera system is implemented by a standalone computing device, the camera control system 300 can send data representing the event location and commands to the camera system to adjust the position and / or orientation of the camera to capture a particular view of the event location.

[0077] In some examples where the camera control system determines the event location based on marker-based tracking or signal-based tracking, the camera control system 300 can direct adjustments to point the camera toward a marker or emitted signal associated with the event location.

[0078] In additional or alternative examples, the amount and direction of adjustment of the camera's position and / or orientation may be determined based on the spatial relationship between the camera and the event location. In some examples where the camera system is physically coupled (e.g., mounted) directly to a component of the surgical system, the spatial relationship between the camera and the event location may be determined based on surgical session data (e.g., kinematic data, sensor data, etc.). Thus, adjustment of the camera's field of view may be based on the surgical session data. For example, the camera control system 300 may determine the camera's position and orientation relative to the event location based on kinematic data and sensor data associated with the camera system. Based on the determined position and orientation of the camera, the camera control system 300 may direct automatic adjustment of the camera to capture a particular view of the event location.

[0079] In other examples where the camera system is not physically coupled to a component of the surgical system, the spatial relationship between the camera and the event location may not be readily available or determinable from the surgical session data alone. In such examples, images captured by the cameras of the camera system and 3D position data tracked by the computer-assisted surgery system can be aligned in a common 3D space to facilitate automatic adjustment of the camera's view to capture a particular view of the event location. Additionally or alternatively, the camera system (e.g., the camera 122 and / or the camera manipulation system 124) can be aligned in a common 3D space. As described above, the computer-assisted surgery system can use kinematic information generated from the surgical system sensors to track (e.g., determine the position of) various components of the surgical system in 3D space. Aligning the images captured by the cameras in the same 3D space used by the computer-assisted surgery system can enable the camera control system 300 to determine the spatial relationship between the camera and the event location using kinematic information associated with the computer-assisted surgery system, including the camera system. Such spatial relationship information can be utilized by the camera control system 300 to determine the direction and amount of adjustment to the camera's position and / or orientation to capture a view of the event location. Even when tracking-based techniques are used to determine the component's location (and thus the event location), alignment allows the camera's position and / or orientation to be estimated based on the surgical session data, thereby speeding up the tracker-based determination of the component's location.

[0080] The camera control system 300 can perform the alignment process once (e.g., as part of a calibration, setup procedure, or other initial alignment) and / or periodically or continuously after the initial alignment to refine the initial alignment (e.g., to account for changes in physical position (where various kinematic errors may exist)) and / or account for positional adjustments of surgical system components. Alignment can be performed by any suitable alignment technique, including, but not limited to, vision-based alignment, model-based alignment, and marker-based alignment.

[0081] To illustrate, images captured by a camera may be used for the registration process. For example, the camera control system 300 may determine the location of components included in a computer-assisted surgical system based on the images captured by the camera and using a trained neural network. In particular examples, this may include the camera control system 300 using a trained neural network to identify objects of interest depicted in the images, associate the objects of interest with components of the surgical system in any suitable manner (e.g., object recognition), and determine the location of the objects (e.g., image-based location).

[0082] The camera control system 300 can then perform an optimization to fit a model of the surgical system component to the determined position of the object (e.g., the position of the object in the image). In the case of a component such as the cart 132 positioned on the floor, the camera control system 300 can be configured to constrain the optimization to search only for solutions that are rotations and translations in the plane of the floor on which the component is located. This constrained optimization can provide faster and / or more accurate results than traditional optimization performed in all six degrees of freedom.

[0083] The registration process can be used by the camera control system 300 to determine missing links in a kinematic chain connecting a camera that is not physically coupled to a component of a computer-assisted surgery system. The camera control system 300 can represent the missing links as a transformation that defines an estimated spatial relationship between the camera and the component of the surgery system. The transformation can define rotations and translations that can be applied to transform data points from the camera's frame of reference to the computer-assisted surgery system's frame of reference (e.g., to transform coordinate points from the camera's coordinate system to the computer-assisted surgery system's coordinate system), and vice versa.

[0084] The camera control system 300 may use the missing link to complete the kinematic chain connecting the camera and surgical system so that the complete kinematic chain is known and accessible to the camera control system 300. The camera control system 300 may then use the kinematics of the camera system to track the position of the camera and determine the appropriate position and / or orientation adjustments of the camera to capture a view of the event location.

[0085] As previously described, images captured by the camera may be transmitted to and displayed on a display device associated with the computer-assisted surgery system and / or a display device associated with a remote computing device. In this manner, surgical team members, remote supervisors, remote technicians, etc. may view contextual visual content related to surgical session events and use the contextual visual content to, for example, perform one or more tasks during the surgical session (e.g., to perform a surgical procedure, monitor or train surgical team members, troubleshoot technical problems, etc.). An example of this process will now be described.

[0086] For example, if camera control system 300 detects that manipulator arm 118-1 is colliding with manipulator arm 118-2 (labeled "MA1_Collide_MA2" in table 900), camera control system 300 can determine that the detected event location is the location of manipulator arm 118-1 (labeled "MA1" in table 900). Based on this determination, camera control system 300 can utilize surgical session data (e.g., kinematic data) to determine the location of manipulator arm 118-1 within the surgical suite. Camera control system 300 can then instruct camera operation system 124 to adjust the position and / or orientation of camera 122 to capture a view of manipulator arm 118-1. The images captured by camera 122 can then be displayed on a display device associated with a surgical team member 114 (e.g., surgeon 114-1), thereby enabling surgical team member 114 to quickly identify (determine) why manipulator arm 118-1 is not moving.

[0087] As another example, if the camera control system 300 determines that no signal is detected from a surgical instrument coupled to the fourth manipulator arm 118-4 (labeled “MA4_NoSignal” in table 900), the camera control system 300 can determine that the detected event is at a location on the fourth manipulator arm 118-4 to which a surgical instrument should be connected (labeled “MA4” in table 900). Based on this determination, the camera control system 300 can utilize surgical session data (e.g., kinematic data) to determine the location of the fourth manipulator arm 118-4 within the surgical suite. The camera control system 300 can then instruct the camera operation system 124 to adjust the position and / or orientation of the camera 122 to capture a view of the fourth manipulator arm 118-4. The image captured by camera 122 may then be displayed on a display device associated with surgical team member 114 (e.g., a user device used by assistant 114-3), thus enabling surgical team member 114 to quickly determine whether the surgical instrument is properly connected to fourth manipulator arm 118-4.

[0088] In some examples, multiple surgical system components may be associated with a particular event. For example, FIG. 10 shows an exemplary event location table 1000 that may be maintained by or otherwise accessed by camera control system 300. As shown in column 1002, table 1000 may include multiple entries representing various events that may occur during a surgical session. As shown in column 1004, table 1000 may list the primary component associated with each event. As shown in column 1006, table 1000 may list the secondary components associated with each event. Camera control system 300 may then determine the event location as the location of the primary component associated with the detected event. The camera control system 300 can respond to a position switch event by switching the event position to the position of the secondary component.

[0089] In some examples, the position switch event may include the passage of a predetermined period of time (e.g., 20 seconds) since a particular view of the primary component was first displayed. In some examples, the user may specify the duration of the predetermined period.

[0090] Alternatively, the position switch event may be the receipt by camera control system 300 of a user input instructing camera control system 300 to switch to a secondary position. The user input may be received via any suitable computing device associated with the computer-assisted surgery system (e.g., user device 502, remote computing system 402, or a computing device included in user control system 104, auxiliary system 106, or camera system 108).

[0091] To illustrate, if camera control system 300 detects that a surgical instrument coupled to manipulator arm 118-3 does not move (labeled "MA3_NoMovement" in table 1000) when manipulated by surgeon 114-1, camera control system 300 can determine that the primary location of the detected event is the position of manipulator arm 118-3 (labeled "MA3" in table 1000). Based on this determination, camera control system 300 can utilize surgical session data (e.g., kinematic data) to determine the position of manipulator arm 118-3 within the surgical suite. Camera control system 300 can then instruct auxiliary camera system 108 to adjust the view of camera 122 to capture a particular view of manipulator arm 118-3. When the image captured by camera 122 is displayed on a display associated with the user control system (e.g., on a touchscreen or user device 502), the surgeon can quickly determine whether a surgical instrument is not properly coupled to manipulator arm 118-3 or whether manipulator arm 118-3 is colliding with another manipulator arm. The surgeon can then provide input via user control system 104 or user device 502 (e.g., via a touchpad or foot pedal on user control system 104) to view a secondary position. In response to this user input, camera control system 300 can instruct auxiliary camera system 108 to adjust the view of camera 122 to capture a particular view of auxiliary cart 132-1. This allows the surgeon to determine whether the surgical instrument cables are properly connected to auxiliary cart 132-1.

[0092] In some examples, a user may be able to manually adjust the camera view after it has been adjusted to capture a view of the event location. Manual adjustments may include position and / or orientation adjustments, as well as optical adjustments (such as zoom and focus adjustments).

[0093] In some examples, camera control system 300 may be configured to provide contextual information in addition to images captured by the camera for display by a display device associated with a computer-assisted surgery session. For example, camera control system 300 may identify contextual content associated with an identified surgical session event and / or an identified event location. The contextual content may be stored, for example, in table 900 or table 1000. The contextual content may include, for example, a message displayed along with an image captured by the camera.

[0094] By way of example, the camera control system 300 may determine that the surgical procedure has progressed to the stapling stage based on surgical session data, user profile data, and / or user input data. As a result, the camera control system 300 may direct an automatic adjustment of the camera to capture a particular view of the manipulator arm and direct the display device to display an image of the manipulator arm along with a message such as, "Please replace this surgical instrument with a stapler instrument."

[0095] Additionally or alternatively, camera control system 300 may be configured to provide context information in ways other than a display device, such as by directing a lighting device (e.g., task light, laser light, etc.) to illuminate an event location. For example, camera control system 300 may direct a laser device in a computer-assisted surgery session to adjust illumination for a manipulator arm to which a surgical instrument is to be coupled. Adjustments to the lighting device may be accomplished in a manner similar to the adjustments to the cameras described herein.

[0096] Figure 11 illustrates an exemplary camera control method 1100. While Figure 11 illustrates exemplary operations according to one embodiment, other embodiments may omit, add, rearrange, combine, and / or modify any of the steps illustrated in Figure 11. One or more of the operations illustrated in Figure 11 may be performed by camera control system 300, any components included therein, and / or any implementation thereof.

[0097] In step 1102, the camera control system accesses surgical session data relating to a surgical session, the surgical session including the performance of one or more actions by the computer-assisted surgery system. Step 1102 can be performed in any of the ways described herein.

[0098] The camera control system identifies events associated with the surgical session based on the surgical session data in step 1104. Step 1104 can be performed in any of the ways described herein.

[0099] In step 1106, the camera control system determines a location associated with the event based on the surgical session data. Step 1106 can be performed in any of the ways described herein.

[0100] In step 1108, the camera control system, in response to determining the event location, directs automatic adjustment of the camera's view to capture a particular view of the location associated with the event. Step 1108 can be performed in any of the ways described herein.

[0101] In some examples, a non-transitory computer-readable medium storing computer-readable instructions may be provided in accordance with the principles described herein. The instructions, when executed by a processor of a computing device, may direct the processor and / or computing device to perform one or more actions, including one or more operations (procedures) described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0102] Non-transitory computer-readable media referred to herein may include any non-transitory storage medium that participates in providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by a processor of the computing device). For example, non-transitory computer-readable media may include, but are not limited to, any combination of non-volatile and / or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tape, etc.), ferroelectric random access memory (RAM), and optical disks (e.g., compact disks, digital video disks, Blu-ray® disks, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0103] FIG. 12 illustrates an exemplary computing device 1200 that may be specially configured to execute one or more processes described herein. As shown in FIG. 12, computing device 1200 may include a communication interface 1202, a processor 1204, a storage device 1206, and an input / output (I / O) module 1208, communicatively coupled to each other via a communication infrastructure 1210. While an exemplary computing device 1200 is illustrated in FIG. 12, the components illustrated in FIG. 12 are not intended to be limiting. Additional or alternative components may be used in other embodiments. The components of computing device 1200 illustrated in FIG. 12 are configured as described in further detail herein.

[0104] Communications interface 1202 may be configured to communicate with one or more computing devices. Examples of communications interface 1202 include, but are not limited to, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0105] Processor 1204 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more instructions, processes, and / or operations described herein. Processor 1204 may perform operations by executing computer-executable instructions 1212 (e.g., applications, software, code, and / or other executable data instances) stored on storage device 1206.

[0106] Storage device 1206 may include one or more data storage media, devices, or configurations, and may use any type, form, and combination of data storage media and / or devices. For example, storage device 1206 may include, but is not limited to, any combination of non-volatile and / or volatile media described herein. Electronic data, including data described herein, may be stored temporarily and / or permanently in storage device 1206. For example, data representing computer-executable instructions 1212 configured to instruct processor 1204 to perform any of the operations described herein may be stored in storage device 1206. In some examples, data may be located in one or more databases residing in storage device 1206.

[0107] I / O module 1208 may include one or more I / O modules configured to receive user input and provide user output. One or more I / O modules may be used to receive input for a single virtual experience. I / O module 1208 may include any hardware, firmware, software, or combination thereof that supports input and output functionality. For example, I / O module 1208 may include hardware and / or software to capture user input, including, but not limited to, a keyboard or keypad, a touchscreen component (e.g., a touchscreen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0108] I / O module 1208 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In particular embodiments, I / O module 1208 is configured to provide graphical data to a display for presentation to a user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may be useful in a particular implementation.

[0109] In some examples, any of the systems, computing devices, and / or other components described herein may be implemented by computing device 1200. For example, processing unit 304 may be implemented by processor 1204, and storage device 302 may be implemented by storage device 1206.

[0110] In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes can be made to the exemplary embodiments, and additional embodiments can be implemented, without departing from the scope of the invention as set forth in the following claims. For example, certain features of one embodiment described herein can be combined with or substituted for features of other embodiments described herein. Accordingly, the description and drawings should be regarded in an illustrative sense, and not in a restrictive sense.

[0111] The following describes the scope of the claims as originally filed as an example. [Example 1] 1. A system comprising: a memory for storing instructions; a processor communicatively coupled to the memory; The processor executes the instructions to: accessing surgical session data relating to a surgical session, the surgical session including performance of one or more actions by a computer-assisted surgery system; identifying an event associated with the surgical session based on the surgical session data; determining a location associated with the event based on the surgical session data; and and in response to the determining the location of the event, directing an automatic adjustment of a camera view to capture a particular view of the location associated with the event. system. [Example 2] The determining of the location associated with the event comprises: In response to the identification of the event, identifying a component of the computer-assisted surgery system associated with the event; and determining a location of the component relative to the event. system. [Example 3] 3. The system of claim 2, wherein the determination of the position of the component relative to the event is based on kinematic data representing the position of the component relative to the event. [Example 4] The system of Example 2, wherein the determination of the location of the component associated with the event is based on at least one of a marker provided on the component associated with the event and a signal emitted from the component associated with the event. [Example 5] The system of Example 1, wherein the instruction to automatically adjust the view of the camera includes instructing an adjustment of at least one of the camera's orientation and the camera's position to capture the particular view of the location associated with the event. [Example 6] The system of Example 1, wherein the processor is further configured to execute the instructions to align the camera image and three-dimensional (3D) position data tracked by the computer-assisted surgery system into a common 3D space. [Example 7] 2. The system of claim 1, wherein the camera is coupled to a component of the computer-assisted surgery system. [Example 8] The system of Example 1, wherein the surgical session data includes data representing the one or more actions performed by the computer-assisted surgery system during the surgical session. [Example 9] The system of example 1, wherein the surgical session data includes kinematic data representing at least one of a position, a posture, and an orientation of components of the computer-assisted surgical system. [Example 10] The system of Example 1, wherein the surgical session data includes image data representing one or more images captured by at least one of the camera and imaging device coupled to a manipulator arm of the computer-assisted surgical system. [Example 11] The system of Example 1, wherein the surgical session data includes data generated based on user input received during the surgical session via a user device communicatively paired with the computer-assisted surgery system during the surgical session. [Example 12] The system of Example 11, wherein the user input specifies a task to be performed by the computer-assisted surgery system or a member of a surgical team during the surgical session. [Example 13] The processor executes the instructions to historical surgical session data generated during one or more additional surgical sessions preceding the surgical session; and accessing at least one of global surgical session data generated based on operations performed by one or more computer-assisted surgery systems other than the computer-assisted surgery system; applying at least one of the historical surgical session data and the global surgical session data to a machine learning model executed by at least one physical computing device; The system of Example 1, wherein the machine learning model uses at least one of the historical surgical session data and the global surgical session data to associate patterns of surgical system behavior with multiple events. [Example 14] 1. A method, comprising: accessing, by a camera control system, surgical session data relating to a surgical session, the surgical session including performance of one or more actions by a computer-assisted surgery system; the camera control system identifying events associated with the surgical session based on the surgical session data; the camera control system determining a location associated with the event based on the surgical session data; the camera control system, in response to the determination of the location of the event, directing automatic adjustment of a camera view to capture a particular view of the location associated with the event; method. [Example 15] The step of determining the location associated with the event comprises: In response to the identification of the event, identifying a component of the computer-assisted surgery system associated with the event; determining a location of the component relative to the event. [Example 16] 16. The method of claim 15, wherein the determining the position of the component relative to the event is based on kinematic data representing the position of the component relative to the event. [Example 17] The method of Example 15, wherein the step of determining the position of the component associated with the event is based on at least one of a marker provided on the component associated with the event and a signal emitted from the component associated with the event. [Example 18] The method of example 14, wherein the step of instructing the automatic adjustment of the view of the camera includes a step of instructing an adjustment of at least one of the orientation of the camera and the position of the camera to capture the particular view of the location associated with the event. [Example 19] 15. The method of example 14, further comprising the step of the camera control system registering the image of the camera and three-dimensional (3D) position data tracked by the computer-assisted surgery system into a common 3D space. [Example 20] 1. A system comprising: a memory for storing instructions; a processor communicatively coupled to the memory; The processor executes the instructions to: receiving, during a surgical session, from a user device communicatively paired with the computer-assisted surgery system, a user input indicating a workflow segmentation of the surgical session; identifying an event associated with the surgical session based at least in part on the user input; determining a location associated with the identified event based on surgical session data relating to the surgical session; and configured to, in response to the determination of the location of the event, direct an automatic adjustment of a camera view to capture a particular view of the location associated with the event. system.

Claims

1. 1. A system comprising: a memory for storing instructions; a processor communicatively coupled to the memory; The processor executes the instructions to: accessing surgical session data relating to a surgical session, the surgical session including performance of one or more actions by a computer-assisted surgery system; determining, based on the surgical session data, that an event associated with the surgical session is likely to occur; determining a location associated with the event; and in response to the determination of the location of the event, directing an automatic adjustment of a camera view to capture a particular view of the location associated with the event before the event occurs. system.

2. The system of claim 1 , wherein the determination that the event associated with the surgical session is likely to occur is further based on user profile data representing user profiles of team members associated with the surgical session.

3. The system of claim 2 , wherein the user profile data indicates a pattern of operation of the computer-assisted surgery system specific to the team member.

4. the process further includes providing an image of the view of the location captured by the camera to a display device and displaying it on the display device; The system of claim 1 , wherein the determining that the event associated with the surgical session is likely to occur is further based on user profile data representing a user profile of a user of the display device.

5. the computer-assisted surgery system includes an operation system and a user control system; the manipulation system includes a manipulator arm configured to couple with a surgical instrument; the user control system is configured to control at least one of the manipulator arm or the surgical instrument based on user input; The system of claim 1 , wherein the location associated with the event includes a location associated with the operating system or the user control system.

6. the process further includes providing an image of the view of the location captured by the camera to a display device and displaying the image by the display device; The system of claim 5 , wherein the display device is remote from the user control system.

7. the computer-assisted surgery system further includes an additional user control system configured to remotely control at least one of the manipulator arm or the surgical instrument based on user input; The system of claim 6 , wherein the display device is included in the additional user control system.

8. The process includes: determining, based on the surgical session data, that an additional event associated with the surgical session is likely to occur; determining a location associated with the additional event; and 6. The system of claim 5, further comprising: in response to the determination of the location associated with the additional event, directing automatic adjustment of a view of an additional camera to capture a view of the location associated with the additional event before the additional event occurs.

9. The system of claim 8 , wherein the location associated with the additional event includes a location associated with the operating system.

10. The process includes: determining additional locations associated with the event; and The system of claim 1 , further comprising directing an adjustment of the view of the camera to capture a view of the additional location associated with the event.

11. The process includes: determining additional locations associated with the event; and 5. The system of claim 1, further comprising: instructing, in response to the determination of the additional location associated with the event, automatic adjustment of a view of an additional camera to capture a view of the additional location associated with the event before the event occurs.

12. The process includes: identifying contextual content associated with the event; and The system of claim 1 , further comprising providing the contextual content and an image of the view of the location captured by the camera to a display device and displaying the contextual content and the image on the display device.

13. The system of claim 1 , wherein the locations associated with the event include locations separate from a surgical area associated with an object of the surgical session.

14. 1. A method, comprising: a camera control system accessing surgical session data for a surgical session, the surgical session including performance of one or more actions by a computer-assisted surgery system; determining, by the camera control system based on the surgical session data, that an event associated with the surgical session is likely to occur; the camera control system determining a location associated with the event; the camera control system, in response to determining the location associated with the event, directing automatic adjustment of a camera view to capture a view of the location associated with the event before the event occurs; method.

15. A non-transitory computer-readable medium storing instructions that, when executed, cause at least one processor of a computing device to: the camera control system accessing surgical session data for a surgical session during the surgical session, the surgical session including performance of one or more actions by a computer-assisted surgery system; the camera control system determining, based on the surgical session data, that an event associated with the surgical session is likely to occur; the camera control system determining a location associated with the event; and causing the camera control system to execute a process including, in response to determining the location associated with the event, directing automatic adjustment of a camera view to capture a view of the location associated with the event before the event occurs; Non-transitory computer-readable medium.

Images