Method and system for controlling and displaying a video stream

Abstract

A method performed by a video controller is provided. The method receives a first video stream captured by an endoscope of a surgical system, and receives a second video stream including surgical data. The method displays the second video stream superimposed over an area of the first video stream, and determines that the second video stream will stop being superimposed. In response to determining that the second video stream will cease being overlaid, the method continues to display the first video stream.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. Provisional Patent Application Serial No. 63 / 256,514, filed on October 15, 2021, which is hereby incorporated by reference in its entirety. Technical Field

[0003] Various aspects of the present disclosure relate generally to surgical systems, and more particularly to surgical systems that control and display video streams. Other aspects are also described. Background Art

[0004] Minimally invasive surgery (MIS) such as laparoscopic surgery uses techniques designed to reduce tissue damage during surgical procedures. Laparoscopic procedures typically require multiple small incisions to be made in the patient's body (e.g., in the abdomen), through which several surgical tools (such as endoscopes, knives, graspers, and needles) are then inserted into the patient's body. Gas is injected into the abdomen, which inflates the abdomen, thereby providing more space around the end of the tool, making it easier for the surgeon to see (via an endoscope) and manipulate tissue at the surgical site. MIS can be performed faster and with less fatigue for the surgeon using a surgical robotic system, in which a surgical tool is operably attached to the distal end of a robotic arm, and a control system actuates the arm and its attached tool. When the surgeon manipulates a handheld user input device (UID), the end of the tool will simulate its position and orientation motion. A surgical robotic system can have multiple surgical arms, one or more of which have an attached endoscope, and other surgical arms have attached surgical instruments for performing certain surgical actions.

[0005] Control inputs from a user (e.g., a surgeon or other operator) are captured via one or more user input devices and then translated into control of the robotic system. For example, when a surgical tool is positioned at a surgical site of a patient, a tool driver having one or more motors may actuate one or more degrees of freedom of the surgical tool in response to user commands. Summary of the invention

[0006] Visualization systems are often used intraoperatively to image tissue, perform biopsies, surgical procedures, diagnose, and / or other medical procedures. As used herein, the term "intraoperative" refers to actions performed or events that occur during any medical procedure (invasive or non-invasive). A surgical system such as a surgical robotic system (e.g., a system that can use one or more robotic components to perform surgical procedures) may include a single computing device (e.g., a desktop computer) that controls various aspects of the surgical system and provides real-time analysis and feedback to the operator. As an example, a computing device of a surgical system may (e.g., in response to receiving operator input) receive a control signal from an input device, and may control (e.g., manipulate) one or more components based on the received control signal (e.g., in the case of a surgical robotic system, the control signal may manipulate one or more robotic arms). In addition, the computing device may provide feedback by displaying surgical information on one or more displays. For example, a computing device may be coupled to one or more surgical cameras (e.g., an endoscope), and image data may be received and displayed from the one or more surgical cameras during a surgical procedure. Specifically, the computing device may include a graphics processing unit (GPU) whose task is to receive image data and render it on a display. In addition to rendering image data, the GPU may also receive other surgical information for display along with the image data. For example, the computing device may receive surgical information from one or more other devices of the system (such as energy and intelligent surgical tools, ultrasound scanner systems, etc.). The computing device may also receive notifications based on the status of other devices (e.g., based on operating room event detection) to display to the operator. Thus, the GPU may render the received information (e.g., along with the image data) in a graphical user interface (GUI), and the received information is presented to the operator on one or more displays (e.g., as information overlaid on image data captured by one or more surgical cameras).

[0007] However, there are several disadvantages to having a centralized computer system. In the case where the computing device mixes and blends multiple streams (e.g., to overlay a GUI on one or more video streams), an unexpected failure may result in no image (or an erroneous image) being displayed to medical personnel. Because visualization is often extremely important to the execution of many medical operations, failures may adversely affect the execution of the operation, causing delays, increasing the possibility of errors, interrupting flow, reducing device utilization, and in some cases causing the operation to be terminated. Therefore, as described herein, such systems in which an integrated GPU receives video data from one or more endoscopes and receives surgical information and mixes the surgical information (e.g., as a GUI) with the video data have the following inherent risks: any failure (e.g., blocking process) on a single computer device may potentially freeze the display, and thus will disable any image / surgical information updates that would otherwise be displayed on the display of the surgical system. As an example, since resources (e.g., memory, processing power, etc.) are shared between surgical software applications being executed by a computing device during a surgical procedure, a failure or blocked process of one software application may adversely affect other software applications, including applications that are using (and sharing) the GPU. In this case, when a failure occurs at (or affects) the GPU, the image data from the endoscope will become unreliable (e.g., frozen, obscured by overlaid surgical information, etc.) during the surgical procedure. For example, a system failure (or error) may cause a GUI overlaid on top of the endoscopic video to obscure some or all of the critical endoscopic video. Therefore, a video control device is needed that controls the video stream to ensure that necessary video (e.g., critical video during an intraoperative procedure) is displayed reliably and continuously.

[0008] The present disclosure provides a video control device that uses (at least) two video controllers to control and display video streams for a surgical system. Specifically, the video device includes: a first video controller, such as a field programmable gate array (FPGA), which first video controller (e.g., directly) receives a clinically critical video stream (e.g., video captured by an endoscope); and a second video controller, such as a GPU that receives surgical data (e.g., a GUI including surgical information), which second video controller provides the surgical data as another video stream to the first video controller, wherein the first video controller displays (e.g., renders for display) the surgical data layered on top of (or superimposed on) an area of ​​the critical video stream on a display of the surgical system. In essence, the first video controller mixes the video stream from the GPU with the critical video stream from the endoscope, and passes the mixed video stream for display so that the surgical data is superimposed on a (e.g., predefined) area of ​​the endoscopic video. If a failure occurs that causes the video stream from the GPU to no longer be superimposed over the area (e.g., but instead would cause the video stream from the GPU to cover a larger area of ​​the critical endoscopic video (e.g., adversely obscuring the critical endoscopic video)), the video control device may mix the endoscopic video with the hybrid video (e.g., produce another hybrid video stream) so that the endoscopic video is superimposed over the erroneous hybrid stream. Thus, any potential problems that may occur with the GPU (and / or other resources within the surgical system, e.g., due to a blocking process) will not prohibit the clinically critical video stream from being displayed (and / or will not obscure the critical video stream). Thus, this greatly reduces (or eliminates) issues with the video stream from the endoscope freezing and / or being obscured from the operator's field of view, thereby increasing the viewing reliability of the video stream from the system's camera (e.g., when the surgical system experiences performance issues or failures (or errors)).

[0009] The present disclosure provides a surgical system for controlling and displaying video streams. Specifically, the system may include a video controller that receives a first video stream captured by a camera (e.g., an endoscope) of a surgical system and receives a second video stream that includes surgical data (e.g., surgical overlay information). The video controller displays the second video stream superimposed on a portion (e.g., an area) of the first video stream, determines that the second video stream is no longer superimposed on the area of ​​the first video stream (e.g., the second video stream may obscure a larger area of ​​the first video stream), and in response to determining that the second video stream is no longer superimposed on the area of ​​the first video stream, continues to display the first video stream. Specifically, the video controller may superimpose the first video stream on (at least a portion of) the second video stream so that the user can continue to view the key video stream from the endoscope without interruption.

[0010] In one aspect, the surgical data includes a graphical user interface (GUI) including at least one of: notifications associated with the surgical procedure, image data captured by one or more cameras of the surgical system, and user interface (UI) items for allowing a user to interact with the GUI via a (e.g., touch-sensitive) display. In another aspect, a display includes a number of rows of pixels, wherein displaying a second video stream superimposed over an area of ​​a first video stream includes: generating a hybrid video stream by blending the first video stream with the second video stream; storing selected pixels for a group of rows in the number of rows in a row buffer based on the hybrid video stream; and providing the stored selected pixels from the row buffer to the display.

[0011] In one aspect, the area is a first area, wherein when the second video stream is superimposed over the area of ​​the first video stream, the surgical system is in a hybrid mode in which a hybrid video stream of the first video stream and the second video stream is displayed, wherein the method further comprises presenting a notification providing a recommendation to a user of the surgical system to switch to a failover mode in which the first video stream is superimposed over the second area of ​​the hybrid video stream, wherein the first video stream continues to be displayed in response to receiving a user input via a user input device for switching from the hybrid mode to the failover mode.

[0012] In one aspect, determining that the second video stream is no longer superimposed includes determining that the area of ​​the first video stream on which the second video stream is displayed exceeds a threshold area. In another aspect, determining that the second video stream is no longer superimposed is based on content analysis of the second video stream.

[0013] In one aspect, the video controller is a first video controller, wherein the second video stream is received from a second video controller of the surgical system. In some aspects, the first video controller is a field programmable gate array (FPGA) and the second video controller is a graphics processing unit (GPU). In one aspect, the FPGA draws power from a first power source and the GPU draws power from a second power source of the surgical system.

[0014] In one aspect, displaying a second video stream superimposed over an area of ​​a first video stream includes providing a first mixed video stream comprising the first video stream and the second video stream to a display, wherein continuing to display the first video stream includes generating a second mixed video stream by retrieving a first group of one or more video frames of the first video stream from a frame buffer and mixing the first group of video frames with a second group of one or more video frames of the first mixed video stream; and providing the second mixed video stream to the display.

[0015] In one aspect, the second video stream and the first video stream are displayed on a display of a surgical system, the display comprising a plurality of rows of pixels (e.g., extending along the length of the display), wherein the display comprises: storing selected pixels for a group of rows of the plurality of rows in a row buffer based on a composite (or mixed) video stream comprising the first video stream and the second video stream; and providing the stored selected pixels from the row buffer to the display.

[0016] The above summary does not include an exhaustive list of all aspects of the present disclosure. It is contemplated that the present disclosure includes all systems and methods that can be implemented by all suitable combinations of the various aspects summarized above and those disclosed in the detailed description below and specifically pointed out in the claims. Such combinations may have specific advantages not specifically recited in the above summary. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Various aspects are illustrated in the figures of the accompanying drawings by way of example and not limitation, wherein like reference numerals indicate similar elements. It should be noted that references to "one" or "an" aspect of the present disclosure are not necessarily to the same aspect, and that they refer to at least one. In addition, for the sake of brevity and to reduce the total number of drawings, a given drawing may be used to illustrate features of more than one aspect, and not all elements in the drawings may be required for a given aspect.

[0018] Figure 1 A pictorial view of an example surgical system in a surgical setting is shown.

[0019] Figure 2 is a block diagram of a surgical system including a video control device according to one aspect.

[0020] Figure 3 is a flow chart of a process for controlling and displaying aspects of a video stream by a video control device of a surgical system.

[0021] Figure 4 is a flowchart of a process for determining whether to switch between a hybrid mode, in which a hybrid video stream of a second video stream is displayed superimposed on a first video stream, and a failover mode, in which a first video stream is superimposed on the hybrid video stream to continue displaying the first video stream.

[0022] Figure 5 Several stages are illustrated in which a surgical system switches from hybrid mode to failover mode. DETAILED DESCRIPTION

[0023] Several aspects of the present disclosure are now explained with reference to the accompanying drawings. Whenever the shapes, relative positions and other aspects of the parts described in a given aspect are not explicitly defined, the scope of the present disclosure is not limited to the parts shown, which are shown for illustrative purposes only. In addition, although many details are set forth, it should be understood that some aspects can be practiced without these details. In other cases, well-known circuits, structures and technologies are not shown in detail to avoid obscuring the understanding of this specification. In addition, unless the meaning is clearly contrary, all ranges listed herein are deemed to include the endpoints of each range.

[0024] Figure 1 A pictorial view of an example (e.g., laparoscopic) surgical system (which may be referred to hereinafter as a "system") 1 in a surgical setting is shown. System 1 includes a user console 2, a control tower 3, and one or more surgical robotic arms 4 at a surgical robotic table (surgical table or surgical platform) 5. In one aspect, arm 4 may be mounted to a Figure 1 The example of the operating table or bed on which the patient is located is shown. In one aspect, at least some of the arms 4 may be configured differently. For example, at least some of the arms may be mounted on a ceiling, a side wall, or another suitable structural support (such as a cart separate from the operating table). The system 1 can be combined with any number of devices, tools, or accessories for performing surgery on the patient 6. For example, the system 1 may include one or more surgical tools (instruments) 7 for performing a surgical operation (surgical procedure). The surgical tool 7 can be an end effector attached to the distal end of the surgical arm 4 for performing a surgical procedure.

[0025] Each surgical tool 7 can be manually manipulated, robotically manipulated, or both during a surgical procedure. For example, a surgical tool 7 can be a tool for entering, viewing, or manipulating the internal anatomical structure of a patient 6. In one aspect, the surgical tool 7 is a gripper that can grasp the patient's tissue. The surgical tool 7 can be manually controlled by a bedside operator 8; or it can be robotically controlled via actuation of a surgical robot arm 4 to which it is attached. For example, when manually controlled, the operator can (e.g., physically) hold a portion of the tool (e.g., a handle), and can manually control the tool by moving the handle and / or pressing one or more input controls (e.g., buttons) on the tool (e.g., the handle of the tool). On the other hand, when robotically controlled, the surgical system can manipulate the surgical tool based on user input (e.g., received via a user console 2, as described herein).

[0026] Generally, a remote operator 9 (such as a surgeon or other operator) may use the user console 2 to remotely manipulate the arm 4 and / or attached surgical tools 7, for example during a remote operation. The user console 2 may be located in the same operating room as the rest of the system 1, such as in a remote operating room. Figure 1 As shown. However, in other environments, the user console 2 may be located in an adjacent or nearby room, or it may be located in a remote location, for example, in a different building, city or country. The user console 2 may include one or more components, such as a seat 10, one or more foot controls (or foot pedals) 13, one or more (handheld) user input devices (UID) 14, and at least one display 15. The display is configured to display a view of a surgical site within a patient 6, for example. The display may be configured to display image data (e.g., static images and / or videos). In one aspect, the display may be any type of display, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, a head mounted display (HMD), etc. In some aspects, the display may be a 3D immersive display for displaying a 3D (surgery) presentation. In one aspect, the display may display a high definition (HD) 3D or stereoscopic image. For example, during a surgical procedure, one or more endoscopes (e.g., an endoscopic camera) may capture image data (e.g., a video stream) of a surgical site, and the display presents the image data to the user in 3D. In one aspect, the 3D display can be an autostereoscopic display that provides 3D perception to the user without the need for special glasses. As another example, the 3D display can be a stereoscopic display that provides 3D perception by using glasses (e.g., via active shutter or polarization).

[0027] On the other hand, the display 15 may be configured to display at least one graphical user interface (GUI) that can provide information and / or interactive content, thereby assisting the user in performing surgical procedures through one or more instruments in the surgical system 1. For example, some of the displayed content may include image data captured by one or more endoscopic cameras, as described herein. On the other hand, the GUI may include selectable UI items that can cause the system to perform one or more operations when manipulated by the user. For example, the GUI may include UI items as interactive content to switch control between robot arms. In one aspect, in order to interact with the GUI, the system may include an input device (such as a keyboard, a mouse, etc.). On the other hand, the user may use UID14 to interact with the GUI. For example, the user may manipulate the UID to navigate through the GUI (e.g., using a cursor), and make a selection, hover the cursor over the UI item and manipulate the UID (e.g., select a control or button). In some aspects, the display may be a touch-sensitive display screen. In this case, the user can perform a selection by navigating and selecting via the touch display. In some aspects, any method can be used to navigate and / or select UI items.

[0028] In some aspects, the surgical system may be able to display an enhanced view on display 15 that may include, for example, an endoscopic view enhanced with one or more additional overlays (e.g., based on a video stream captured by an endoscope). As an example, the overlays may be simple (such as a tool kit for instrument use) or complex (such as a fusion of preoperative images that may be displayed in addition to the tool kit, etc.). As another example, the overlays may include one or more GUIs superimposed on one or more areas of image data captured by one or more cameras (e.g., an endoscope). More about the enhanced view is described herein.

[0029] As shown, the teleoperator 9 sits in the chair 10 and views the user display 15 while manipulating the foot control 13 and the handheld UID 14 to remotely control one or more of the arm 4 and the surgical tool 7 (which is mounted on the distal end of the arm 4).

[0030] In some variations, the bedside operator 8 can also operate the system 1 in an "in bed" mode, where the bedside operator 8 (user) is now located on the side of the patient 6 and manipulates the robot-driven tools (the end effector attached to the arm 4) at the same time, for example, holding the handheld UID 14 and the manual laparoscopic tools with one hand. For example, the left hand of the bedside operator can manipulate the handheld UID to control the robotic components, while the right hand of the bedside operator can manipulate the manual laparoscopic tools. Therefore, in these variations, the bedside operator 8 can perform both robot-assisted minimally invasive surgery and manual laparoscopic surgery on the patient 6.

[0031] During the example procedure (surgical operation), the patient 6 is prepared for surgery and covered with a sterile drape in a sterile manner to achieve anesthesia. When the arm of the system 1 is in a stowed configuration or a withdrawn configuration, the initial access to the surgical site (to facilitate access to the surgical site) can be performed manually. Once the access is completed, the initial positioning or preparation of the system 1 (including its arm 4) can be performed. Then, the surgical operation continues, and the remote operator 9 at the user console 2 uses the foot control 13 and the UID 14 to manipulate various end effectors and possible imaging systems to perform the surgical operation. Manual assistance can also be provided at the operating bed or operating table by a bedside person (e.g., a bedside operator 8) wearing a sterile surgical gown, and the bedside person can perform tasks on one or more arms in the robot arm 4, such as retracting tissue, performing manual repositioning, and tool replacement. Non-sterile personnel may also be present to assist the remote operator 9 at the user console 2. When the procedure or surgical operation is completed, the system 1 and user console 2 may be configured or set to a state to facilitate completion of post-operative procedures, such as cleaning or disinfection, and inputting or printing healthcare records via the user console 2 .

[0032] In one aspect, the remote operator 9 holds and moves the UID 14 to provide input commands to drive (move) one or more robot arm actuators 17 (or drive mechanisms) in the robotic system 1 for remote operation. The UID 14 may be communicatively coupled to the rest of the system 1, for example, via a console computer system 16 (or host). The UID 14 may generate a spatial state signal corresponding to the movement of the UID 14, such as the position and orientation of the handheld housing of the UID, and the spatial state signal may be an input signal to control the movement of the robot arm actuator 17. The system 1 may use a control signal derived from the spatial state signal to control the proportional movement of the actuator 17. In one aspect, the console processor of the console computer system 16 receives the spatial state signal and generates a corresponding control signal. Based on these control signals that control how the actuator 17 is powered to drive the segments or connectors of the arm 4, the movement of the corresponding surgical tool attached to the arm may simulate the movement of the UID 14. Similarly, interaction between the remote operator 9 and the UID 14 may generate, for example, a grasp control signal that causes the jaws of a grasper of the surgical tool 7 to close and grasp tissue of the patient 6 .

[0033] The system 1 may include several UIDs 14, wherein respective control signals are generated for each UID controlling the actuators and surgical tools (end effectors) of the respective arms 4. For example, the teleoperator 9 may move the first UID 14 to control the movement of an actuator 17 located in the left robotic arm, wherein the actuator responds by moving a link, gear, etc. in the arm 4. Similarly, the movement of the second UID 14 by the teleoperator 9 controls the movement of another actuator 17, which in turn drives other links, gears, etc. of the system 1. The system 1 may include a right arm 4 fixed to a bed or table on the right side of a patient, and a left arm 4 located on the left side of the patient. The actuator 17 may include one or more motors that are controlled so that they drive the joint of the arm 4 to rotate, for example, to change the orientation of an endoscope or gripper of a surgical tool 7 attached to the arm relative to the patient. The movement of several actuators 17 in the same arm 4 may be controlled by a spatial state signal generated from a particular UID 14. The UID 14 may also control the movement of the respective surgical tool gripper. For example, each UID 14 may generate a corresponding grasping signal to control the movement of an actuator (eg, a linear actuator) that opens or closes the jaws of a grasper at the distal end of the surgical tool 7 to grasp tissue within the patient 6 .

[0034] In some aspects, the communication between the surgical robotic table 5 and the user console 2 can be performed through a control tower 3, which can translate user commands received from the user console 2 (and more specifically from the console computer system 16) into robotic control commands transmitted to the arm 4 on the surgical table 5. The control tower 3 can also transmit status and feedback from the surgical table 5 back to the user console 2. The communication connection between the surgical table 5, the user console 2 and the control tower 3 can be via a wired link (e.g., optical fiber) and / or a wireless link, using any suitable data communication protocol in a variety of wireless data communication protocols, such as the Bluetooth protocol. Any wired connection can optionally be built into the floor and / or wall or ceiling of the operating room. The system 1 can provide video output to one or more displays (including displays in the operating room and remote displays accessible via the Internet or other networks). The video output or feed can also be encrypted to ensure privacy, and all or part of the video output can be saved to a server or electronic health record system.

[0035] Figure 2 1 is a block diagram of a surgical system 1 including a video control device 20 according to one aspect. The system 1 includes a camera (e.g., an endoscope) 18, an input device 19, a video control device 20, a user input device 21, and a display 22 (e.g., which may be Figure 1 The device 15 described herein. In one aspect, the display can be any type of video display (e.g., a liquid crystal display (LCD), an HMD, etc.). In some aspects, the display can be composed of several rows of pixels, each of which extends along at least a portion of the length of the display. In one aspect, the system can include more elements, such as having one or more displays, one or more input devices, and / or one or more cameras. In another aspect, the system can have fewer elements, such as having no user input device.

[0036] The input device 19 may be any electronic device configured to determine (generate or produce) digital (e.g., surgical) data (surgical stack information) associated (related) with the surgical system and configured to provide the surgical data to the video control device 20. For example, the input device 19 may be configured to produce surgical data associated with a remote operation being performed by an operator (e.g., Figure 1The input device may be one or more sensors, such as a camera that captures images (within its field of view) as video (image) data, a microphone that captures ambient sounds as one or more microphone signals, a motion (e.g., proximity) sensor, or a temperature sensor. On the other hand, the input device may be a surgical device (e.g., an electronic device used during a surgical procedure) that provides surgical (e.g., patient) information, such as heart rate, etc. As another example, the input device may be a surgical tool (e.g., tool 7) and / or an electronic device that controls the tool (and / or powers the tool), wherein the surgical data may include the state of the surgical tool and / or parameters associated with the tool.

[0037] In another aspect, the input device may be a device of a surgical robotic system, such as a robotic arm 4 being manipulated by an operator, wherein the surgical data may indicate a state (e.g., position) of the robotic arm. In another aspect, the input device may be Figure 1 Any computing device of the surgical system 1, such as the control tower 3 and / or console computer system 16 of the system 1, can provide status and feedback as surgical data. For example, the surgical data may include instrument sensor data, robot information and parameters (e.g., during robotic-assisted surgery), information provided by an artificial intelligence (AI) engine (not shown) (which may be based on analysis of stored data (e.g., acquired during previous intraoperative procedures)), etc. The information provided by the AI ​​engine may include intraoperative guidance, suggestions, warnings, and / or assistance.

[0038] In one aspect, the user input device 21 can be any electronic device that is configured to receive user input and provide one or more control signals (generated by the user input device based on the user input) to another electronic device to the video control device 20. For example, the user input device can be a peripheral computer device, such as a keyboard or a mouse. On the other hand, the device can be a touch-sensitive display screen (e.g., an electronic device with a touch-sensitive display screen) that can display a GUI with one or more user interface (UI) items, wherein the device 21 can generate one or more control signals based on the user touching a portion of the display screen that is presenting the UI items. For example, the user device 21 can be a tablet computer, a laptop computer, or a smart phone.

[0039] In one aspect, the user input device may be part of the video control device 20. For example, the user input device may be (or include) one or more (e.g., physical) buttons (e.g., toggle buttons, push buttons, etc.) disposed on the video control device 20 (e.g., the housing 39). In another aspect, the user input device may be a touch screen (not shown) that is part of the video control device.

[0040] The video control device 20 is an electronic device configured to control and display one or more video streams on a display 22. As shown, the video control device is communicatively coupled to one or more other electronic devices of the surgical system to exchange digital data, such as video streams (e.g., video data). Specifically, the video device is coupled to the camera 18, the input device 19, the user input device 21, and the display 22. In one aspect, the video control device can be configured to establish a wireless connection with one or more of these devices via a wireless communication protocol (e.g., a Bluetooth protocol or any other wireless communication protocol). During the established wireless connection, the video control device can exchange (e.g., transmit and / or receive) data packets (e.g., Internet Protocol (IP) packets) with one or more of the other devices, which data packets may include video data in any video format.

[0041] On the other hand, the video control device may be communicatively coupled to one or more electronic devices via other methods. For example, the video control device may be coupled to the camera 18 via a wired connection (e.g., a high-definition multimedia interface (HDMI) connection, etc.) to receive image data captured by the camera (as a video stream), and may be coupled to the display 22 via another wired connection (e.g., another HDMI connection) to provide one or more video streams to the display to be displayed. In one aspect, the video controller may include one or more input connections (e.g., for coupling to the camera 18, to the input device 19, and to the user input device 21) and one or more output connectors (e.g., the display 22) for (e.g., removably) coupling to one or more electronic devices. For example, the video control device may include one or more video input connectors and video output connectors (e.g., an HDMI connector, a digital video interface (DVI), a serial digital interface (SDI) connector, a composite video connector) for coupling to a camera and to a display, respectively.

[0042] The video control device 20 includes a housing 39, a first video controller 23, a second video controller 24, a first power source 38, and a second power source 37. The first power source 38 is configured to provide power to the first video controller 23, and the second power source 37 is configured to provide power to the second video controller 24. In one aspect, both power sources are configured to draw power from a power source (e.g., an external power source, such as AC mains, one or more internal batteries, etc.), and each of the video controllers is configured to draw power from their respective power sources (which can be configured to regulate the input power drawn from the external power source for use by their respective controllers). In one aspect, the first power source 38 can be arranged to provide power only to the first video controller, while the second power source 37 can be arranged to supply power to the second video controller and one or more other components of the video control device (such as a motherboard (not shown) to which the second video controller can be coupled). By coupling only, the first video controller to the first power source 38 ensures that the controller 23 is always provided with sufficient power to perform one or more video processing operations, as described herein. In another aspect, the two controllers may be coupled to two power sources, with only one power source providing power at any given time. Thus, the other power source may be a redundant power source (e.g., used in situations where the other power source is inoperable). In another aspect, the video control device may include only one power source (e.g., only the first power source 38) that is arranged to provide power to each (or at least some) of the components of the control device.

[0043] In one aspect, each of the components of the video control device 20 may be disposed (e.g., housed) within the housing 39. In this case, the video control device may be designed so that it can be integrated into an existing surgical system. For example, in an existing surgical system, a display may be (e.g., directly) coupled to a computing device that is coupled to one or more cameras. In this case, the computing device receives video from the camera and provides (e.g., using an onboard GPU) image data (which may include surgical information layered on the image data) to the display. In one aspect, the video control device may be designed to be disposed (e.g., removably coupled) between 1) one or more cameras and / or computing devices (e.g., input device 19) and 2) one or more displays, as shown herein, so as to control the video stream provided to the display 22. On the other hand, the video control device may be designed to be removably received in another computing device (such as a control tower 3). More content about a video control device for controlling a video stream is described herein.

[0044] As shown herein, the video control device includes two video controllers 23 and 24. In one aspect, any of these controllers can be a dedicated processor, such as an application specific integrated circuit (ASIC), a general purpose microprocessor, a field programmable gate array (FPGA), a digital signal controller, or a set of hardware logic structures (e.g., filters, arithmetic logic units, and dedicated state machines). Specifically, the first video controller 23 can be an FPGA and the second video controller 24 can be a GPU. In one aspect, each of these controllers can be coupled to the same printed circuit board (PCB), such as a motherboard (not shown) of the video control device, for example, via a physical interface (such as a peripheral component interconnect express (PCIe)). On the other hand, each of these controllers can be coupled to a separate PCB (or a portion thereof) disposed in a housing 39 of the video control device. In some aspects, the first video controller can be an FPGA including onboard software (e.g., firmware) stored in a memory (e.g., memory 27) of the FPGA, which performs one or more operations when used to program (configure) the hardware elements (e.g., configurable logic blocks (CLBs)) of the FPGA to control one or more video streams being received and display the one or more video streams, as described herein. In one aspect, the FPGA can be a stand-alone hardware-based solution (e.g., relative to other components of the surgical system, such as the second video controller 24 of the video device 20), such that operations performed by the FPGA use only resources (e.g., logic gates) that are part of the FPGA, and the FPGA may not share resources with other electronic components of the surgical system (and / or may not perform operations other than those preprogrammed into the FPGA). In this case, other operations performed by the surgical system may not be performed by the FPGA. Thus, the FPGA may be a more reliable solution to ensure that critical video streams (e.g., video streams received from the endoscope 18) are less sensitive to blocking processes that may occur due to other operations performed by other components of the surgical system (e.g., the second video controller 24).

[0045] The second video controller 24 is configured to receive surgical data from one or more input devices 19, and is configured to generate a (e.g., a second) video stream including surgical data (e.g., generate one or more video streams, each of which has at least some of the surgical data received). In one aspect, the second video controller may receive data and generate a graphical user interface (GUI) including at least some of the surgical data (e.g., as one or more video frames). For example, the second video controller may aggregate and arrange at least some of the surgical data as a GUI (e.g., part of a GUI). In one aspect, the second video controller may arrange the data based on a predefined layout (or may be a user-defined layout). For example, the surgical data may include one or more notifications associated with a surgical procedure being performed by one or more operators of the surgical system. Such notifications may indicate the status of a patient, a status update of one or more electronic devices (e.g., a surgical tool of a robotic arm of a surgical system), etc., and / or information related to the surgical procedure. On the other hand, the GUI may include one or more user interface (UI) items for interacting with the GUI through a display (e.g., by touch when the display is a touch-sensitive display, and / or by one or more peripheral devices coupled to the video control device, such as a keyboard). For example, UI items can control what the GUI displays and / or can control the surgical system. For example, a surgical system (e.g., one or more processors of a surgical system) can execute one or more software applications. In this case, the GUI may include UI items associated with those programs, which can cause these programs to perform one or more operations when the user interacts. On the other hand, the second video controller can receive one or more video streams from an input device. For example, the input device 19 may include one or more cameras. In this case, the GUI may include captured image data of the camera. On the other hand, the second video controller 24 can retrieve data from a memory (e.g., of a video control device) for rendering into one or more video streams.

[0046] Using the surgical data, the second controller can create a layout of a GUI that includes the data, which the second controller transmits as a video stream (as one or more video frames, as described herein) to the first video controller 23. In some aspects, the second video controller can be configured to pass one or more video streams from one or more input devices to the first video controller 23. As described herein, the first video controller can receive the video streams from the second video controller and can generate a mixed (or composite) video stream that includes the video stream from the second video controller superimposed on the image data captured by one or more cameras (e.g., camera 18) for display on one or more displays, so that an operator of the surgical system can view the surgical data simultaneously with the key video content captured by the one or more cameras 18. More content about generating and displaying a composite video stream is described herein.

[0047] The first controller 23 includes several operating blocks, such as a system monitor 25, a memory 27, a mixing block 70, and a failover mixing block 72. Examples of memory (e.g., a non-transitory machine-readable storage medium) may include read-only memory, random access memory, CD-ROM, DVD, tape, optical data storage device, flash memory device, and phase change memory. As described herein, the first video controller 23 can operate in one of one or more operating modes for mixing and displaying video content from one or more video sources. For example, the first video controller can operate in a mixed (video) mode in which the video (e.g., frame) received from the second video controller is mixed with the video captured by the camera 18 to produce a mixed (composite) video stream that, when displayed on the display 22, provides the operator with a stacked view of surgical data on top of the video captured by the camera 18.

[0048] The first video controller 23 is configured to receive a (e.g., first) video stream (or one or more video streams) captured by (e.g., from) a camera 18 (of the surgical system), and is configured to receive another (e.g., second) video stream (or one or more video streams) including at least some surgical data from the input device 19 from the second video controller 24. In one aspect, any video codec may be used to encode the received video stream. For example, the first video stream may be encoded by the camera 18 using an H.264 codec, wherein the first video controller 23 may be configured to decode the video stream into (e.g., uncompressed) video data in any video format (e.g., in an RGBα signal format). In one aspect, the second video stream may have the same video format as the first video stream. In some aspects, the video stream received from the camera 18 and / or the second video controller 24 may be a high-definition (HD) video, which may include 10-bit 4K video, such as, for example, a resolution of 3840×2160 pixels (also referred to as 2160p), 1920×1080 pixels (also referred to as 1080p video), and 1280×720 pixels (also referred to as 720p video), with a frame rate of 59.94 image frames per second and / or 60 image frames per second (fps). On the other hand, the second video stream may be in an uncompressed video format (e.g., the video format in which the first video controller 23 decodes the first video stream from the camera 18).

[0049] A description of the first video controller operating in a mixed mode will now be described. In one aspect, the mixing block 70 is configured to receive a (e.g., first) video stream from a camera 18 (which includes, for example, video content of a surgical site during a surgical procedure) and a (e.g., second) video stream including surgical data, and to generate a mixed video stream by mixing the first video stream with the second video stream. In one aspect, the second video stream generated by the second video controller can be a graphics frame (e.g., it can be an uncompressed video stream) that allows the mixing block 70 to generate a mixed video stream of several video streams. For example, the video stream of the second video controller 24 can be an RGBα stream, where "RGB" represents the three color channels (red, green, and blue) of each pixel of the graphics frame of the stream, and "α" is an alpha (α) channel representing the degree of transparency associated with each pixel of the frame. In one aspect, the α channel can be separate or pre-multiplied. In one aspect, the α channel allows the mixing block to α-mix one video stream (e.g., part of one video stream) to another video stream. Specifically, the blending block uses the alpha channel for individual pixels to overlay a portion of the surgical data of the second video stream as an overlay stream over (or on top of) another video stream (e.g., the first video stream of the camera 18) by causing the channel to have a value between 0.0 for full transparency and 1.0 for full opacity. Specifically, the blending block is configured to receive a first video stream from the camera 18 and a second video stream from the second video controller 24, and is configured to blend the second video stream (e.g., at least a portion of the second video stream) with the first video stream. For example, the blending block may perform alpha blending, wherein the blending block generates a blended video stream c by blending two streams (e.g., video frames of the two streams) together. Specifically, the blended video stream may be as follows

[0050] c=αf+(1-α)b

[0051] Where b represents a first video stream in the background (e.g., one or more video frames of the first video stream), and f represents a second video stream in the foreground (e.g., one or more video frames of the second video stream). In one aspect, the alpha blending equation mentioned above assumes that the gamma γ is 1 for the color space of each RBG color channel in the video stream. Therefore, in the case of alpha blending, the blending block can generate a blended video stream by combining pixels of the second video stream (based on the alpha value) with corresponding pixels of the first video stream so as to (e.g., on a per-pixel basis) 1) (at least partially) bring the second video stream to the foreground or 2) only display the first video stream. For example, the blending block can alpha blend a first video frame of the first video stream with a (e.g., corresponding) second video frame of the second video stream such that at least a portion of the second video frame is superimposed over an area of ​​the first video frame. For example, pixels of the second video frame having an alpha value of 0.0 are transparent, which means that when the blended video is displayed, the pixel values ​​of the first video frame are shown. However, if the alpha value is greater than 0.0, such as 1.0, the pixel value of the second video frame will be shown because the pixel is completely opaque. Therefore, in order to overlay the GUI within the video frame of the second video stream on the video frame of the first video stream, each pixel associated with the GUI may include an alpha value greater than 0.0, while the portion of the video frame without the GUI may have an alpha value of 0.0. In some aspects, the second video controller 24 may be configured to define the alpha value of the second video stream.

[0052] As described herein, the first video controller may be configured to "pass through" the mixed video stream to the display 22. Specifically, the first video controller 23 (e.g., an operating block of the first video controller) may perform very few (or even no) digital signal processing operations on the received (e.g., first and / or second) video streams and / or the mixed video stream, so that the first video controller acts like a cable (e.g., a circuit trace) from the camera 18 to the display 22 (e.g., electronically connecting the camera to the display). In one aspect, the first video controller may have very little latency (e.g., compared to conventional video controllers (such as GPUs)) to pass through the video streams, where latency may be defined as the added delay (or amount of time) between the first moment that the camera 18 captures a video frame and the second (subsequent) moment that the display 22 outputs the frame. In one aspect, to reduce latency, the first video controller may be configured to not store the mixed video stream for an extended period of time (e.g., no longer than the time required to output the video stream within a threshold amount of time from when the stream was received from the camera 18).

[0053] To reduce latency, the first video controller 23 buffers the hybrid video stream generated by the blending block 70 on a row-by-row basis in the row buffer 28 rather than on a frame-by-frame basis in the frame buffer as would otherwise be used by a conventional GPU. For example, considering the frame rate of the hybrid video stream (which may be the same for both the first video stream and the second video stream), e.g., 60 frames per second (FPS), buffering of the entire video frame (e.g., buffering into the memory 27 of the first video controller 23) will add a minimum latency L from input (e.g., input into the controller 23) to output (e.g., output from the controller 23). FB_min , the minimum waiting time is

[0054]

[0055] In one aspect, the first video controller 23 can use the line buffer 28 instead of the frame buffer to write / read the video stream so as to significantly reduce the waiting time caused by having to write and read the entire video frame from the memory in other ways. Specifically, the mixing block 70 provides (e.g., mixes) the video stream (e.g., at least a portion of the video frame of the video stream) to the line buffer 28. Specifically, the first video controller 23 writes one or more rows of the video stream into the line buffer at any given time (e.g., each row includes one or more pixel values ​​of one or more pixels of the display 22 constituting the row). For example, in order to display the video on the display 22, the first video controller 23 can store selected pixels (e.g., pixel values, such as RGB values) of multiple rows in the total number of pixel rows for the display in the line buffer 28 based on the mixed video stream. In one aspect, the number of rows written to the line buffer may be less than the number of rows that would otherwise constitute the entire video frame of the mixed video stream that can be rendered by the display 22. Once multiple rows are written, the first video controller 23 reads the stored rows and provides these rows to the display 22 for display. In one aspect, the hybrid video stream may be scaled down and placed into a row buffer of an appropriate size (e.g., based on the hybrid video stream and / or the display). Specifically, the first video controller may provide the selected pixels stored in the row buffer 28 to the display 22 for display (e.g., rendering). The first video controller may repeat these operations to fill up the pixel count of the display 22 with one frame, and then begin adding new frames to the display 22 (e.g., row by row or row by row group (e.g., two or more rows), as described herein).

[0056] As described herein, the use of line buffer 28 can significantly reduce latency to allow the first video controller to pass the mixed video stream from the video source to the display 22. For example, considering a frame rate of 60fps and a resolution of 1080p (e.g., 1920 horizontal columns x 1080 vertical rows), buffering, for example, eight rows of a video frame will add a minimum latency of L from input to output. LB_min , the minimum waiting time is

[0057]

[0058] In one aspect, the process of blending the second video stream α into the first video stream adds little (or no) latency for processing the video stream by the first video controller. In some aspects, once the blending block produces a blended video frame, it can provide (at least a portion of) the blended frame to the line buffer 28 for display. In some aspects, the first video controller 23 can perform one or more other video signal processing operations on the blended video stream.

[0059] In one aspect, the second video stream can be superimposed over a predefined area (e.g., of pixels) that would otherwise display a portion of the first video stream. In one aspect, the area over which the second video stream is displayed can be reconfigurable (e.g., user configurable). For example, the surgical system can be configured to receive user input indicating how large (or small) the area that includes the second video stream is.

[0060] As described herein, the first video controller 23 may be configured to operate in a hybrid mode in which video streams from one or more cameras (e.g., key video captured by the endoscope 18) are mixed with one or more video streams generated by the second video controller. On the other hand, the first video controller may be configured to operate in a "failover" mode in which, based on a determination that the surgical system includes an error (e.g., based on content analysis of the hybrid video stream, the second video stream generated by the second video controller, etc.), the first video controller may be configured to mix the first video stream of the camera 18 (e.g., one or more video frames of the first video stream) with the (e.g., first) hybrid video stream generated by the mixing block 70 to produce another (e.g., second) hybrid video stream for output to a display. Thus, the video control device 20 may continue to pass the first video stream even when a system error may adversely affect the mixing being performed by the mixing block 70. More content about these system errors that adversely affect the hybrid video streams is described herein.

[0061] In one aspect, the system monitor 25 is configured to determine in which mode the first video controller will operate. Specifically, the system monitor determines whether the second video stream is no longer superimposed over an area of ​​the first video stream. Specifically, the system monitor may determine whether the mixed stream does not include appropriate video content of the second video stream (e.g., a solid area with the same (or similar) pixel values) and / or may determine that the area where the second video stream is superimposed has grown above a threshold (thereby not allowing the operator to view the video of the camera 18). In one aspect, the system monitor may make these determinations based on at least some of the received data. For example, the system monitor 25 is configured to receive data that the monitor may use to determine whether the video control device continues to operate in one of these modes or switches between modes (e.g., when the video control device is displaying video on the display 22). For example, when operating in a mixed mode, the monitor 25 may use (at least a portion of) the received data to determine whether to switch to a failover mode and, therefore, continue to display the first video stream (e.g., by alpha-mixing one or more video frames of the first video stream with one or more video frames of the mixed video stream).

[0062] In one aspect, the system monitor 25 can be configured to receive user input (data) from the user input device 21 and determine in which mode the surgical system 1 will operate based on the user input. For example, the user input device can be one or more physical buttons, each button indicating a particular mode in which the system will operate. In another aspect, the input device can be a button with one or more levels, each of which is user-configurable (e.g., by pressing a switch), and each level indicates a mode in which the system will operate. Once the user input device receives the user input, the device can transmit a control signal to the system monitor 25 that can indicate a particular mode.

[0063] On the other hand, the system monitor 25 may be configured to receive system data that indicates the state of the surgical system. For example, the system data may indicate whether the electronic device (e.g., input device 19) of the surgical system 1 has a blocked process, or whether one or more operations being performed by the surgical system have failed to be performed, which may adversely affect the surgical data being received by the video control device 20 (e.g., frozen data). On the other hand, the system data may indicate the allocation of computing resources (e.g., hardware, such as memory and one or more processors) being used by the surgical system to perform computing operations. More about system data is described herein. In some aspects, the system monitor 25 may be configured to determine whether to switch between modes based on image analysis of the second video stream being received from the second video controller 24. Specifically, the system monitor determines whether there is a problem with the second video stream (e.g., whether the image within the stream has been frozen, which indicates that the surgical system has one or more blocked processes), and based on this, it can be determined whether to switch modes (e.g., from hybrid mode to failover mode). On the other hand, content analysis may be performed on the hybrid video stream generated by the hybrid block 70, as described herein. More about performing content analysis is described herein.

[0064] Based on the received data, the system monitor 25 determines whether to switch modes, and in response, may generate a control signal that is sent to the failover mixing block 72. The operation of switching from the mixed mode to the failover mode will now be described. In response to receiving the control signal, the failover mixing block 72 may be configured to cause the video control device to operate in the mode indicated by the control signal. For example, in order to operate in the mixed mode, the failover mixing block may receive a control signal so that it does not perform any mixing operations, and thus the first video controller extracts the mixed video stream (e.g., the lines of the mixed video stream) generated by the mixing block 70 to display the video of the camera 18 and / or the input device 19. When it is determined that the first video controller 23 will switch to the failover mode, the system monitor 25 transmits a control signal to the failover mixing block 72, which generates a failover mixed video stream for display by the display 22 (e.g., replacing the mixed video stream generated by the mixing block 70). For example, the frame buffer 71 may be configured to receive one or more video frames of the first video stream (e.g., capable of holding several video frames for a period of time). Upon receiving the control signal, the failover blending block 72 may be configured to generate a failover (e.g., second) hybrid video stream by retrieving one or more video frames (of one or more video streams received from one or more cameras 18) from the frame buffer 71 and blending at least one of the retrieved video frames with one or more video frames of the (e.g., first) hybrid video stream generated by the blending block 70. Thus, the failover hybrid video stream generated by the failover blending block 72 may include (at least) three layers of video content: a first (background) layer including the first video stream from the camera 18, a second (middle) layer that may include video content of the second video stream from the second video controller 24 (and / or may include erroneous video content, as described herein), and 3) a third (top) layer including the first video stream from the camera 18. In one aspect, the first video stream from the frame buffer 71 may be superimposed over an area (less than the total area) of the first hybrid video stream from the blending block 70. The first video controller may be configured to provide the failover hybrid video stream to the display 22 in place of the first hybrid video stream (e.g., from the line buffer 28).

[0065] In one aspect, the frame buffer 71 may be configured to continuously receive video frames of the first video stream when the video control device operates in the hybrid mode. This may allow the video content device to seamlessly switch between the hybrid mode and the failover mode. For example, when an error is detected within the hybrid video stream, the failover hybrid block 72 may extract the video frames of the first video stream from the frame buffer at the time the error is detected. Thus, the video control device may automatically (e.g., without user intervention) and transparently switch between the two modes.

[0066] In one aspect, the hybrid mode can be the default mode of the video control device 20. For example, once the surgical system is activated (turned on), and the first video controller begins receiving the first video stream and the second video stream, the video control device 20 can (e.g., immediately) begin displaying the hybrid video stream.

[0067] As presently described, the video control device 20 can be configured to switch from the hybrid mode to the failover mode based on the system monitor 25 monitoring the data of the surgical system. In another aspect, the video control device can be configured to switch from the failover mode back to the hybrid mode based on one or more criteria, as described herein. For example, upon determining that the system no longer includes a blocking process (e.g., based on system data received by the system monitor), the video control device can begin displaying the hybrid video stream from the line buffer 28. In this case, the system monitor can transmit a control signal to the failover hybrid block 72 to stop hybridization and output the hybrid video stream.

[0068] Figure 3 and Figure 4 Flowcharts of processes 30 and 40, respectively, for performing one or more operations to control and display one or more video streams. In one aspect, these processes can be performed by one or more devices of the surgical system 1. For example, at least some of these operations can be performed by the video control device 20 of the surgical system 1 (e.g., by the first video controller 23 and the second video controller 24, respectively, of the video control device).

[0069] about Figure 3 , which shows a flow chart of a process 30 for controlling and displaying aspects of a video stream by a video control device 20 of a surgical system 1. Process 30 begins with the video control device 20 (e.g., a first video controller 23 of the video control device) receiving a first video stream captured by a camera such as camera 18 (at box 31). For example, once the camera 18 is activated and / or the video control device is activated (e.g., the power source receives power from an external power source), the first video controller may begin receiving the first video stream. On the other hand, the video control device may be activated in response to receiving a user input (e.g., a user pressing a power button of the video control device 20). The first video controller 23 may (e.g., optionally) display the first video stream (at box 32). For example, as described herein, the video control device transmits the first video stream (e.g., stores a portion of the stream in a line buffer (or one or more line buffers) for display on the display 22).

[0070] The first video controller 20 receives a second video stream including surgical data (at box 33). For example, the second video controller 24 may receive surgical data (e.g., as text, image data, and / or video data), and may generate a second video stream including the data, and provide the stream to the first video controller. The first video controller displays the second video stream (at box 34) superimposed on an area of ​​the first video stream. For example, the mixing block 70 may generate a mixed video stream (e.g., c) that includes (e.g., alpha-mixed) a second video stream and a first video stream, as described herein. In this case, the displayed mixed video shows (at least a portion of) the second video stream in an area within the first video stream (e.g., where the alpha value of the pixel associated with the second video stream in the area is greater than 0.0). In one aspect, the mixed video may show the second video stream in one or more areas within the first video stream. For example, when the second video stream includes a GUI having one or more UI items and a separate video stream (captured by a camera other than the camera 18), the mixed video stream may include an area containing the GUI and another area containing the separate video stream. In one aspect, the first video controller may use a line buffer 28 to display the mixed video stream. For example, the first video controller may store selected pixel values ​​for one or more pixel rows for display on the display 22 in a row buffer (or one or more row buffers) 28 (in the memory 27 of the first video controller) based on the mixed video stream, and may provide the stored selected pixels from the row buffer to the display 22.

[0071] The first video controller 23 determines that the second video stream is no longer superimposed on the area of ​​the first video stream (at box 35). Specifically, the system monitor 25 is configured to determine that the video control device will switch from the hybrid mode to the failover mode based on one or more criteria. For example, the system monitor 25 may determine that the second video stream will no longer be superimposed based on the content analysis of the second video stream (and / or the composite video stream). For example, the second video stream may no longer be superimposed by the hybrid video stream, which has another video content displayed instead of the video content of the second video stream (for example, a black screen is displayed in the area opposite to the surgical data). More content about determining whether the video control device will switch between modes is described herein. In response to determining that the second video stream is no longer superimposed, the video control device continues to display the first video stream (for example, in failover mode) (at box 36). For example, the failover hybrid block 72 can be activated to generate a second hybrid video stream from the first (original) hybrid video stream generated by the hybrid block 70, the second hybrid video stream including the first video stream (or a reduced version of the stream) superimposed on the first hybrid stream.

[0072] As currently described, the first video controller may receive a first video stream from the camera 18 and may be configured to mix the first video stream with another (second) video stream generated by the second video controller 24 to produce a mixed video stream. In one aspect, the first video stream may include one or more 3D video streams. In this case, the camera 18 may be a 3D or stereo camera associated with an endoscope, wherein the controller receives a left video channel and a right video channel each at 60fps. In this case, the first video controller may perform at least some of the operations described herein to control and display the 3D video stream, such as alpha blending both the left video stream and the right video stream with the video stream received from the second video controller 24. In some aspects, the second video controller may generate a video stream to be mixed with the 3D video stream captured by the camera 18. On the other hand, the second video controller may generate separate video streams (e.g., a left surgical data video stream including surgical data and a right surgical data video stream including surgical data, which may be the same or different from the surgical data in the left video stream). Thus, the video control device may mix the left video streams with each other and mix the right video streams with each other, thereby generating a left-mixed video stream and a right-mixed video stream for display on a display (or multiple displays).

[0073] In one aspect, the first video controller 23 may generate one or more failover hybrid video streams for one or more 3D video streams, as described herein. In the case where the video stream from the camera has one or more 3D video streams, a left channel and a right channel, the memory 27 may include one or more frame buffers for each channel. In this case, after switching to failover mode, the failover hybrid block 72 may mix one or more video frames of the left channel retrieved from the frame buffer with the left hybrid video stream, as described herein.

[0074] Steering Figure 4 , which shows a flowchart of process 40 for determining whether to switch between a hybrid mode in which a first hybrid video stream of a second video stream is displayed superimposed on a first video stream and a failover mode in which a second (e.g., failover) hybrid video stream of the first video stream is displayed superimposed on the first hybrid video stream. Thus, at least some of the operations of the process may be performed while the video control device is in operation and is displaying at least one video stream from one or more cameras and at least one video stream from a second video controller in failover mode. In one aspect, at least some of the operations of process 40 may be performed by a system monitor 25 of a first video controller 23. Specifically, at least some of the operations of process 40 may be performed in a Figure 335 in the process 40. The process 40 begins when the system monitor 25 receives system data of the surgical system (at box 41). For example, the system data may indicate the state of the (e.g., overall) system, or the state of one or more devices (or components of the device). Specifically, the system data may indicate whether the surgical system (e.g., the input device 19 of the surgical system) has a blocked process. The system monitor performs (e.g., video) content analysis of the second video stream and / or the mixed video stream generated by the mixing block 70 (at box 42). Specifically, the monitor 25 may analyze the second video stream (e.g., one or more video frames of the second video stream) to determine one or more video characteristics of the stream. For example, the monitor may execute an object recognition algorithm to detect objects contained therein. In one aspect, the recognition algorithm can be used to determine whether the stream has been frozen (e.g., where the video stream should have movement (e.g., movement by one or more objects, notification updates should be displayed, UI items should be changed, etc.) in addition to the video frame indicating that the object has not moved). For example, an object recognition algorithm may determine that a video stream is frozen based on an object suddenly stopping movement in a position and / or orientation where it would not normally stop (e.g., a bouncing ball stops in mid-air). On the other hand, a monitor may determine video characteristics, such as one or more pixel values ​​of video content contained therein. On the other hand, a monitor may perform any type of video content analysis to analyze the video to detect and / or determine object and / or video characteristics of the stream (e.g., whether the video is in focus, whether portions of the video are missing, etc.). On the other hand, a system monitor 25 may perform this analysis on a mixed video stream. In some aspects, a monitor may analyze a portion of the mixed video stream associated with a second video stream.

[0075] The system monitor determines whether the video control device should switch to a failover mode based on the content analysis and / or the system data (at decision block 43). For example, the system monitor may determine that the video control device is to operate in a failover mode in response to determining, based on the system data, that the surgical system has a blocked process. In one aspect, the system data may indicate that the system has a blocked process based on a (e.g., overall) state of the system indicated by the data. In another aspect, the system monitor may determine that the system has a blocked process (and / or is not operating properly and / or efficiently) based on historical trends in the system data.

[0076] On the other hand, the system monitor may determine the switching mode based on whether content analysis is performed on the second (and / or mixed) video stream. For example, the monitor determines whether the second video stream has been frozen based on whether an object (e.g., video, image, UI item, etc.) within the video stream has not moved within a period of time. Specifically, the system monitor may determine that the video stream is frozen based on the same number of video frames received from the second video controller 24 within a period of time. On the other hand, the system monitor may determine the switching mode in response to determining that the second video stream is not showing (not including) at least a portion of the surgical data that should have been shown. For example, in the event of a (e.g., partial) system failure, the second video stream generated by the second controller 24 may stop including surgical data (or surgical data that would have been included during normal operation) and may (at least partially) be a black (or blank) screen (e.g., a monochrome screen in which all (or most) pixel values ​​are the same (e.g., RGB) values). Thus, the system monitor may perform a content analysis to determine whether at least some pixel values ​​(e.g., a number of pixels above a threshold) will display a color (e.g., black), thereby indicating that the second video stream includes a blank screen, which would contrast with a normal second video stream that would show surgical information (e.g., text, images, etc.) and / or surgical video.

[0077] If it is determined that the video control device is to switch to a failover mode (e.g., the surgical data is based on not including (e.g., above a threshold amount) the second video stream and / or based on the second video stream blocking (e.g., greater than a threshold area) the first video stream, etc.), the first video controller presents a notification (e.g., on the display 22) that provides a recommendation to the user (e.g., the operator of the surgical system 1) to switch from the hybrid mode to the failover mode in which the first video stream (from one or more cameras 18) is superimposed over an area of ​​the hybrid video stream (at box 44). Specifically, the first video stream may be superimposed over an area of ​​the hybrid video stream that is different from the area of ​​the first video stream, and the second video stream is superimposed in the original hybrid stream over the area of ​​the first video stream. In one aspect, the notification may be a pop-up notification that may include text content, image data, and / or video data. In another aspect, the notification may be an audible notification. In this case, the video control device 20 may retrieve (e.g., from a memory) an audio signal containing the audible notification (e.g., voice), and may use the signal to drive one or more speakers of the surgical system 1 (e.g., of the video control device of the surgical system). On the other hand, the video control device may present the notification via other methods such as email, SMS, etc.

[0078] The monitor 25 receives user input (e.g., via the user input device 21) to switch to the failover mode (at box 45). For example, the user input device can be a physical switch that, when pressed by the user, sends a control signal to the system monitor indicating that the user desires a mode switch of the video control device (from a current mode to another mode). In this case, the system monitor can switch from the hybrid mode to the failover mode (at box 46). Specifically, the system monitor can generate a control signal and provide the signal to the failover hybrid block 72, which can begin to generate a failover hybrid video stream using the (original) hybrid video stream generated by the hybrid block 70 and the first video stream for display, as described herein. Thus, by displaying the failover hybrid video stream, the surgical system is able to continue (e.g., seamlessly) displaying the video stream of the camera 18, as described herein.

[0079] Returning to decision block 43, the system monitor determines (e.g., in response to determining based on system data and / or content analysis that the video control device is not switching) an area of ​​the portion of the first video stream (e.g., relative to the total area in which the first video stream is displayed) over which the second video stream is displayed (at block 47). Specifically, the system monitor 25 may perform content analysis on the hybrid video stream to determine whether to switch between modes. For example, an error in the hybrid video stream will result in a non-display or obstruction of the first (e.g., live) video stream captured by the camera 18, and therefore, the monitor may analyze the composite video for an indication that the hybrid video has an error or is obscuring a view of the first video stream. In one aspect, the error in the hybrid video stream may be due to the video stream from the second controller not (erroneously) including an alpha channel and / or alpha values, thereby erroneously causing more of the second video stream to be displayed than desired. In one aspect, the area over which the second video stream is to be displayed is determined based on one or more video frames of the second video stream. For example, the system monitor may count the number of lines of the video frame of the second video stream that are solid lines (eg, having alpha values ​​that make the second video (partially or completely) opaque over one or more areas of the first video stream).

[0080] The system monitor 25 determines whether the area exceeds a threshold area (at decision box 48). For example, the system monitor may determine whether the number of solid lines is greater than a certain fraction "x" of the lines of the corresponding frame of the first video stream. Specifically, the system monitor determines whether the second video stream will cover a larger portion of the first video stream than it should cover (e.g., the monitor determines whether the percentage of the area of ​​the first video stream covered by the second video stream is greater than a threshold percentage). In one aspect, due to computational issues within the surgical system, the area on which the second video stream indicates that it should be displayed may increase. In some aspects, the system monitor compares the area of ​​pixels having an alpha value greater than the value of the second video stream (e.g., 1) to the threshold area. If the area is larger, the system monitor proceeds to box 44.

[0081] Otherwise, the system monitor 25 determines whether a user input to switch modes has been received (at box 49). Specifically, even if the second video stream is fully overlaid and displayed on top of the first video stream, the system can determine whether the operator wishes to switch to the failover mode (e.g., based on receiving a control signal from the user input device 21). In one aspect, the video control device can receive such user input when the operator only wants to focus on the image being displayed in the first video stream. If so, the system monitor switches to the failover mode (at box 46). Otherwise, the first video stream and the second video stream continue to be displayed in a mixed mode (at box 50).

[0082] Some aspects may perform changes to processes 30 and 40 described herein. For example, specific operations of at least some processes may not be performed in the exact order shown and described. Specific operations may not be performed in a series of continuous operations, and different specific operations may be performed in different aspects. For example, the operations within the dashed box may be optional operations that may not be performed when the corresponding process is performed. As an example, in process 40, frames 44 and 45 are optional, so that the first video controller 23 may omit those operations (either one or two operations). For example, once it is determined that the area exceeds the threshold area, the first video controller may automatically (e.g., without user intervention) switch to failover mode.

[0083] As described so far, process 40 describes operations for switching the video control device 20 from hybrid mode to failover mode. In another aspect, the video control device 20 may switch (e.g., switch back) from failover mode to hybrid mode. For example, the system monitor 25 may continuously monitor data to determine whether the second video stream from the second video controller is suitable for display on the display 22. As an example, upon determining that the pixel values ​​of the second (e.g., hybrid) video stream indicate that the stream no longer displays a blank screen (e.g., a monochrome screen), but instead includes various pixel values ​​(e.g., indicating that the stream includes objects, such as UI items of a GUI), the system monitor may direct the first video controller 23 (e.g., the failover hybrid block 72 of the first video controller) to switch back to hybrid mode. Thus, the system monitor may perform at least some of the operations described herein to determine whether to switch between modes.

[0084] As described so far, the first video stream can be a video stream (or one or more combined video streams) displayed in failover mode and hybrid mode, which video stream is received from one or more cameras 18 (e.g., endoscope, laparoscope, colonoscope, bronchoscope, etc.). In one aspect, the first video stream can be a "critical" video stream that is a live video stream (e.g., displayed within a time threshold after it is captured by the camera 18) and / or is always displayed on the display during a surgical (e.g., intraoperative) procedure. In another aspect, the first video stream may include images and / or video data from other image / video sources.

[0085] As currently described, the video control device may receive user input to switch between the hybrid mode and the failover mode (e.g., an operator changes a switch). In another aspect, the video control device may be configured to switch between one or more video sources. For example, the first video controller may receive one or more video streams from one or more cameras 18. As an example, the system may include two endoscopes, wherein the video control device is coupled to the two endoscopes. In one aspect, the user input device 21 may be configured to allow the operator to switch between endoscopes. For example, when active, the video control device may display a video stream from the first endoscope (e.g., as a first video stream, as described herein). Upon receiving user input, the video control device may switch to receiving video from the second endoscope. In another aspect, the first video stream may be a mixed video stream of one or more (e.g., key) video streams. In this case, the key video stream may be a mixed video stream, wherein one area of ​​the mixed stream includes video from one endoscope, and another area of ​​the mixed stream includes video from another endoscope.

[0086] Figure 5Several stages 60 to 62 are illustrated in which a surgical system switches from a hybrid mode to a failover mode. The first stage 60 shows the video content of a hybrid video stream of the surgical system 1 when operating in the hybrid mode. Specifically, the stage is showing a second video stream 63 superimposed over an area 65 of a first video stream 64, the second video stream including a GUI with surgical data, both of which are displayed on the display 22. The second stage 61 shows the video content of the system 1 when operating in the hybrid mode in which the surgical system includes an error (e.g., a blocked process). Specifically, due to the failure, the second video stream generated by the second video controller (which may be adversely affected by the error) now covers the entire area of ​​the first video stream and is monochrome (e.g., pixels of one color (such as blue, black, white, etc.)). As described herein, as a result of a system error, this may occur due to the second video controller 24 erroneously adding a high (e.g., 1.0) alpha value to each (or most) pixels during the generation of the second video stream.

[0087] The third stage 62 shows the video content of the failover hybrid video stream displayed when the surgical system is operating in failover mode. Specifically, the figure is showing that the first video stream is superimposed above the area of ​​the hybrid video stream (e.g., above the monochrome screen of the second video stream). In one aspect, the first video stream can be superimposed across the entire area of ​​the screen (e.g., as large as the screen). On the other hand, the first video screen can be set in an area smaller than the screen, as shown herein. This allows the surgical system to display one or more notifications in one or more areas that do not block the first video stream (e.g., the surgical system can display the notification between the first video stream and the top of the display in the third stage 62).

[0088] Although certain aspects have been described and shown in the accompanying drawings, it should be understood that such aspects are merely illustrative of the invention and not limiting, and the invention is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those skilled in the art. Therefore, this description should be regarded as illustrative and not limiting.

[0089] To assist the Patent Office and any reader of any patent issuing on this application in interpreting the claims appended hereto, Applicants wish to note that unless the words "means for" or "step for" are expressly used in a particular claim, Applicants do not intend for any appended claim or claim element to invoke 35 U.S.C. § 112(f).

[0090] As previously explained, one aspect of the present disclosure may be a non-transitory machine-readable medium (such as a microelectronic memory) having instructions stored thereon that program one or more data processing components (collectively referred to herein as "processors") to automatically (e.g., without user intervention) perform video stream control and display operations as described herein. In other aspects, some of these operations may be performed by specific hardware components that include hardwired logic. These operations may alternatively be performed by any combination of programmed data processing components and fixed hardwired circuit components.

[0091] Although certain aspects have been described and shown in the accompanying drawings, it should be understood that such aspects are merely illustrative of the present disclosure and not limiting, and the present disclosure is not limited to the specific constructions and arrangements shown and described, as various other modifications may occur to those skilled in the art. Therefore, this description should be regarded as illustrative rather than limiting.

[0092] In some aspects, the present disclosure may include language such as, “at least one of [element A] and [element B]”. The language may refer to one or more of the elements. For example, “at least one of A and B” may refer to “A”, “B”, or “A and B”. Specifically, “at least one of A and B” may refer to “at least one of A and at least one of B” or “at least one of A or B”. In some aspects, the present disclosure may include language such as, “[element A], [element B], and / or [element C]”. The language may refer to any one of these elements or any combination thereof. For example, “A, B, and / or C” may refer to “A”, “B”, “C”, “A and B”, “A and C”, “B and C”, or “A, B, and C”.

Claims

1. A method performed by a video controller, the method comprising: receiving a first video stream captured by an endoscope of a surgical system; receiving a second video stream comprising surgical data; displaying the second video stream superimposed over an area of ​​the first video stream; determining that the second video stream is no longer superimposed over the area of ​​the first video stream; In response to determining that the second video stream is no longer superimposed over the area of ​​the first video stream, continuing to display the first video stream.

2. A method according to claim 1, wherein the surgical data includes a graphical user interface (GUI), which includes at least one of: notifications associated with surgical procedures, image data captured by one or more cameras of the surgical system, and user interface (UI) items for allowing a user to interact with the GUI through a display.

3. The method of claim 1 , wherein the display comprises a plurality of rows of pixels, wherein displaying the second video stream superimposed over an area of ​​the first video stream comprises: generating a mixed video stream by mixing the first video stream with the second video stream; storing selected pixels for a group of rows among the plurality of rows in a row buffer based on the mixed video stream; as well as The stored selected pixels are provided from the row buffer to the display.

4. The method of claim 1 , wherein the area is a first area, wherein when the second video stream is superimposed over the area of ​​the first video stream, the surgical system is in a hybrid mode in which a hybrid video stream of the first video stream and the second video stream is displayed, wherein the method further comprises presenting a notification providing a recommendation to a user of the surgical system to switch to a failover mode in which the first video stream is superimposed over the second area of ​​the hybrid video stream, wherein the first video stream continues to be displayed in response to receiving a user input via a user input device for switching from the hybrid mode to the failover mode. 5 . The method of claim 1 , wherein determining that the second video stream is no longer superimposed comprises determining that the area of ​​the first video stream on which the second video stream is displayed exceeds a threshold area.

6. The method of claim 1, wherein the video controller is a first video controller, wherein the second video stream is received from a second video controller of the surgical system.

7. The method of claim 6, wherein the first video controller is a field programmable gate array (FPGA) and the second video controller is a graphics processing unit (GPU).

8. The method of claim 1 , wherein displaying the second video stream superimposed over an area of ​​the first video stream comprises providing a first mixed video stream comprising the first video stream and the second video stream to a display, wherein continuing to display the first video stream comprises: generating a second mixed video stream by retrieving a first set of one or more video frames of the first video stream from a frame buffer and mixing the first set of video frames with a second set of one or more video frames of the first mixed video stream; and The second mixed video stream is provided to a display.

9. A video control device for a surgical system including an endoscope and a display, the device comprising: a first video controller configured to receive surgical data associated with the surgical system; and a second video controller, wherein both the first video controller and the second video controller are disposed within a housing of the video control device, wherein the second video controller is configured to receive a first video stream from the endoscope; receiving a second video stream including the surgical data from the first video controller; displaying the second video stream superimposed over an area of ​​the first video stream on the display; determining that the second video stream is no longer superimposed over the area of ​​the first video stream; as well as In response to determining that the second video stream is no longer superimposed over the area of ​​the first video stream, continuing to display the first video stream.

10. The video control device of claim 9, wherein the first video controller draws power from a first power source and the second video controller draws power from a second power source. 11 . The video control apparatus of claim 9 , wherein the first video controller is a graphics processing unit (GPU), and the second video controller is a field programmable gate array (FPGA).

12. A video control device according to claim 9, wherein the surgical data includes a graphical user interface (GUI), which includes at least one of the following: notifications associated with surgical procedures, image data captured by one or more cameras of the surgical system, and user interface (UI) items for allowing a user to interact with the GUI through a display.

13. The video control device of claim 9, wherein the display comprises a plurality of pixel rows, wherein the second video controller displays the second video stream superimposed over an area of ​​the first video stream by: generating a mixed video stream by mixing the first video stream with the second video stream; storing selected pixels for a group of rows of the plurality of rows in a row buffer based on the mixed video stream; and The stored selected pixels are provided from the row buffer to the display.

14. The video control device of claim 9, wherein the area is a first area, wherein when the second video stream is superimposed over the area of ​​the first video stream, the surgical system is in a hybrid mode in which a hybrid video stream of the first video stream and the second video stream is displayed, wherein the second video controller is further configured to present a notification providing a recommendation to a user of the surgical system to switch to a failover mode in which the first video stream is superimposed over the second area of ​​the hybrid video stream, wherein the first video stream continues to be displayed in response to the second video controller receiving a user input via a user input device for switching from the hybrid mode to the failover mode. 15 . The video control apparatus of claim 9 , wherein the second video controller determines that the second video stream is no longer superimposed by determining that the area of ​​the first video stream on which the second video stream is displayed exceeds a threshold area.

16. The video control device according to claim 9, wherein the second video controller displays the second video stream superimposed on the area of ​​the first video stream by providing a first mixed video stream including the first video stream and the second video stream to a display, wherein the second video controller continues to display the first video stream by: generating a second hybrid video stream by retrieving a first set of one or more video frames of the first video stream from a frame buffer and blending the first set of video frames with a second set of video frames of the first hybrid video stream; and The second mixed video stream is provided to a display.

17. A system, comprising: Endoscope; monitor; and Video controller; and a memory having instructions that, when executed by the video controller, cause the system to: receiving a first video stream captured by the endoscope; receiving a second video stream comprising surgical data; displaying the second video stream superimposed over an area of ​​the first video stream on the display; determining that the second video stream is no longer superimposed over the area of ​​the first video stream; In response to determining that the second video stream is no longer superimposed over the area of ​​the first video stream, Continue to display the first video stream.

18. A system according to claim 17, wherein the surgical data includes a graphical user interface (GUI), which includes at least one of: notifications about surgical procedures, image data captured by one or more cameras of the surgical system, and user interface (UI) items for allowing a user to interact with the GUI.

19. The system of claim 17, wherein the display comprises a plurality of rows of pixels, wherein the instructions to display the second video stream superimposed over an area of ​​the first video stream comprise instructions to: generating a mixed video stream by mixing the first video stream with the second video stream; storing selected pixels for a group of rows of the plurality of rows in a row buffer based on the mixed video stream; and The stored selected pixels are provided from the row buffer to the display.

20. The system of claim 17, wherein the area is a first area, wherein when the second video stream is superimposed over the area of ​​the first video stream, the surgical system is in a blended mode in which a blended video stream of the first video stream and the second video stream is displayed, The memory also has instructions for presenting a notification that provides a recommendation to a user of the surgical system to switch to a failover mode in which the first video stream is superimposed over a second area of ​​the hybrid video stream, wherein the first video stream continues to be displayed in response to receiving user input via a user input device for switching from the hybrid mode to the failover mode.

21. The system of claim 17, wherein the instructions to determine that the second video stream is no longer superimposed include instructions to determine that the area of ​​the first video stream over which the second video stream is displayed exceeds a threshold area.

22. The system of claim 17, wherein displaying the second video stream superimposed over an area of ​​the first video stream comprises providing a first mixed video stream comprising the first video stream and the second video stream to a display, wherein continuing to display the first video stream comprises: generating a second hybrid video stream by retrieving a first set of one or more video frames of the first video stream from a frame buffer and blending the first set of video frames with a second set of one or more video frames of the first hybrid video stream; and The second mixed video stream is provided to a display.

23. The system of claim 17, wherein the video controller is a field programmable gate array (FPGA), wherein the second video stream is received from a graphics processing unit (GPU) of the surgical system.

Images