User interface for participant selection during surgical streaming

Abstract

One aspect provides participant selection during surgical streaming of video and / or audio between an operating room and one or more external participants external to the operating room. User interfaces and controls within the operating room may be distinct from external user interfaces and controls. A user within the operating room may invite participants and control rendering of the internal video stream or the external video stream in a major portion of the display. Each user may make a personalized selection of content to be displayed on a major portion of each user display.

Description

Background Technology

[0001] This disclosure relates generally to computing techniques, and more specifically to computing techniques for selecting multiple participants during surgical streaming.

[0002] Computer-assisted systems, particularly computer-assisted surgery (CAS) systems, can rely on video data captured digitally during surgery performed in the operating room. This video data can be stored and / or streamed. In some cases, the video data can be used within the system to enhance a person's bodily sensations, perceptions, and responsiveness. For example, such a system can effectively provide information corresponding to an expanded field of vision in time and space, enabling a person to adjust current and future actions based on portions of the environment not included in their physical field of vision. Alternatively or additionally, the video data (which may include or include audio data captured by one or more microphones) can be stored and / or transmitted for various purposes, such as archiving, surgical notes, training, postoperative analysis, and / or patient consultation. Summary of the Invention

[0003] According to one aspect, a computer-implemented method includes: displaying a first group of one or more video streams inside an operating room on a first panel of an operating room monitor; and displaying a second group of one or more video streams outside the operating room on a second panel of the operating room monitor. The method further includes detecting a selection of one or more video streams in the second group as a selected video stream. The method further includes removing a currently displayed video stream as a removed video stream in a main portion of the operating room monitor based on the detected selection, and moving the selected video stream from the second panel to the main portion of the operating room monitor based on the detected selection.

[0004] According to one aspect, a computer program product includes a memory device storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations including: displaying two or more video streams, the two or more video streams including at least one video stream on a panel of a display showing the interior of an operating room and at least one video stream on a panel of the display showing the exterior of the operating room. These operations also include detecting a selection of one of the two or more video streams as a selected video stream. These operations further include moving a current video stream, which will be displayed in a main portion of the display, to the panel based on the detection of the selection, and maintaining the position of the surgeon's video stream fixed at the same location on the panel before and after moving the current video stream from the main portion of the display to the panel. These operations also include moving the selected video stream from the panel to the main portion of the display based on the detection of the selection.

[0005] According to one aspect, a system includes an operating room monitor, a memory system, and a processing system coupled to the memory system and the operating room monitor. The processing system is configured to execute multiple instructions to: display one or more video streams from inside the operating room on a first panel of the operating room monitor; display one or more video streams from outside the operating room on a second panel of the operating room monitor; detect a selection of one or more video streams from the first or second group as a selected video stream; and based on the detected selection, move the selected video stream to a main portion of the operating room monitor.

[0006] Additional technical features and benefits are achieved through the technology of this invention. Various aspects of the invention are described in detail herein and are considered part of the claimed subject matter. For a better understanding, please refer to the detailed description and accompanying drawings. Attached Figure Description

[0007] The details of the proprietary rights described herein are specifically pointed out and explicitly claimed in the claims at the end of the specification. The foregoing and other features and advantages of various aspects of the invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: Figure 1 A computer-assisted surgery (CAS) system is described based on one or more aspects; Figure 2 It describes a surgical system based on one or more aspects; Figure 3 It describes a system for predicting generation that can be incorporated based on one or more aspects; Figure 4 A user interface with multiple video streams in an operating room, based on one or more aspects, is described; Figure 5A The user interface is described before the participant flow selection within the operating room is based on one or more aspects; Figure 5B The user interface is described after the participant flow selection within the operating room is based on one or more aspects; Figure 6A The user interface is described before the flow of participants outside the operating room is selected based on one or more aspects. Figure 6B The user interface is described after the participant flow selection outside the operating room is based on one or more aspects; Figure 7 A flowchart is provided illustrating a method for video stream management of operating room displays within the operating room, based on one or more aspects. Figure 8A flowchart is depicted illustrating a method for video stream management of an operating room monitor outside the operating room, based on one or more aspects; and Figure 9 It describes a computer system based on one or more aspects.

[0008] The figures depicted herein are illustrative. Many variations of these figures and / or operations can be made without departing from the spirit of the invention. For example, actions may be performed in different orders, or actions may be added, deleted, or modified. Furthermore, the term "coupled" and its variations describe a communication path between two elements and do not imply a direct connection between elements without intermediate elements / connections. All such variations are considered part of this specification. Detailed Implementation

[0009] Exemplary aspects of the technical solutions described herein include systems and methods for selecting video streams via one or more user interfaces during surgical procedures. As an example, a video streaming system can allow multiple participants outside the operating room to observe and interact with a surgeon or surgical team inside the operating room. The user interface inside the operating room can provide the surgeon or surgical team with the ability to invite one or more participants to observe the surgery and interact via audio, video, and / or remote annotation. Participants outside the operating room can use different user interfaces that allow participants to customize their viewing preferences and interact via audio, video, and / or remote annotation. Further interaction is possible through interactive chat while streaming is active and annotations that can be added during or after streaming to link to a recording of the surgical video. For example, the surgical video may include an endoscopic view of the surgery. Furthermore, multiple cameras or selectable viewpoints may be present to capture the surgery. In some aspects, the surgical video may be captured along with overlay content, such as using color and / or text overlay for structural recognition. The overlay content may be merged with the surgical video or managed as a separate stream, allowing viewers to have the option to turn the overlay content on or off. Furthermore, the surgical team or other observers can generate point-of-interest (POI) indicators while the streaming is active to indicate the occurrence of an event captured in at least one video stream within the video stream. POI indicators can be timestamped markers or labels aligned with the recording time of the surgical procedure. POI indicators can be editable to allow for the addition of notes related to the event of interest during or after the completion of the surgical procedure.

[0010] An operating room may include cameras and microphones located on a central control console and / or one or more cameras and microphones attached (e.g., via clips or other devices) to medical personnel or objects within the operating room. Alternatively or additionally, one or more cameras and microphones may be attached to or integrated into one or more devices within the operating room, such as, but not limited to, surgical instruments, goggles, personal computers, smartwatches, and / or smartphones. One or more cameras with microphones may be designated to provide a view of the surgeon or surgical team within the operating room. One or more surgical cameras may be combined with surgical instruments, such as laparoscopes or more generally, endoscopic cameras. Therefore, while some resulting video feeds may include an audio component, others may not.

[0011] Now go to Figure 1 An example CAS system 100 is shown in general, according to one or more aspects. The CAS system 100 includes at least a computing system 102, a video / audio recording system 104, and a surgical instrument system 106. (As...) Figure 1 As shown, actor 112 can be a medical professional performing surgery on patient 110 using CAS system 100. The medical professional or healthcare professional can be a surgeon, assistant, nurse, administrator, or any other actor interacting with CAS system 100 in the surgical environment. The surgery can be any type of procedure, such as, but not limited to, open or laparoscopic hernia repair, laparoscopic cholecystectomy, robotic laparoscopic surgery, or any other surgical procedure with or without the aid of a robot. In other examples, actor 112 can be a surgeon, anesthesiologist, operating room nurse, technician, administrator, engineer, or any other such person interacting with CAS system 100. For example, actor 112 can log data from CAS system 100, configure / update one or more properties of CAS system 100, check past performance of CAS system 100, repair CAS system 100, etc.

[0012] Surgical procedures may include multiple phases, and each phase may include one or more surgical actions. A "surgical action" may include incision, compression, staples, clamping, suturing, cauterization, closure, or any other such action performed to complete a phase of a surgical procedure. A "phase" refers to a surgical event consisting of a series of steps (e.g., closure). A "step" refers to achieving a specified surgical goal (e.g., hemostasis). During each step, certain surgical instruments 108 (e.g., clamps) are used to achieve the specific goal by performing one or more surgical actions.

[0013] Figure 1The illustrated video / audio recording system 104 includes one or more cameras 105, such as operating room cameras, endoscopic cameras, etc. Cameras 105 capture video data of the surgical procedure being performed. The video / audio recording system 104 includes one or more video capture devices, which may include cameras 105 placed in the operating room to capture events around (i.e., outside the body) of the patient undergoing surgery. The video / audio recording system 104 further includes cameras 105 (e.g., endoscopic cameras) that pass through the patient 110 to capture endoscopic data. The endoscopic data provides video and images of the surgical procedure.

[0014] The video / audio recording system 104 also includes one or more microphones 107, which may be located on a central control console, attached (e.g., via clips or other means) to medical personnel or objects in the operating room, and / or attached to or integrated into one or more devices in the operating room. Examples of devices in the operating room may include, but are not limited to, surgical instruments, video recorders, cameras, goggles, personal computers, smartwatches, and / or smartphones. The microphones 107 capture audio data and may be wired, wireless, or a combination of both.

[0015] In an exemplary aspect, both the video data captured by camera 105 and the audio data captured by microphone 107 can include a timestamp (or other marker) for associating the video data with the audio data. The timestamp can be used to associate or synchronize the sound captured in the operating room with images of the medical procedure performed in the operating room.

[0016] The computing system 102 includes one or more memory devices, one or more processors, user interface devices, and other components. Figure 1 All or part of the computing system 102 shown can be, for example, made by Figure 9The computing system 102 is implemented in whole or in part as a computer system 800. The computing system 102 can execute one or more computer-executable instructions. Execution of the instructions facilitates the computing system 102 in performing one or more methods, including those described herein. The computing system 102 can communicate with other computing systems via wired and / or wireless networks. In one or more examples, the computing system 102 includes one or more trained machine learning models capable of detecting and / or predicting features of a surgical procedure being performed or previously performed. Features may include structures (such as anatomical structures) and surgical instruments 108 in captured surgical video. Features may further include events such as stages, actions, etc., during the surgical procedure. Detected features may further include actor 112 and / or patient 110. In one or more examples, based on the detection, the computing system 102 can provide suggestions for subsequent actions to be taken by actor 112. Alternatively or additionally, the computing system 102 may provide one or more reports based on the detection. The detection performed by the machine learning model can be performed autonomously or semi-autonomously.

[0017] Machine learning models can include artificial neural networks (such as deep neural networks, convolutional neural networks, recurrent neural networks), encoders, decoders, or any other type of machine learning model. Machine learning models can be trained in a supervised, unsupervised, or hybrid manner. Machine learning models can be trained to perform detection and / or prediction using one or more types of data acquired by the CAS system 100. For example, a machine learning model can use video data captured via the video / audio recording system 104. Alternatively or additionally, a machine learning model can use surgical instrument data from the surgical instrument system 106. In yet another example, a machine learning model can use a combination of video data and surgical instrument data.

[0018] Additionally, in some examples, the machine learning model may also use audio data captured by one or more microphones 107 during surgical procedures. The audio data may include sounds emitted by the surgical instrument system 106 when one or more surgical instruments 108 are activated. Alternatively or additionally, the audio data may include voice commands, snippets, or dialogues from one or more actors 112. The audio data may further include sounds produced by the surgical instruments 108 during their use.

[0019] After training, one or more machine learning models can then be used in real time to process one or more data streams (e.g., video streams, audio streams, RFID data, etc.). Processing may include visual modifications in images of a surgical procedure video to predict and characterize the procedure based on one or more surgical stages, instruments, and / or other structures within various instantaneous or block time periods. The visualization can be modified to highlight the presence, location, and / or use of one or more structures. Alternatively or additionally, these structures can be used to identify stages within a workflow (e.g., represented via surgical data structures), predict future stages within a workflow, etc.

[0020] In one or more examples, the machine learning model can detect surgical actions, surgical stages, anatomical structures, surgical instruments, activities, events, and various other features from data associated with surgery. In some examples, detection can be performed in real time. Alternatively or additionally, the computing system 102 analyzes surgical data (i.e., various types of data captured during surgery) offline (e.g., post-operatively). In one or more examples, the machine learning model detects surgical stages based on detecting some features among characteristics such as anatomical structures and surgical instruments.

[0021] Data collection system 150 can be used to store surgical data, including multiple video recordings captured during surgery and audio data captured during surgery. Data collection system 150 includes one or more storage devices 152. Data collection system 150 can be a local storage system, a cloud-based storage system, or a combination thereof. Further, data collection system 150 can use any type of cloud-based storage architecture, such as public cloud, private cloud, hybrid cloud, etc. In some examples, the data collection system can use distributed storage, i.e., storage devices 152 are located in different geographical locations. Storage devices 152 can include any type of electronic data storage medium for recording machine-readable data, such as semiconductor-based storage media, magnetic-based storage media, optical-based storage media, or combinations thereof. For example, data storage media can include flash-based solid-state drives (SSDs), magnetic-based hard disk drives, magnetic tape, optical discs, etc.

[0022] In one or more examples, the data collection system 150 may be part of the video / audio recording system 104, or vice versa. In some examples, the data collection system 150, the video / audio recording system 104, and the computing system 102 may communicate with each other via a communication network, which may be wired, wireless, or a combination thereof. Communication between the systems may include the transmission of data (e.g., video data, audio data, instrument data, etc.), data manipulation commands (e.g., browsing, copying, pasting, moving, deleting, creating, compressing, etc.), data manipulation results, etc. In one or more examples, the computing system 102 may manipulate data already stored / being stored in the data collection system 150 based on the output of one or more machine learning models (e.g., stage detection, structure detection, etc.). Alternatively or additionally, the computing system 102 may manipulate data already stored / being stored in the data collection system 150 based on information from the surgical instrument system 106.

[0023] In one or more examples, video captured by video / audio recording system 104 is stored on data collection system 150. In some examples, computing system 102 processes portions of the video data stored on data collection system 150. In some examples, computing system 102 filters the video captured by video / audio recording system 104 before it is stored on data collection system 150. Alternatively or additionally, computing system 102 filters the video after it has been stored on data collection system 150.

[0024] The surgical data management system 160 can provide access to portions of the data captured in the data collection system 150, as well as data and records stored in other systems. Participant systems 165A to 165N can access the surgical data management system 160 through one or more applications or a secure webpage. Participant systems 165A to 165N can include various types of computing devices, such as personal computers, laptops, tablets, mobile devices, smart appliances, etc. The surgical data management system 160 can be a standalone application, module, and / or extension of another system, including processing systems and networking-enabled hardware and software to support the operation of the surgical data management system 160. Video, with or without audio, can be transmitted through the data collection system 150 and the surgical data management system 160 to support real-time streaming between users of participant systems 165A to 165N and one or more actors 112 via camera 105 and / or microphone 107. Surgical instruments 108 can also be a streaming source.

[0025] Now go to Figure 2The surgical system 200 is shown in general according to one or more aspects. Figure 2 The examples depict items that may include or can be coupled to Figure 1 The CAS system 100 includes a surgical support system 202. The surgical support system 202 can utilize one or more cameras 204 (e.g., ...). Figure 1 The surgical support system 202 may also use one or more microphones 220 (e.g., camera 105) to acquire image or video data. Figure 1 The surgical support system 202 can further interface with multiple sensors 206 and effectors 208. Sensors 206 can be associated with surgical support equipment and / or patient monitoring. Effectors 208 can be robotic components or other devices controllable by the surgical support system 202 (e.g., microphone 107) to acquire audio data. Figure 1 Surgical instruments 108). Surgical support system 202 can also interact with one or more user interfaces 210 (such as various input and / or output devices). When... Figure 1 When patient 110 undergoes surgery, surgical support system 202 can store, access, and / or update surgical data 214 associated with training datasets and / or real-time data. Surgical support system 202 can store, access, and / or update surgical objectives 216 to aid in training and guiding one or more surgical procedures. User configuration 218 can track and store user preferences.

[0026] The surgical support system 202 can also communicate with other systems via network 230. For example, the surgical support system 202 can communicate with other types of devices (such as computing devices 232A, 232B, ..., 232N (e.g., mobile phones, tablets, or laptops)) via network 230. As an example, the user interface 210 can be connected to or integrated with the surgical support system 202 via a local connection (e.g., within the operating room), while mobile computing devices 234 are connected to the surgical support system 202 via network 230, which can extend to a large geographical area outside the operating room. Computing devices 232A to 232N are... Figure 1 Examples of participant systems 165A to 165N. Accordingly, Figure 1 The surgical data management system 160 can be inserted between the computing devices 232A to 232N and the network 230 to manage data flow, streaming, and access constraints. Furthermore, portions of the surgical data management system 160 can be executed locally by the computing devices 232A to 232N and / or the surgical support system 202.

[0027] Now go to Figure 3The document generally illustrates a system 300 for analyzing data, including video data, based on one or more aspects. For example, it can be derived from... Figure 1 The video / audio recording system 104 captures video data. This analysis allows the use of machine learning to predict surgical stages and structures (e.g., instruments, anatomical structures, etc.) within the video data. In one or more examples, system 300 may be... Figure 1 The CAS system 100 or a part thereof. According to some aspects, system 300 uses data streams from surgical data to identify the surgical status.

[0028] System 300 includes a data receiving system 305 that collects surgical data, including video data and surgical instrument data. The data receiving system 305 may include one or more devices (e.g., one or more user devices and / or servers) located within and / or associated with the operating room and / or control center. The data receiving system 305 can receive surgical data in real time, i.e., while surgery is being performed. Alternatively or additionally, the data receiving system 305 can receive or access surgical data offline, for example, by accessing data stored in… Figure 1 The data is accessed through the data collection system 150.

[0029] System 300 further includes a machine learning processing system 310 that uses one or more machine learning models to process surgical data to identify one or more features in the surgical data, such as surgical stage, instruments, anatomical structures, etc. It will be understood that the machine learning processing system 310 may include one or more devices (e.g., one or more servers), each of which may be configured to include part or all of one or more of the depicted components of the machine learning processing system 310. In some instances, part or all of the machine learning processing system 310 is located in the cloud and / or remotely from the operating room and / or physical location corresponding to part or all of the data receiving system 305. It should be understood that several components of the machine learning processing system 310 are depicted and described herein. However, these components are only one example structure of the machine learning processing system 310, and in other examples, different combinations of these components may be used to construct the machine learning processing system 310. Such variations in the combination of components are included in the technical solutions described herein.

[0030] The machine learning processing system 310 includes a machine learning training system 325, which can be a separate device (e.g., a server) that stores its output as one or more trained machine learning models 330. The machine learning models 330 can be accessed by a machine learning execution system 340. In some examples, the machine learning execution system 340 can be decoupled from the machine learning training system 325. In other words, in some aspects, the device for “training” the model is separate from the device for “inference” (i.e., using the trained machine learning model 330 to perform real-time processing of surgical data).

[0031] In some examples, the machine learning processing system 310 further includes a data generator 315 for generating simulated surgical data (such as a virtual image set) or recording video data from the video / audio recording system 104 to train the machine learning model 330. The data generator 315 can access (read / write) the data storage 320 to record data, including multiple images and / or multiple videos. Images and / or videos may include images and / or videos collected during one or more surgeries (e.g., one or more surgical procedures). For example, images and / or videos may have already been... Figure 1 The user device worn by the actor 112 (e.g., surgeon, surgical nurse, anesthesiologist, etc.) during the operation, the non-wearable imaging device located in the operating room, or the inserted device Figure 1 The endoscopic camera collects data from the patient 110. In some examples, the data storage device 320 can be connected to... Figure 1 The data collection system 150 is separate. In other examples, the data storage 320 may be part of the data collection system 150.

[0032] Each image and / or video recorded in data storage 320 for training machine learning model 330 can be defined as a base image and can be associated with other data representing the associated surgery and / or rendering specifications. For example, other data can identify the type of surgery, surgical location, one or more persons involved in performing the surgery, surgical objectives, and / or surgical outcomes. Alternatively or additionally, other data can indicate the surgical stage corresponding to the image or video, the rendering specifications corresponding to the image or video, and / or the type of imaging device that captured the image or video (e.g., if the device is a wearable device, and / or the role of the specific person wearing the device, etc.). Further, other data can include image segmentation data that identifies and / or represents one or more objects (e.g., tools, anatomical objects, etc.) depicted in the image or video. The representation can indicate the position, orientation, or pose of an object in the image. For example, the representation can indicate a set of pixels corresponding to an object and / or the object state resulting from past or current user actions. Localization can be performed using various techniques for identifying objects in one or more coordinate systems.

[0033] Machine learning training system 325 uses recorded data in data storage 320 to train machine learning model 330. This recorded data may include simulated surgical data (e.g., a set of virtual images) and actual surgical data. Machine learning model 330 can be defined based on model type and a set of hyperparameters (e.g., based on input from a client device). Machine learning model 330 can be configured based on a parameter set, which can be dynamically defined based on (e.g., continuous or repeated) training (i.e., learning, parameter tuning). Machine learning training system 325 can use one or more optimization algorithms to define this parameter set to minimize or maximize one or more loss functions. The (learned) parameter set can be stored as part of the trained machine learning model 330 using a specific data structure of the trained machine learning model 330. The data structure may also include one or more non-learnable variables (e.g., hyperparameters and / or model definitions).

[0034] The machine learning execution system 340 can access multiple data structures of the machine learning model 330 and configure the machine learning model 330 accordingly for inference (i.e., prediction). The machine learning model 330 may include, for example, a fully convolutional network adaptation, an adversarial network model, an encoder, a decoder, or other types of machine learning models. The type of the machine learning model 330 can be indicated in the corresponding data structure. The machine learning model 330 can be configured based on one or more hyperparameters and the learned parameter set.

[0035] One or more machine learning models 330 receive surgical data to be processed as input during execution and subsequently generate one or more inferences based on training. For example, by Figure 1 The video data captured by the video / audio recording system 104 may include a data stream (e.g., an array of intensity, depth, and / or RGB values) of each of a single image or a set of frames (e.g., images comprising multiple images or images with sequential data), the set of frames representing a fixed or variable-length time window in the video. The video data captured by the video / audio recording system 104 may be received by a data receiving system 305, which may include one or more devices located in the operating room where the surgery is being performed. Alternatively, the data receiving system 305 may include a remote device to which the captured video data is streamed in real time during the execution of the surgery. Alternatively or additionally, the data receiving system 305 may access data offline from the data collection system 150 or any other data source (e.g., local or remote storage devices).

[0036] Data receiving system 305 can process received video and / or other data. When a video stream in an encoded format is received, this processing can include decoding, enabling the extraction and processing of data from the image sequence. Data receiving system 305 can also process other types of data included in the input surgical data. For example, surgical data may include additional data streams such as audio data, RFID data, text data, measurements from one or more surgical instruments / sensors, etc., which may represent the stimulation / surgical state in the operating room. Data receiving system 305 synchronizes different inputs from different devices / sensors before feeding them into machine learning processing system 310. In some aspects, audio data can also be used as a data source to generate predictions. Synchronization can be achieved by generating timestamps with each data stream using a common reference clock. The clock can be shared via a network protocol, hardware locking, or any other means. Such timestamps can be associated with any processed data format, such as, but not limited to, text or other discrete data created from an audio signal. Additional synchronization can be performed by linking actions, events, or stages that have been automatically processed from the raw signal using a machine learning model. For example, text generated from an audio signal can be associated with a specific stage of the surgery extracted from that audio or any other data stream signal. The generated text can be captured and / or displayed through the user interface.

[0037] Once trained, the machine learning model 330 can analyze input surgical data and, in one or more aspects, predict and / or characterize structures included in video data containing the surgical data. The video data can include continuous images and / or encoded video data (e.g., encoded using digital video file / stream formats and / or codecs, such as MP4, MOV, AVI, WEBM, AVCHD, OGG, etc.). The prediction and / or characterization of structures can include segmenting the video data or predicting the location of structures using probabilistic heatmaps. In some instances, one or more machine learning models include or are associated with preprocessing or enhancement (e.g., intensity normalization, resizing, cropping, etc.) performed prior to segmenting the video data. The output of one or more machine learning models can include image segmentation or probabilistic heatmap data indicating which structures (if any) of a defined set of structures were predicted within the video data, the location and / or orientation and / or pose of the structures(s) within the video data, and / or the state of the structures(s). Location can be a set of coordinates in an image / frame within the video data. For example, coordinates can provide a bounding box. Coordinates can provide the boundaries around the predicted structures(s). In one or more examples, the trained machine learning model 330 is trained to perform higher-level predictions and tracking, such as predicting the stage of a surgical procedure and tracking one or more surgical instruments used in the procedure.

[0038] While this document describes some techniques for predicting surgical stages (“stages”) in surgical procedures, it should be understood that any other stage prediction techniques may be used without affecting any aspect of the technical solutions described herein. In some examples, the machine learning processing system 310 includes a detector 350 that uses a machine learning model to identify stages within a surgical procedure (“operation”). Detector 350 uses a specific surgical tracking data structure 355 from a list of surgical tracking data structures. Detector 350 selects the surgical tracking data structure 355 based on the type of surgery being performed. In one or more examples, the type of surgery is predetermined or input by actor 112. The surgical tracking data structure 355 identifies a set of potential stages that may correspond to a portion of a particular type of surgery.

[0039] In some examples, the surgical tracking data structure 355 may be a graph comprising a set of nodes and a set of edges, where each node corresponds to a potential stage. These edges can provide directional connections between nodes, indicating (via direction) the expected order in which stages will be encountered throughout the procedure's iterations. The surgical tracking data structure 355 may include one or more branch nodes that feed to multiple next nodes and / or may include one or more divergence points and / or convergence points between nodes. In some instances, a stage indicates a surgical action being performed or already performed (e.g., a surgical procedure) and / or a combination of actions already performed. In some instances, a stage is related to the biological state of the patient undergoing surgery. For example, the biological state may indicate complications (e.g., blood clots, arterial / venous obstruction, etc.), prerequisites (e.g., lesions, polyps, etc.). In some examples, a machine learning model 330 is trained to detect "abnormalities," such as bleeding, arrhythmias, vascular abnormalities, etc.

[0040] Each node within the surgical tracking data structure 355 can identify one or more characteristics corresponding to the stage of that node. These characteristics can include visual characteristics. In some instances, a node identifies one or more tools that are typically in use or available during that stage (e.g., on a tool tray). Nodes also identify one or more roles of the person typically performing the surgical task, typical types of movement (e.g., movement of hands or tools), etc. Thus, detector 350 can use segmented data generated by machine learning execution system 340, indicating the presence and / or characteristics of specific objects within the field of view, to identify estimated nodes corresponding to real image data. The identification of nodes (i.e., stages) can be further based on previously detected stages of a given procedural iteration and / or other detected inputs (e.g., including verbal audio data of person-to-person requests or comments, explicit identification of current or past stages, information requests, etc.).

[0041] Detector 350 outputs a prediction associated with a portion of video data analyzed by machine learning processing system 310. This prediction is associated with the portion of video data by identifying the start and end times of the video portion analyzed by machine learning execution system 340. The output prediction may include identifiers of surgical stages, activities, or events detected by detector 350 based on the output of machine learning execution system 340. Further, in one or more examples, the prediction may include identifiers of structures (e.g., instruments, anatomical structures, etc.) identified by machine learning execution system 340 in the analyzed video portion. The prediction may also include a confidence score. Various types of information that may be included in the output prediction may include stages, actions, and / or events associated with the surgical procedure.

[0042] It should be noted that although some of the accompanying figures depict endoscopic videos being analyzed, the technical solutions described herein can also be applied to analyzing video and image data captured by cameras that are not endoscopes (i.e., cameras outside the patient's body) during open surgery (i.e., non-laparoscopic surgery). For example, video and image data can be captured by cameras mounted on one or more personnel (e.g., surgeons) in the operating room. Alternatively or additionally, the cameras can be mounted on surgical instruments, walls, or other locations in the operating room.

[0043] Now go to Figure 4 It depicts a user interface with multiple video streams in an operating room, based on one or more aspects. Figure 4 The user interface can be accessed through Figure 2 An example of a graphical user interface 210 that displays content and receives input. Figure 4 In the example, the user interface could be an operating room monitor 400 that displays a video stream in a main section 402 and an illustration section 404. The video stream displayed in the main section 402 can be selected between an operating room video stream and a video stream from outside the operating room. The illustration section 404 can include a surgeon's video stream using a camera in the operating room, oriented to view the surgeon or surgical team. In some aspects, the video source of the content in the illustration section 404 is not selectable, such as when only a single camera input is available for the surgeon's video stream. In other aspects, the video source of the content in the illustration section 404 can be selected among multiple sources. Furthermore, the display of the illustration section 404 can be toggled on or off. Additionally, the position of the illustration section 404 relative to the main section 402 can be adjustable, for example, to avoid obscuring the content in the main section 402.

[0044] The user interface aspects of the operating room monitor 400 may include a control bar 405 with multiple selectable controls, such as a participant selector 406, a chat selector 408, an invitation selector 410, a microphone control 412, a surgical camera control 414, a surgeon camera control 416, a point of interest (POI) selector 418, a remote annotation control 420, and an end-of-flow control 422. The control bar 405 may be located below both the main section 402 and the illustration section 404. Alternatively, the control bar 405 may be located elsewhere on the operating room monitor 400 (e.g., above the main section 402 and the illustration section 404), or arranged in a substantially vertical orientation to the left or right of the main section 402. In some respects, the position of the control bar 405 is configurable based on user preferences and / or other factors. The controls on the control bar 405 can be selected via a touch-sensitive screen, one or more buttons, or other input types such as a mouse or trackball.

[0045] In various ways, selecting the participant selector 406 can cause the operating room monitor 400 to be switched to an adjusted state, thereby opening one or more panels to view the video stream of the current participant in the streaming and / or adjusting the current content source of the main section 402 of the operating room monitor 400. Selecting the chat selector 408 can open a chat interface to exchange messages and content between the participant and the surgeon or their subgroup. The invitation selector 410 can trigger the sending of electronic invitations to allow one or more parties to join as participants.

[0046] The microphone control 412 can be operated as a mute button for one or more microphones in the operating room, wherein a mute or unmute status indicator 413 can be superimposed on the illustrated portion 404 or elsewhere on the operating room monitor 400. In some respects, the microphone control 412 can open an options menu for adjustment. Figure 1 The microphone 107 can be used to mute / unmute one or more selected microphones, adjust gain, activate audio filtering, and / or perform other such actions.

[0047] The surgical camera control 414 can be operated as a camera on / off control for a surgical camera (such as an endoscopic camera, e.g., a laparoscopic camera). In some aspects, the surgical camera control 414 can open an options menu to adjust... Figure 1 The camera 105 includes aspects of one or more cameras, such as turning the camera on / off, adjusting video quality, activating video filtering, activating video overlay, and / or other such actions. Figure 4In the example, main section 402 displays a surgical video stream from a laparoscopic camera. Therefore, selecting the surgical camera control 414 can turn the surgical video stream displayed in main section 402 on or off.

[0048] The surgeon camera control 416 can function as a camera on / off control for a camera (such as one or more surgeons or multiple cameras attached to one or more surgeons) that is guided to capture a scene or viewpoint of one or more surgeons. In some respects, the surgeon camera control 414 can open an options menu to adjust... Figure 1 The camera 105 includes aspects of one or more cameras, such as turning the camera on / off, adjusting video quality, activating video filtering, activating video overlay, and / or other such actions. Figure 4 In the example, main section 402 displays the surgeon's video stream in illustration section 404. Therefore, selecting the surgeon camera control 416 can turn the surgeon's video stream displayed in illustration section 404 on or off.

[0049] POI selector 418 can generate an indication of a POI associated with at least one video stream to indicate the occurrence of an event captured in at least one video stream. The POI can be a timestamp aligned with a clock or time since the start of recording. Furthermore, selecting a POI via POI selector 418 can result in a pop-up message visible to the participant. The POI may also include options for selecting a POI note from a pre-filled list of options or manually entering text.

[0050] The remote annotation control 420 allows users to draw electronically on the video stream as an overlay. For example, a surgeon can use a stylus, finger, mouse, or other such input to draw freehand to highlight features depicted in the main section 402. Furthermore, with remote annotation enabled, remote participants can also have the ability to make remote annotations visible to the surgeon and other participants. In some aspects, remote annotations can remain active for a predetermined time period (e.g., 2 to 15 seconds) and then be automatically erased. When a series of remote annotations is performed, the automatic erasure time can be set and reset based on the most recent remote annotation. Further, the automatic erasure countdown can be based on the start of the first remote annotation via user input or the end of the most recent remote annotation via user input. For example, as an example, if the predetermined time period is set to 5 seconds, a 5-second countdown to zero can begin at the first drawing action in the sequence. As another example, the 5-second countdown to zero can be reset at each subsequent remote annotation action. In other words, if a user draws a first shape using remote annotation at time T and then draws a second shape 3 seconds later (e.g., T+3 seconds), the countdown timer can be based on the most recently drawn second shape, so that in this example, both the first and second shapes will be automatically erased at time T+8 seconds. Furthermore, selecting the remote annotation control 420 can open an options menu, such as various colors, line thicknesses, and behavior differences (e.g., automatic or manual erasure).

[0051] The End Stream control 422 can terminate the streaming session for all participants and the surgeon. In some respects, selecting the End Stream control 422 can turn off the operating room display 400 and prevent audio and video from the operating room from being streamed to the participants. In other respects, even after the streaming from the operating room has ended, participants can still be able to participate in shared streaming sessions with each other, such as reviewing surgical procedures, sharing notes, etc.

[0052] Figure 5A The user interface is depicted before the participant flow selection within the operating room, based on one or more aspects. Figure 5A In the example, when a user in the operating room selects participant selector 406, a first group of one or more video streams 504A (e.g., up to 504N (not depicted)) inside the operating room can be displayed on the first panel 506 of the operating room monitor 500, and a second group of one or more video streams 512A, 512B, 512C (e.g., up to 512N (not depicted)) outside the operating room can be displayed on the second panel 508 of the operating room monitor 500. Figure 4Compared to the main section 402, opening the first panel 506 and the second panel 508 can reduce the size of the main section 502. The video stream in the main section 502 may include a fixed control 503 that returns the displayed video stream to the appropriate panel. Figure 5A In the example, the current video stream in the main section 502 is the surgical video stream, so it will be returned to the first panel 506 after selecting the fixed control 503. Similarly, video stream 504A is a surgeon's video stream with a fixed control 505, which, when the first panel 506 is selected, will cause video stream 504A to be moved to the main section 502 and removed from the first panel 506. If the current video stream occupies the main section 502, selecting the fixed control 505 can cause the current video stream to be moved to the appropriate panel (e.g., returning the surgical video stream to the first panel or returning the participant video stream 512 to the second panel 508), while also moving video stream 504A to the main section 502.

[0053] In some respects, the selection of one of video streams 512A to 512C to be pinned to the main section 502 can be performed through a multi-step process. For example, extension commands 514A and 514B can be selected to open an extension command menu 517 for the selected video stream (such as video stream 512C). The extension command menu 517 can be represented as a semi-transparent overlay of video stream 512C. Examples of commands on the extension command menu 517 may include a mute command 518 for turning the associated user's audio stream on or off, a pinning control 520 for extending video stream 512C into the main section 502, and other commands.

[0054] In some respects, the second panel 508 may include other controls, such as a full mute control 510 that disables audio feeds to one or more participants (e.g., all participants) of the second panel 508 via a user interface. The mute status of each participant can be visually seen through mute status indicators 516A and 516B, respectively, associated with video streams 512A and 512B.

[0055] Figure 5A An example of an invitation state 522 that can be depicted when the invitation selector 410 is used to invite a new participant is also shown. Invitation state 522 may also include a nudge control that can trigger the sending of one or more notifications to the party invited to join the current streaming session.

[0056] Figure 5B The user interface is depicted after the participant flow selection within the operating room is based on one or more aspects. Figure 5BIn the example, the user in the operating room selects to pin the video stream 512C to the main section 502 (e.g., by selecting...). Figure 5A After fixing the control (520), remove the display based on detecting the selection. Figure 5A The current video stream in the main section 502 of the operating room monitor 500 is treated as a removed video stream, and based on the detection of this selection, the selected video stream (e.g., video stream 512C) is moved from the second panel 508 to the main section 502 of the operating room monitor 500. Figure 5B In the example, selecting the fixed control 503 in the main section 502 will cause the video stream 512C to be moved back to the second panel 508.

[0057] Figure 6A The user interface is depicted before the participant flow selection outside the operating room, based on one or more aspects. Figure 6A In the example, two or more video streams may be displayed, including at least one video stream inside the operating room on panel 604 of display 600 (e.g., video stream 606) and at least one video stream outside the operating room on panel 604 of display 600 (e.g., 608A, 608B). Any video stream can be selected for display in the main portion 602 of display 600. For example, one of two or more video streams (e.g., video streams 606, 608A, 608B) can be selected as the selected video stream. For example, selection can be based on dragging actions, clicking actions (e.g., double-clicking), or target location selection (e.g., icon selection). The current video stream to be displayed in the main portion 602 of display 600 can be moved to panel 604 based on the detection of selection. If the selected video stream is not the surgeon's video stream (e.g., video stream 606A), the position of the surgeon's video stream can be fixed and maintained in the same position on panel 604 before and after moving the current video stream from the main portion 602 of display 600 to panel 604. For example, when video stream 608A is selected, the selected video stream 608A can be moved from panel 604 to main section 602, and the current video stream (e.g., video stream 606B) can be moved back to panel 604, as shown. Figure 6B As depicted in the example.

[0058] The relative positioning of video streams (e.g., video streams 606A, 606B, 608A, 608B) on panel 604 can be modified based on one or more placement rules. The relative placement of each stream on panel 604 can vary for each user. For example, video streams inside the operating room can be given a higher display priority than video streams outside the operating room. This way, when a new video stream is added, the surgeon's video stream (e.g., video stream 606A) can still remain at the top of panel 604. New video streams or video streams returning from the main section 602 can be placed below or adjacent to the surgeon's video stream. Participant video streams for users outside the operating room can be placed below the video streams inside the operating room. Furthermore, from a user's perspective, users of monitor 600 can remain in panel 604, while other users can be added or repositioned according to placement rules. For example, users who are currently or recently speaking can be given a higher placement priority (e.g., closer to the top of panel 604) than users who haven't recently spoken. As another example, users performing screen sharing or remote annotation can be given a higher placement priority. As a further example, the video stream, which recently appeared in main section 602, can be given a higher placement priority. The audio-only stream can be given a lower placement priority if the participants have their respective cameras off. Other variations of the placement rules are envisioned.

[0059] In various aspects, the user interface of display 600 may include several other controls. For example, display 600 may include a control bar 610 with multiple selectable controls, such as a mute control 612, a camera control 616, a POI selector 618, a remote annotation control 620, and an end control 622. The mute control 612 can turn the user's microphone on or off. The camera control 616 can turn the user's camera on or off. The POI selector 618 can function in conjunction with... Figure 5A , Figure 5B The POI selector 418 is similar. The remote annotation control 620 can function similarly to... Figure 5A , Figure 5B The remote annotation control 420 is similar. The end control 622 can end the user's current session. Further examples of controls on the user interface of the display 600 may include a selection bar 630 that can trigger various actions, such as displaying a menu, navigating to the main interface, contacting other users, viewing facility information, and other such actions. As another example, a toolbar 632 may support search, content uploading, notification management, user account management (e.g., login / logout / profile editing), and other such actions. Additionally, other user interfaces may be selectable, such as a workflow interface 634, an analysis interface 636, an evaluation interface 638, an annotation interface 640, and other such interfaces.

[0060] Now go to Figure 7 The diagram generally illustrates a flowchart of a method 700 for video stream management of an operating room display within an operating room, according to one or more aspects. All or part of method 700 may be derived, for example, by… Figure 1 CAS system 100, surgical system 200, and / or Figure 9 The computer system 800 is used to implement all or part of it.

[0061] At box 702, a first group of one or more video streams 504 inside the operating room can be displayed on a first panel 506 of the operating room monitor 500. At box 704, a second group of one or more video streams 512 outside the operating room can be displayed on a second panel 508 of the operating room monitor 500. At box 706, selection of one or more video streams 512 in the second group can be detected as the selected video stream. At box 708, based on the detection of this selection, the currently displayed video stream in the main section 502 of the operating room monitor 500 can be removed as the removed video stream, and based on the detection of this selection, the selected video stream can be moved from the second panel 508 to the main section 502 of the operating room monitor 500. Other types of selection and movement can be performed for panels 506 and 508. For example, a surgeon's video stream or other operating room video streams can be moved to the main section 502. Further, one of the participants can switch from a camera-based video stream to a shared-screen-based video feed, and a user in the operating room can choose to display the shared-screen-based video feed in the main section 502. Other variations are envisioned.

[0062] Figure 7 The processing shown is not intended to indicate that operations should be performed in any particular order, nor does it indicate that they should be included in every case. Figure 7 All the operations shown. Additionally, Figure 7 The processing shown can include any appropriate number of additional operations.

[0063] Figure 8 A flowchart depicts a method 750 for video stream management of an operating room display outside the operating room, based on one or more aspects. All or part of method 750 may be derived, for example, from… Figure 1 CAS system 100, surgical system 200, and / or Figure 9 The computer system 800 is used to implement all or part of it.

[0064] At box 752, two or more video streams, including at least one video stream inside the operating room (e.g., video streams 606A, 606B), can be displayed on panel 604 of display 600, and at least one video stream outside the operating room (e.g., 608A, 608B) can be displayed on panel 604 of display 600. At box 754, selection of one of the two or more video streams can be detected as the selected video stream. At box 756, based on the detection of this selection, the current video stream displayed in the main portion 602 of display 600 can be moved to panel 604, and the surgeon's video stream position (e.g., topmost / top left position) can be fixed and held in the same position on panel 604 before and after moving the current video stream from the main portion 602 of display 600 to panel 604. At box 758, based on the detection of this selection, the selected video stream can be moved from panel 604 to the main portion 602 of display 600. Other variations are contemplated.

[0065] Figure 8 The processing shown is not intended to indicate that operations should be performed in any particular order, nor does it indicate that they should be included in every case. Figure 8 All the operations shown. Additionally, Figure 8 The processing shown can include any appropriate number of additional operations.

[0066] Now go to Figure 9 The document generally illustrates a computer system 800 according to one aspect. As described herein, computer system 800 can be an electronic computer framework comprising and / or employing any number of computing devices and networks utilizing various communication technologies and combinations thereof. Computer system 800 can be easily expanded, extended, and modularized, and has the ability to be reconfigured for different services or independently of other features. Computer system 800 can be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer system 800 can be a cloud computing node. Computer system 800 can be described in the general context of computer-executable instructions (such as program modules) executed by the computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer system 800 can be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can reside in local and remote computer system storage media, including memory storage devices.

[0067] like Figure 9As shown, the computer system 800 has one or more central processing units (CPUs) 801a, 801b, 801c, etc. (collectively or generally referred to as processors 801). Processors 801 can be single-core processors, multi-core processors, computing clusters, or any number of other configurations. Processors 801 can be any type of circuit capable of executing instructions. Processors 801, also referred to as processing circuits, are coupled to system memory 803 and various other components via system bus 802. System memory 803 may include one or more memory devices, such as read-only memory (ROM) 804 and random access memory (RAM) 805. ROM 804 is coupled to system bus 802 and may include a basic input / output system (BIOS) that controls certain basic functions of computer system 800. RAM is a read-write memory coupled to system bus 802 for use by processor 801. System memory 803 provides temporary storage space for the operation of the instructions during operation. System memory 803 may include random access memory (RAM), read-only memory, flash memory, or any other suitable memory system.

[0068] Computer system 800 includes an input / output (I / O) adapter 806 and a communication adapter 807 coupled to a system bus 802. I / O adapter 806 may be a Small Computer System Interface (SCSI) adapter that communicates with a hard disk 808 and / or any other similar component. I / O adapter 806 and hard disk 808 are collectively referred to herein as mass storage device 810.

[0069] Software 811 for execution on computer system 800 may be stored in mass storage device 810. Mass storage device 810 is an example of a tangible storage medium readable by processor 801, wherein software 811 is stored as instructions executed by processor 801 to operate computer system 800, as described below with reference to the various accompanying drawings. Examples of computer program products and the execution of such instructions are discussed in more detail herein. Communication adapter 807 interconnects system bus 802 with network 812, which may be an external network enabling computer system 800 to communicate with other such systems. In one aspect, a portion of system memory 803 and mass storage device 810 jointly store an operating system, which may be a coordinating... Figure 9 The functions of the various components shown can be performed on any suitable operating system.

[0070] Additional input / output devices are shown connected to system bus 802 via display adapter 815 and interface adapter 816. In one aspect, adapters 806, 807, 815, and 816 may be connected to one or more I / O buses, which are connected to system bus 802 via an intermediate bus bridge (not shown). Display 819 (e.g., a screen or display monitor) is connected to system bus 802 by display adapter 815, which may include a graphics controller and a video controller for improving the performance of graphics-intensive applications. Keyboard, mouse, touchscreen, one or more buttons, speakers, etc., may be interconnected to system bus 802 via interface adapter 816, which may include, for example, a super I / O chip integrating multiple device adapters into a single integrated circuit. Suitable I / O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols such as Peripheral Component Interconnect (PCI). Therefore, as Figure 9 The computer system 800 configured therein includes processing capabilities in the form of a processor 801, storage capabilities including a system memory 803 and a mass storage device 810, such as input devices like buttons and touch screens, and output capabilities including a speaker 823 and a display 819.

[0071] In some respects, the communication adapter 807 can use any suitable interface or protocol (such as an Internet Minicomputer System Interface) to transmit data. The network 812 can be a cellular network, radio network, wide area network (WAN), local area network (LAN), or the Internet. External computing devices can connect to the computer system 800 via the network 812. In some examples, the external computing device can be an external network server or a cloud computing node.

[0072] It should be understood that Figure 9 The block diagram is not intended to indicate that the computer system 800 will include Figure 9 All the components shown. Conversely, computer system 800 may include... Figure 9 Any suitable fewer or additional components not shown herein (e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.). Furthermore, the aspects described herein with respect to the computer system 800 can be implemented with any suitable logic, wherein, in each aspect, the logic as mentioned herein can include any suitable hardware (e.g., processor, embedded controller, or application-specific integrated circuit, etc.), software (e.g., application programs, etc.), firmware, or any suitable combination of hardware, software, and firmware. Aspects can be combined to include two or more aspects described herein.

[0073] According to one aspect, a computer-implemented method includes: displaying a first group of one or more video streams inside an operating room on a first panel of an operating room monitor; displaying a second group of one or more video streams outside the operating room on a second panel of the operating room monitor; detecting a selection of one or more video streams in the second group as a selected video stream; and removing a current video stream displayed in a main portion of the operating room monitor as a removed video stream based on the detected selection, and moving the selected video stream from the second panel to the main portion of the operating room monitor based on the detected selection.

[0074] According to one aspect, the first group of one or more video streams may include at least one surgeon video stream and at least one endoscopy video stream.

[0075] According to one aspect, the method may include moving the removed video stream to the first panel based on determining that the removed video stream is associated with the first group of one or more video streams inside the operating room.

[0076] According to one aspect, the method may include moving the removed video stream to the second panel based on determining that the removed video stream is associated with the second group of one or more video streams inside the operating room.

[0077] According to one aspect, the method may include: detecting a next selection of one or more video streams inside the operating room as the selected video stream; and moving the selected video stream from the first panel to the main portion of the operating room display based on the detection of the next selection.

[0078] According to one aspect, the method may include opening the second panel based on detecting a first user selection to view one or more participants through a user interface.

[0079] According to one aspect, the method may include closing the second panel based on detecting that a second user has chosen to stop viewing one or more participants through the user interface.

[0080] According to one aspect, the method may include opening the first panel based on detecting the first user's selection to view one or more participants through the user interface.

[0081] According to one aspect, the method may include: wherein the second panel includes a full mute control that disables audio feeding to one or more participants of the second panel through the user interface.

[0082] According to one aspect, the method may include: wherein selecting one of the video streams in the second group includes detecting a first selection of an extended command menu and a second selection of a fixed control on the extended command menu.

[0083] According to one aspect, a computer program product may include a memory device storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations including: displaying two or more video streams, the two or more video streams including at least one video stream inside an operating room on a panel of a display and at least one video stream outside the operating room on a panel of the display. These operations may further include detecting a selection of one of the two or more video streams as the selected video stream. These operations may further include moving a current video stream, which will be displayed in a main portion of the display, to the panel based on the detection of the selection, and maintaining the position of the surgeon's video stream fixed at the same location on the panel before and after moving the current video stream from the main portion of the display to the panel. These operations may additionally include moving the selected video stream from the panel to the main portion of the display based on the detection of the selection.

[0084] According to one aspect, the at least one video stream inside the operating room may include at least one endoscopic video stream accessible by the one or more processors.

[0085] According to one aspect, the operation performed by the one or more processors may include holding the at least one endoscopic video stream on the panel at a location on the panel that has a higher viewing priority than the at least one video stream outside the operating room.

[0086] According to one aspect, the operations performed by the one or more processors may include adjusting the relative positions of one or more video streams of at least one video stream outside the operating room on the panel based on one or more placement rules.

[0087] According to one aspect, this selection can be individually controlled by each user through interaction with the display by the one or more processors performing these operations.

[0088] According to one aspect, the operations performed by the one or more processors may include: detecting the selection of a point of interest by one of a plurality of users through at least one user interface; and generating an indication of the point of interest associated with at least one video stream of the display to indicate the occurrence of an event captured in at least one of the video streams.

[0089] According to one aspect, a system includes an operating room monitor, a memory system, and a processing system coupled to the memory system and the operating room monitor. The processing system is configured to execute multiple instructions to: display one or more video streams from inside the operating room on a first panel of the operating room monitor; display one or more video streams from outside the operating room on a second panel of the operating room monitor; detect a selection of one or more video streams from the first or second group as a selected video stream; and based on the detected selection, move the selected video stream to a main portion of the operating room monitor.

[0090] According to one aspect, the first panel can be arranged horizontally on the operating room monitor, while the second panel is arranged vertically on the operating room monitor.

[0091] According to one aspect, the first panel and the second panel can be opened based on the detection of user selections made by one or more participants through the user interface.

[0092] According to one aspect, the selection of one or more video streams in the first group may include a single user action, while the selection of one or more video streams in the second group may include at least two user actions.

[0093] This invention can be a system, method, and / or computer program product at any possible level of integration technical detail. A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon to cause a processor to execute aspects of the invention.

[0094] Computer-readable storage media can be tangible devices that can retain and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanical encoding devices (such as punched cards or raised structures in grooves on which instructions are recorded), and any suitable combination of the foregoing. The computer-readable storage media used herein should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses transmitted through fiber optic cables), or electrical signals transmitted through wires.

[0095] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. Multiple wireless networks may include, but are not limited to, fifth-generation (5G) and sixth-generation (6G) protocols and connections. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to store them in a computer-readable storage medium within the corresponding computing / processing device.

[0096] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, high-level languages ​​such as Python, and procedural programming languages ​​such as "C" or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet, using an Internet service provider). In some aspects, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may be personalized to execute computer-readable program instructions by utilizing the status information of the computer-readable program instructions in order to perform aspects of this invention.

[0097] This document describes aspects of the invention with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to various aspects of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0098] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a manner for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0099] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device, thereby producing a computer-implemented process, such that the instructions, which are executed on the computer, other programmable apparatus or other device, implement the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0100] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various aspects of the invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a particular logical function(s). In some alternative implementations, the functions indicated in the blocks may occur in a non-linear order as shown in the drawings. For example, depending on the function involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It will also be noted that each block in the block diagram and / or flowchart illustration, and combinations of blocks in the block diagram and / or flowchart illustration, can be implemented by a dedicated hardware-based system that performs a specified function or action or executes a combination of dedicated hardware and computer instructions.

[0101] The description of various aspects of the invention is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed aspects. Various modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described aspects. The terminology used herein has been chosen to best explain the principles, practical applications, or technical improvements to existing technologies, or to enable those skilled in the art to understand the aspects described herein.

[0102] Various aspects of the invention have been described herein with reference to the accompanying drawings. Alternative aspects of the invention may be devised without departing from its scope. Various connections and positional relationships (e.g., above, below, adjacent, etc.) between elements are illustrated in the following description and drawings. Unless otherwise stated, these connections and / or positional relationships may be direct or indirect, and the invention is not intended to be limited in this respect. Accordingly, coupling of entities may refer to direct or indirect coupling, and positional relationships between entities may be direct or indirect positional relationships. Furthermore, the various tasks and process steps described herein may be incorporated into a more comprehensive procedure or process with additional steps or functions not described in detail herein.

[0103] The following definitions and abbreviations are used to interpret the claims and specification. As used herein, the terms “comprises,” “has,” or “contains,” or any other variation thereof, are intended to cover non-exclusive inclusion. For example, a composition, mixture, process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such a composition, mixture, process, method, article, or apparatus.

[0104] Additionally, the term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect or design described herein as "exemplary" is not necessarily to be construed as being more preferred or advantageous than other aspects or designs. The terms "at least one" and "one or more" can be understood to include any integer greater than or equal to one, i.e., one, two, three, four, etc. The term "multiple" can be understood to include any integer greater than or equal to two, i.e., two, three, four, five, etc. The term "connection" can include both indirect "connection" and direct "connection."

[0105] The terms “about,” “substantially,” “approximately,” and their variations are intended to include the degree of error associated with measuring a particular quantity based on the equipment available at the time of application. For example, “about” could include a range of ±8%, 5%, or 2% of a given value.

[0106] For the sake of brevity, conventional techniques related to the manufacture and use of various aspects of the present invention may or may not be described in detail herein. In particular, various aspects of the computing systems and specific computer programs used to implement the various technical features described herein are well known. Accordingly, for the sake of brevity, many details of conventional implementation methods are only briefly mentioned or omitted entirely herein, without providing well-known system and / or process details.

[0107] It should be understood that the various aspects disclosed herein can be combined in ways different from those specifically presented in the specification and figures. It should also be understood that, by way of example, certain actions or events of any process or method described herein may be performed in a different order, or may be added, combined, or omitted entirely (e.g., not all described actions or events may be necessary for implementing these techniques). Furthermore, although some aspects of this disclosure are described for clarity as being performed by a single module or unit, it should be understood that the techniques of this disclosure can be performed by a combination of units or modules associated with, for example, a medical device.

[0108] In one or more examples, the described techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include non-transitory computer-readable media, which corresponds to tangible media such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer).

[0109] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), graphics processing units (GPUs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term "processor" as used herein can refer to any of the above-described structures or any other physical structures suitable for implementing the described techniques. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

Claims

1. A computer-implemented method, comprising: The first group of one or more video streams inside the operating room are displayed on the first panel of the operating room monitor; A second set of one or more video streams from outside the operating room are displayed on the second panel of the operating room monitor. The detection selects one or more video streams from the second group as the selected video stream; as well as Based on the detection of the selection, the current video stream displayed in the main part of the operating room monitor is removed as a removed video stream, and based on the detection of the selection, the selected video stream is moved from the second panel to the main part of the operating room monitor.

2. The method as described in claim 1, wherein, The first group of one or more video streams includes at least one surgeon video stream and at least one endoscopy video stream.

3. The method of claim 1 or claim 2, further comprising: Based on the determination that the removed video stream is associated with one or more video streams in the first group inside the operating room, the removed video stream is moved to the first panel.

4. The method according to any of the preceding claims, further comprising: Based on the determination that the removed video stream is associated with one or more video streams in the second group inside the operating room, the removed video stream is moved to the second panel.

5. The method according to any of the preceding claims, further comprising: The next selection of one or more video streams inside the operating room is determined by the detection mechanism. as well as Based on the detection of the next selection, the selected video stream is moved from the first panel to the main part of the operating room monitor.

6. The method of any of the preceding claims, further comprising: The second panel is opened based on the detection of a first user's selection to view one or more participants through the user interface.

7. The method of claim 6, further comprising: The second panel is closed based on the detection that a second user has stopped viewing one or more participants through the user interface.

8. The method of claim 6 or claim 7, further comprising: The first panel is opened based on the detection of the first user's selection to view one or more participants through the user interface.

9. The method as described in any of the preceding claims, wherein, The second panel includes a full mute control that disables audio feeds to one or more participants in the second panel via the user interface.

10. The method as claimed in any of the preceding claims, wherein, The selection of one or more video streams in the second group includes detecting a first selection to the extended command menu and a second selection to the fixed controls on the extended command menu.

11. A computer program product comprising a memory device storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations including: Displaying two or more video streams, including at least one video stream inside the operating room on the panel of the display and at least one video stream outside the operating room on the panel of the display; The detection selects one of the two or more video streams as the selected video stream; Based on the detection of the selection, the current video stream, which will be displayed in the main part of the display, is moved to the panel, and the surgeon's video stream position is kept fixed at the same location on the panel before and after moving the current video stream from the main part of the display to the panel; and Based on the detection of this selection, the selected video stream is moved from the panel to the main part of the display.

12. The computer program product of claim 11, wherein, The at least one video stream inside the operating room includes at least one endoscopic video stream.

13. The computer program product of claim 12, wherein, These operations further include: The at least one endoscopic video stream is held on the panel at a position on the panel that has a higher viewing priority than the at least one video stream outside the operating room.

14. The computer program product as claimed in claim 13, wherein, These operations further include: The relative positions of one or more video streams from at least one video stream outside the operating room on the panel are adjusted based on one or more placement rules.

15. The computer program product as claimed in any one of claims 11 to 14, wherein, This selection is controlled individually by each user through interaction with the display.

16. The computer program product of claim 15, wherein, These operations further include: Detecting the selection of points of interest by one of multiple users through at least one user interface; and Generate an indication of the point of interest associated with at least one video stream of the display to indicate the occurrence of an event captured in at least one of these video streams.

17. A system comprising: Operating room monitor; Memory system; as well as A processing system, coupled to the memory system and the operating room display, is configured to execute multiple instructions to perform the following operations: The first panel of the operating room monitor displays one or more video streams from inside the operating room. A second set of one or more video streams from outside the operating room are displayed on the second panel of the operating room monitor. The selection of one or more video streams from the first group or the second group is taken as the selected video stream. as well as Based on the detection of this selection, the selected video stream is moved to the main part of the operating room monitor.

18. The system of claim 17, wherein, The first panel is arranged horizontally on the operating room monitor, while the second panel is arranged vertically on the operating room monitor.

19. The system of claim 17 or claim 18, wherein, The first panel and the second panel are opened based on the detection of user selections made by one or more participants through the user interface.

20. The system as claimed in any one of claims 17 to 19, wherein, The selection of one or more video streams in the first group includes a single user action, while the selection of one or more video streams in the second group includes at least two user actions.

Images