User interface for structures detected during surgery

Abstract

Technical solutions are provided for facilitating computer-aided prevention of complications during a surgical video by detecting, identifying, and indicating identifications of particular structures in a field of view of the surgical video. According to some aspects, a computer vision system is trained to detect structures in a video of the procedure, and further differentiate the structures. Further, a user interface element is displayed that indicates an identification of the structure by updating a visual attribute of the user interface element to match a visual attribute for representing the indicated structure itself.

Description

Background Art

[0001] The present disclosure relates generally to computing techniques, and more particularly to computing techniques that use machine learning to automatically detect structures during surgery and provide user feedback based on the automatic detection.

[0002] Computer-assisted systems can be used to enhance a person's physical senses, perceptions, and responsiveness. For example, such a system can effectively provide information corresponding to a field of view that is extended in time and space, which enables a person to adjust current and future actions and decisions based on portions of the environment that are not included in his or her physical field of view. In addition, the system can draw attention to portions of the view that are obscured, for example, by structures, blood, etc. However, providing such information relies on the ability to process a portion of the extended view in a useful manner. Highly variable, dynamic, and / or unpredictable environments present challenges in defining rules that dictate how representations of the environment are to be processed to output data to effectively assist a person in performing actions. Summary of the invention

[0003] According to one or more aspects, a computer-implemented user interface includes multiple user interface elements, each user interface element corresponds to a structure in a list of expected structures in a surgical video, and each user interface element has a visual attribute, wherein, in response to detecting that a first structure is in a field of view of the surgical video, the visual attribute of the first user interface element is set to a first state, and in response to detecting that the first structure is not in the field of view, the visual attribute of the first user interface element is set to a second state.

[0004] According to one aspect, the visual attribute of the first user interface element is based on marking the first structure with an overlay using the visual attribute.

[0005] According to one aspect, it is detected using machine learning that the first structure is in the field of view.

[0006] According to one aspect, the plurality of user interface elements are grouped in a single menu user interface. According to one aspect, the menu user interface is a toolbar displayed as a graphical overlay on the surgical video. According to one aspect, the surgical video is a real-time video stream. According to one aspect, the position of the rendering toolbar is fixed or user configurable.

[0007] According to one aspect, the structure list includes a predetermined structure list based on a type of surgery in the surgical video.

[0008] According to one aspect, the structure list comprises a dynamic structure list.

[0009] According to one aspect, the visual attribute is one of color, pattern, shape, image, and animation.

[0010] According to one aspect, the user interface element includes a label.

[0011] According to one or more aspects, a computer-implemented method includes: using machine learning by one or more processors to identify a structure in a video of a surgical procedure. The method further includes: in response to at least a portion of the structure being visible in a field of view, using a first visual attribute by the one or more processors to represent a user interface element corresponding to the structure. The method further includes: in response to the structure not being visible in the field of view, using a second visual attribute by the one or more processors to represent the user interface element corresponding to the structure.

[0012] According to one aspect, the structure is one of an anatomical structure and a surgical instrument.

[0013] According to one aspect, the anatomical structure is one of an organ, an artery, a tube, a surgical item, and an anatomical landmark.

[0014] According to one aspect, the surgical instrument is one of a clamp, a stapler, a knife, a scalpel, a sealer, a divider, a dissector, a tissue fusion instrument, a monopolar instrument, a Maryland instrument, and a fenestrated instrument.

[0015] According to one aspect, the structure is one of a list of predetermined structures.

[0016] According to one aspect, the symbol comprises a geometric shape that is displayed at a predetermined location during the video display of the surgical procedure.

[0017] According to one aspect, the first visual attribute is used to represent the symbol based on the first visual attribute for highlighting the structure in response to the structure being in view.

[0018] According to one aspect, the video is a real-time video stream of the surgical procedure.

[0019] According to one or more aspects, a computer program product includes a memory device having computer executable instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform a method for generating a user interface to depict the presence of a structure in a field of view. The method includes: using a neural network model to identify a structure in a video of a surgical procedure, the neural network model trained using surgical training data. The method further includes generating a visualization, the visualization including a graphical overlay at the location of the structure in the video of the surgical procedure, the graphical overlay using a first visual attribute. The method further includes identifying a symbol corresponding to the structure from a list of display symbols. The method further includes updating the symbol by displaying the symbol using the first visual attribute.

[0020] Additional technical features and benefits are achieved through the technology of the present invention. Various aspects of the present invention are described in detail herein and are considered part of the subject matter claimed for protection. For a better understanding, please refer to the detailed description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The details of the exclusive rights described herein are particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features and advantages of various aspects of the present invention will be apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0022] Figure 1 An example snapshot of a laparoscopic cholecystectomy being performed is shown;

[0023] Figure 2 A system for detecting structures in surgical data using machine learning according to one or more aspects is shown;

[0024] Figure 3 depicts a flow chart of a method for generating a user interface according to one or more aspects;

[0025] Figure 4 Described is a computer system according to one or more aspects; and

[0026] Figure 5 Surgical systems according to one or more aspects are described.

[0027] The figures depicted herein are illustrative. Without departing from the spirit of the present invention, there may be many variations in these figures or the operations described therein. For example, actions may be performed in different orders, or actions may be added, deleted, or modified. In addition, the term "coupling" and its variations describe having a communication path between two elements, and do not imply a direct connection without intermediate elements / connections between elements. All of these variations are considered part of this specification. DETAILED DESCRIPTION

[0028] Exemplary aspects of the technical solutions described herein relate, among other things, to devices, systems, methods, computer-readable media, techniques, and methodologies for improving surgical safety and workflow by automatically detecting one or more structures in surgical data using machine learning and computer vision. In some aspects, these structures can be considered critical to actors (e.g., by surgeons) involved in performing one or more actions during a surgical procedure. In one or more aspects, these structures are detected dynamically and in real time when surgical data is captured by the technical solutions described herein. The detected structures can be anatomical structures, surgical instruments, etc. Further, various aspects of the technical solutions described herein address the technical challenges of distinguishing structures and indicating the identity of the identified structures to actors.

[0029] One or more specific surgical procedures (e.g., laparoscopic cholecystectomy) are used as examples to provide a description of the technical solutions herein. However, it should be understood that the technical solutions described herein are not limited to these types of surgical procedures. The technical solutions described herein are applicable to any other type of surgical procedure where identification of structures and clear depiction of the identified structures to the actor is helpful.

[0030] Laparoscopic cholecystectomy is a common procedure to remove the gallbladder. This involves exposure of one or more anatomical structures, such as the gallbladder, liver, cystic duct, cystic artery, etc. The procedure may include clipping and division of one or more structures and then removal of the gallbladder. Figure 1 An example snapshot 10 of a laparoscopic cholecystectomy is shown with the anatomy labeled. Figure 1 In the illustrated snapshot 10, the cystic duct 14 is labeled. As can be seen, it is difficult to distinguish anatomical structures by visual cues without context (such as viewing direction, etc.). Complications may occur when these structures are misidentified or confused with other nearby structures. For example, the cystic artery may be located near the cystic duct 14, and it may be difficult to distinguish them, especially without a thorough dissection.

[0031] In some examples, a computer-assisted surgery (CAS) system is provided that uses one or more machine learning models trained with surgical data to augment environmental data directly sensed by an actor (e.g., a surgeon) involved in performing one or more actions during a surgical procedure. This augmentation of perception and action can improve action accuracy, optimize ergonomics, improve action effectiveness, enhance patient safety, and improve the standard of surgical procedures.

[0032] The surgical data provided for training the machine learning model may include data captured during the surgical procedure as well as simulated data. The surgical data may include time-varying image data corresponding to the surgical environment (e.g., simulated / real video streams from several types of cameras). The surgical data may also include other types of data streams, such as audio, radio frequency identifiers (RFID), text, robotic sensors, other signals, etc. The machine learning model is trained to detect and recognize "structures" in the surgical data, including specific tools, anatomical objects, and actions performed during the simulated / real surgical phase. In one or more aspects, the machine learning model is trained to define one or more parameters of the model in order to learn how to transform new input data (data on which the model is not trained) to identify one or more structures. During training, one or more data streams are input to the model, which may be enhanced with data indicating structures in the data stream (as indicated by metadata and / or image segmentation data associated with the input data). The data used during training may also include a time series of one or more input data.

[0033] In one or more aspects, simulation data can be generated to include image data (e.g., which can include time-series image data or video data and can be generated at any sensitivity wavelength) associated with variable viewpoints, camera poses, lighting (e.g., intensity, hue, etc.), and / or motion of an imaged object (e.g., a tool). In some instances, multiple data sets can be generated—each corresponding to the same imaged virtual scene, but differing in viewpoint, camera pose, lighting, and / or motion of an imaged object, or differing in a modality used to sense, for example, a red-green-blue (RGB) image, or depth or temperature. In some instances, each of the multiple data sets corresponds to a different imaged virtual scene, and further differs in viewpoint, camera pose, lighting, and / or motion of an imaged object.

[0034] The machine learning model may include a fully convolutional network adaptation (FCN) and / or a conditional generative adversarial network model configured with one or more hyperparameters for performing image segmentation classification. For example, a machine learning model (e.g., a fully convolutional network adaptation) may be configured to perform supervised, self-supervised, or semi-supervised semantic segmentation in multiple categories-each category corresponds to a specific surgical instrument, anatomical body part (e.g., typically or in a specific state) and / or environment. Alternatively or additionally, a machine learning model (e.g., a conditional generative adversarial network model) may be configured to perform unsupervised domain adaptation to convert simulated images into semantic segmentation. In one or more aspects, the machine learning model uses the neural network architecture of DeepLabV3+ and ResNet101 encoders. It should be understood that other types of machine learning models or combinations thereof may be used in one or more aspects.

[0035] The trained machine learning model can then be used in real time to process one or more data streams (e.g., video streams, audio streams, RFID data, etc.). Processing can include detecting and characterizing one or more structures at various instants or over a period of time. The structure(s) can then be used to identify the presence, location, and / or use of one or more features. Alternatively or additionally, the structures can be used to identify stages within a workflow (e.g., represented via a surgical data structure), predict future stages within a workflow, etc.

[0036] Figure 2 A system 100 for detecting structures in surgical data using machine learning according to one or more aspects is shown. According to some aspects, the system 100 uses a data stream in the surgical data to identify a surgical state. The system 100 includes a surgical control system 105 that collects image data and coordinates outputs in response to detected structures and states. The surgical control system 105 may include one or more devices (e.g., one or more user devices and / or servers) located in and / or associated with a surgical operating room and / or a control center. The system 100 further includes a machine learning processing system 110 that uses a machine learning model to process surgical data to identify a surgical state (also referred to as a period or stage), which is used to identify a corresponding output. It will be understood that the machine learning processing system 110 may include one or more devices (e.g., one or more servers), each of which may be configured to include part or all of one or more of the depicted components of the machine learning processing system 110. In some instances, part or all of the machine learning processing system 110 is located in the cloud and / or remote from an operating room and / or physical location corresponding to part or all of the surgical control system 105. For example, the machine learning training system 125 may be a separate device (e.g., a server) that stores its output as one or more trained machine learning models 130 that are accessible by a model execution system 140 that is separate from the machine learning training system 125. In other words, in some aspects, the device that "trains" the model is separate from the device that "infers" (i.e., uses the trained model 130 to perform real-time processing of surgical data).

[0037] The machine learning processing system 110 includes a data generator 115 configured to generate simulated surgical data (such as a virtual image set) or record surgical data from an ongoing surgery to train a machine learning model. The data generator 115 can access (read / write) a data storage 120 with recorded data (including multiple images and / or multiple videos). The images and / or videos may include images and / or videos collected during one or more surgeries (e.g., one or more surgical procedures). For example, the images and / or videos may have been collected by a user device worn by a participant (e.g., a surgeon, a surgical nurse, anesthesiologist, etc.) during surgery and / or a non-wearable imaging device located in an operating room.

[0038] Each image and / or video included in the recorded data may be defined as a base image and may be associated with other data characterizing the associated surgery and / or rendering specification. For example, the other data may identify the type of surgery, the location of the surgery, one or more persons involved in performing the surgery, and / or the outcome of the surgery. Alternatively or in addition, the other data may indicate the surgical stage to which the image or video corresponds, the rendering specification to which the image or video corresponds, and / or the type of imaging device that captured the image or video (e.g., if the device is a wearable device, and / or the role of a specific person wearing the device, etc.). Further, the other data may include image segmentation data that identifies and / or characterizes one or more objects (e.g., tools, anatomical objects, etc.) depicted in the image or video. The representation may indicate the position, orientation, or posture of an object in the image. For example, the representation may indicate a set of pixels corresponding to an object and / or an object state resulting from past or current user operations.

[0039] The data generator 115 identifies one or more sets of rendering specifications for a virtual image set. Which rendering specifications are particularly fixed and / or variable are identified. Alternatively or additionally, the rendering specifications to be fixed (or variable) are predefined. Identification can be based on, for example, input from a client device, the distribution of one or more rendering specifications on a base image and / or video, and / or the distribution of one or more rendering specifications on other image data. For example, if a particular specification is substantially constant over a relatively large data set, the data generator 115 defines a fixed corresponding value for the specification. As another example, if the rendering specification values ​​from at least a predetermined amount of data span a range, the data generator 115 defines the rendering specification based on the range (e.g., spanning the range or spanning another range that is mathematically related to the distribution range of these values).

[0040] A set of rendering specifications can be defined to include discrete or continuous (finely quantized) values. A set of rendering specifications can be defined by a distribution, such that a particular value is selected by sampling from the distribution using a random or biased process.

[0041] One or more sets of rendering specifications may be defined independently or in a related manner. For example, if the data generator 115 identifies five values ​​for a first rendering specification and four values ​​for a second rendering specification, the one or more sets of rendering specifications may be defined to include combinations of twenty or fewer rendering specifications (e.g., if one of the second rendering specifications is used in conjunction with only an incomplete subset of the first rendering specification values, or vice versa). In some instances, different rendering specifications may be identified for different surgical stages and / or other metadata parameters (e.g., surgical type, surgical location, etc.).

[0042] Using the rendering specifications and the base image data, the data generator 115 generates simulated surgical data (e.g., a set of virtual images), which are stored in the data storage 120. For example, the base image data can be used to generate a three-dimensional model of the environment and / or one or more objects. Given a specific set of rendering specifications (e.g., background lighting intensity, viewing angle, zoom, etc.) and other surgery-related metadata (e.g., surgery type, surgical status, imaging device type, etc.), the model can be used to generate virtual image data to be determined. Generation can include, for example, performing one or more transformations, conversions, and / or scaling operations. Generation can further include adjusting the overall intensity of pixel values ​​and / or transforming RGB values ​​to achieve specific color-specific specifications.

[0043] The machine learning training system 125 trains one or more machine learning models using the recorded data in the data storage 120, which may include simulated surgical data (e.g., a virtual image set) and actual surgical data. The machine learning model can be defined based on a model type and a hyperparameter set (e.g., defined based on input from a client device). The machine learning model can be configured based on a parameter set, which can be dynamically defined based on (e.g., continuous or repeated) training (i.e., learning, parameter adjustment). The machine learning training system 125 can define the parameter set using one or more optimization algorithms to minimize or maximize one or more loss functions. The set of (learned) parameters can be stored in a trained machine learning model data structure 130, which can also include one or more non-learnable variables (e.g., hyperparameters and / or model definitions).

[0044] The model execution system 140 can access the machine learning model data structure 130 and configure the machine learning model for inference (i.e., detection) accordingly. The machine learning model can include, for example, a fully convolutional network adaptation, an adversarial network model, or other types of models indicated in the data structure 130. The machine learning model can be configured according to one or more hyperparameters and a learned parameter set.

[0045] The machine learning model receives as input the surgical data to be processed during execution and generates inferences based on the training. For example, the surgical data may include a data stream (e.g., an array of intensity, depth, and / or RGB values) for each frame in a single image or a set of frames that represent a fixed or variable length time window in the video. The input surgical data may be received from a real-time data collection system 145, which may include one or more devices located in the operating room and / or streaming real-time imaging data collected during the performance of the surgery. The surgical data may include additional data streams, such as audio data, RFID data, text data, measurements from one or more instruments / sensors, etc., which may represent the stimulus / surgical state in the operating room. Different inputs from different devices / sensors are synchronized before being input into the model.

[0046] The machine learning model analyzes the surgical data and, in one or more aspects, detects and / or characterizes structures included in visual data from the surgical data. The visual data may include images and / or video data in the surgical data. The detection and / or characterization of the structure may include segmenting the visual data or detecting the location of the structure with a probabilistic heat map. In some instances, the machine learning model includes or is associated with preprocessing or enhancement (e.g., intensity normalization, resizing, cropping, etc.) performed before segmenting the visual data. The output of the machine learning model may include image segmentation or probabilistic heat map data indicating which structures (if any) of a defined set of structures are detected within the visual data, the location and / or location and / or posture of (multiple) structures within the image data, and / or the state of (multiple) structures. The location may be a set of coordinates in the image data. For example, the coordinates may provide a bounding box. Alternatively, the coordinates provide a boundary around the detected (multiple) structures.

[0047] The state detector 150 can use the output from the execution of the machine learning model to identify the state within a surgical procedure ("operation"). The surgical tracking data structure can identify a set of potential states that may correspond to a portion of the execution of a particular type of surgery. Different surgical data structures (e.g., different machine learning model parameters and / or hyperparameters) can be associated with different types of surgeries. The data structure may include a set of nodes, each corresponding to a potential state. The data structure may include directed connections between nodes, which indicate (via directions) the expected order in which states will be encountered throughout the program iteration. The data structure may include one or more branch nodes that feed into multiple next nodes, and / or may include one or more divergence points and / or convergence points between nodes. In some instances, the surgical state indicates a surgical action being performed or that has been performed and / or indicates a combination of actions that have been performed. "Surgical actions" may include operations such as incision, compression, anastomosis, clipping, suturing, cauterization, sealing, or any other such actions performed to complete a step / stage in a surgical procedure. In some instances, the surgical state is related to the biological state of a patient undergoing surgery. For example, the biological state may indicate a complication (eg, blood clot, arterial / venous obstruction, etc.), or a pre-existing condition (eg, lesion, polyp, etc.).

[0048] Each node within the data structure can identify one or more characteristics of a state. These characteristics can include visual characteristics. In some instances, the node identifies one or more tools that are typically in use or available for use (e.g., trying on a tool) during the state, one or more roles of a person who is typically performing a surgical task, typical types of movement (e.g., movement of a hand or tool), etc. Therefore, the state detector 150 can use the segmented data generated by the model execution system 140 indicating the presence and / or characteristics of a specific object in the field of view to identify the estimated node corresponding to the real image data. The identification of the node (and / or state) can be further based on the previously detected state of a given program iteration and / or other detected input (e.g., verbal audio data including requests or comments from one person to another, explicit identification of the current or past state, information requests, etc.).

[0049] Output generator 160 can use the state to generate output. Output generator 160 can include an alarm generator 165, which generates and / or obtains information associated with the state and / or potential next events. For example, the information can include details about warnings and / or suggestions corresponding to current or expected surgical actions. The information can further include one or more events to be monitored. The information can identify the next recommended action.

[0050] The user feedback may be transmitted to an alarm output system 170, which may output the user feedback via a user device and / or other device located, for example, in an operating room or control center. The user feedback may include visual, auditory, tactile, or haptic outputs of the indicative information. The user feedback may facilitate the issuance of an alarm to an operator (e.g., a surgeon or any other user of the system).

[0051] The output generator 160 may also include an enhancer 175 that generates or obtains one or more graphics and / or text to be visually presented on or near (e.g., overlaid on) the real-time capture of the surgery (e.g., presented below, near, or on a separate screen). The enhancer 175 may further identify the location where the graphics and / or text are to be presented (e.g., within a specified size of the display). In some instances, a defined portion of the field of view is designated as the portion of the display that includes the enhanced data. In some instances, the location of the graphics and / or text is defined so as not to obscure the view of important portions of the surgical environment and / or to overlay certain graphics (e.g., graphics of tools) with corresponding real-world representations.

[0052] The enhancer 175 may send graphics and / or text and / or any positioning information to an augmented reality device 180, which may integrate the graphics and / or text with the user's environment in real time as an augmented visualization. The augmented reality device 180 may include a pair of goggles that may be worn by a person participating in the portion of the surgery. It will be appreciated that in some instances, the augmented display may be presented on a non-wearable user device, such as on a computer or tablet computer. The augmented reality device 180 may present graphics and / or text at a location and / or predefined location identified by the enhancer 175. Thus, the user may maintain a real-time view of the surgical operation and further view relevant status related information.

[0053] Current existing solutions provide official guidance requiring the surgeon to establish a “critical safety view” (CVS) before performing clipping and segmentation. In the CVS, these two structures can be clearly and individually identified and tracked when entering the gallbladder. Some existing technologies create bounding box detection systems based on anatomical landmarks that include the common bile duct and the cystic duct14 but not the cystic artery. Existing solutions provide different techniques, including machine learning techniques for detecting and identifying one or more anatomical structures in an input image. However, there are technical challenges in conveying information about the identified structures to the actor.

[0054] In existing solutions, for example, using a graphical overlay (e.g. Figure 1Here, “marking” an anatomical structure, surgical instrument, or other feature in the surgical data includes visually highlighting the feature for the surgeon or any other user by using the graphical overlay 502. The graphical overlay 502 may include a heat map, outline, bounding box, mask, highlight, or any other such visualization overlaid on the image 10 displayed to the user. Further, in one or more aspects, the identified particular anatomical structure is marked using a predetermined value assigned to the corresponding anatomical structure. For example, Figure 1 As shown, the cystic duct 14 is marked with a first color value (e.g., green), and the cystic duct can be marked with a second color value (e.g., purple), etc. It will be appreciated that visual attributes other than color or a combination thereof can also be assigned to a particular structure. The assignment of visual attributes to respective structures can be user configurable. The examples herein depict the use of masks and heat maps as the graphic overlay 502. However, different techniques can be used in other aspects. Various visual attributes of the graphic overlay 502 can be adjusted, such as color, transparency, visual pattern, line thickness, etc.

[0055] To identify the type of structure identified, existing solutions display annotations in addition to the graphical overlay 502. An annotation may be a text label identifying the name of a structure (e.g., (multiple) anatomical structures or other objects). However, there is a technical challenge in that such annotations may not be visible because the location of the structure in the view may change. Further, such annotations may cover portions of the surgical data that the user may wish to see or whose visibility is critical to the user. In addition, when multiple structures are identified, the corresponding annotations of the structures may occupy an undesirable amount of the view. In addition, multiple annotations may overlap, making them visually difficult to discern and aesthetically unappealing.

[0056] The technical solution described herein provides a user interface that seeks to address such technical challenges and provides several technical improvements for surgical systems. The technical solution described herein provides a dynamic structure index, which includes a list of symbols corresponding to a list of structures, respectively. Visual attributes that match the visual attributes (e.g., color) of a graphic overlay 502 used to mark a structure corresponding to a symbol in the index are used to highlight the symbol. The symbol is highlighted only when the corresponding structure is in the field of view, identified and marked (e.g., using the graphic overlay 502). When the structure is not in the view, the symbol is not highlighted. Although various aspects of the technical solution herein are described as detecting that a "structure" is in the field of view (or out of the field of view), it should be noted that in some aspects, the entire structure may not be detected / identified. In some aspects, only a portion of the structure is in the field of view (or out of the field of view) to enable the symbol to be highlighted (or not highlighted). In one or more aspects, the identified portion is a predetermined portion. Therefore, there are no annotations and labels in the surgical field of view, and only structures (such as anatomical structures), surgical instruments, etc. remain visible to the actor in the surgical field of view. The index helps to identify the structure.

[0057] like Figure 1 As described, aspects described herein facilitate generating a computer-implemented user interface 500. In some aspects, the user interface 500 can be a collection of toolbars, menu bars, or any other such user interface elements (510). In other aspects, the user interface 500 can be a single user interface element (510), and multiple such user interfaces 500 are used. Figure 1 Several types of user interface elements (UIEs) 510 are depicted (a first UIE type 510A, a second UIE type 510B, and a third UIE type 510C) which are collectively referred to as UIEs 510.

[0058] In some aspects, each user interface element 510 includes a symbol 515 that corresponds to a specific structure that can be identified in the video of the surgical procedure. In some aspects, the UIE 510 may also include a note 516 (a text name / description of the corresponding structure). In some aspects, other components may be included in the UIE 510, not shown in this article.

[0059] In some aspects, UIEs 510 of specific types (i.e., corresponding to structures of specific types) are grouped together. For example, a first UIE type 510A may represent an anatomical structure detected and identified in a video; 510B may represent a surgical instrument detected and identified in a video; and 510C may represent an energy platform (e.g., Valleylab TMFT10 energy platform). In other aspects, the grouping and sequence of UIE 510 can be similar to Figure 1 The groupings and sequences shown are different.

[0060] In some aspects, in response to detecting that a first structure (e.g., the cystic duct 14) is in the field of view in the surgical video, the visual attribute of the first UIE 510 (e.g., 511) is set to a first state / value. In response to detecting that the first structure is not in the field of view, the visual attribute of the first UIE 511 is set to a second state / value. Thus, when the corresponding structure is in the field of view, the UIE 511 is highlighted, and when the corresponding structure is not in the field of view, it is not highlighted. Highlighting (or not highlighting) can be achieved by changing / updating the visual attributes of the symbol 515 and / or the annotation 516.

[0061] In some aspects, the visual attributes used to highlight UIE 511 are based on the visual attributes of the graphic overlay 502 used to depict the first structure (cystic duct 14); that is, the graphic overlay 502 and UIE 511 are represented using the same (multiple) visual attributes. Examples of visual attributes that are updated to depict the identification of the structure in the field of view are color (e.g., foreground, background, border, etc.), pattern, shape, size, icon, image, and other such attributes and combinations thereof. In one or more aspects, UIE 511 includes attributes that facilitate the animation of UIE 511. For example, UIE 511 can depict flashing colors, lights, or any other visual cues to attract the user's attention. Additionally or alternatively, in other aspects, another sensory cue can be provided to the user along with the visual attributes of UIE511. For example, an audible cue can be issued, such as a notification tone. In some aspects, tactile feedback can be provided. Tactile feedback can be provided via the surgical instrument being used.

[0062] In some aspects, user interface 500 is depicted at a predetermined fixed position on a display. The position can be based on a predetermined configuration of the display, which indicates the position of one or more components such as user interface 500, video playback area, and other components (e.g., menus, etc., not shown). In some aspects, the position of user interface 500 can be dynamically updated by the user. For example, the user can move user interface 500.

[0063] In some aspects, based on the type of surgery being performed, the user interface 500 is populated with UIE 510. Based on the type of surgery, specific structures that can be predictably detected in the surgical video of the surgery by using machine learning are listed to be identified in the user interface 500. Alternatively or additionally, the list of structures to be identified and indicated can be a list of key structures associated with the type of surgery. Alternatively or additionally, the user can provide a list of structures to be identified through the user interface 500. Based on the list of structures for which identification is to be indicated, the user interface 500 is populated with corresponding UIE 510.

[0064] The structures to be identified and indicated by the user interface 500 may include anatomical structures, surgical instruments, etc. The identification of the structures is performed automatically by using one or more machine learning models that may be known or yet to be developed.

[0065] Figure 3 A flow chart of a method 200 for generating and displaying a user interface indicating identification of one or more structures in a field of view in a surgical video is depicted according to one or more aspects. The method 200 may be performed by the system 100 as a computer-implemented method.

[0066] The method 200 includes training and using (inference phase) (multiple) machine learning models 130 at box 202 to detect structures in a surgical video. In some examples, the surgical video can be a real-time video stream. In other aspects, the surgical video can be a playback of a recorded video. Artificial deep neural networks (DNNs) or other types of machine learning models can be used to achieve automatic, accurate stage structure detection and identification in surgical procedures (such as cataract surgery, laparoscopic cholecystectomy, endoscopic endonasal transsphenoidal approach (eTSA) pituitary adenoma resection, or any other surgical procedure).

[0067] The (multiple) machine learning models 130 include a feature encoder for detecting features from surgical data of a surgery. The feature encoder can be based on one or more artificial neural networks, such as a convolutional neural network (CNN), a recurrent neural network (RNN), a feature pyramid network (FPN), a transformer network, or any other type of neural network or a combination thereof. The feature encoder can use known supervised, self-supervised, or unsupervised (e.g., autoencoder) techniques to learn efficient data "encodings" in the surgical data. The "encoding" maps the input data to a feature space that can be used by a feature decoder to perform semantic analysis of the surgical data. In one or more aspects, the machine learning model includes a task-specific decoder that detects the instrument being used in an instance in the surgical data based on the detected features.

[0068] The detected structures may include anatomical structures, surgical instruments, and other such features in the surgical features. The detected anatomical structures may include organs, arteries, tubes, implants, surgical supplies (e.g., staples, sutures, etc.), etc. Further, based on the type of surgical procedure being performed, one or more of the detected anatomical structures may be identified as key structures for the success of the surgery. The detected surgical instruments may include clamps, staples, knives, scalpels, sealers, dividers, dissectors, tissue fusion instruments, monopolar instruments, Maryland instruments, and perforated instruments, etc.

[0069] The machine learning model 130 can detect and identify whether a particular structure is in (or not in) the field of view. The machine learning model 130 can further indicate (in the input image(s)) where the structure is detected and identified.

[0070] Figure 3 The method 200 further includes generating an enhanced visualization of the surgical video using the information obtained by the processing at block 204. The enhanced visualization may include, for example, displaying a graphical overlay 502 on one or more identified structures in the surgical video. The graphical overlay 502 may represent a segmentation mask or a probability map, etc. Figure 1 Depicted are example enhanced visualizations of surgical views generated in accordance with one or more aspects.) It should be understood that those shown are examples and that various other enhanced visualizations may be generated in other aspects.

[0071] Further, depending on the stage of the surgical procedure, the critical anatomical structures identified may also change. For example, in eye surgery, the iris and the specific portion of the iris to be operated on may be the only structures to be identified using the graphic overlay 502. Likewise, the sclera, which is visible (i.e., in view), may not be marked, for example, because it may not be considered a "critical structure" for the surgical procedure or stage of surgery being performed.

[0072] The user may configure which detections from the machine learning system 100 are to be displayed by the augmentor 175. For example, the user may configure the overlay 502 to be displayed on a partial set of markers while other markers are not marked in the augmented reality device 180.

[0073] "Critical anatomical structures" may be specific to the type of surgical procedure being performed and automatically identified. Additionally, the surgeon or any other user may configure the system 100 to identify specific anatomical structures that are critical to a particular patient. The selected anatomical structures may be critical to the success of the surgical procedure, such as anatomical landmarks that need to be identified during surgery (e.g., Carlo's triangle, His angle, cystic artery 12, cystic duct 14, etc.) or anatomical landmarks that result from a previous surgical task or procedure (e.g., anastomosed or sutured tissue, clips, etc.).

[0074] In some aspects, surgical instruments in the surgical video can also be marked using a graphic overlay 502. As described herein, surgical instruments are identified by a machine learning model.

[0075] In one or more aspects, a user can adjust properties of the graphical overlay 502. For example, a user can select a type of highlighting, color, line thickness, transparency, shading pattern, label, outline, or any other such property for generating and displaying the graphical overlay on the surgical video. In some aspects, the color and / or transparency of the graphical overlay 502 is adjusted based on a confidence score associated with the machine learning model(s)' recognition of the underlying anatomical structure or surgical instrument.

[0076] In some aspects, in response to a structure visible in the field of view being identified as a structure to be indicated, the UIE 510 corresponding to the structure is highlighted at box 206. The highlighting is performed using the visual attributes of the graphic overlay 502 used to depict the structure. In response to the structure no longer being in the field of view (or not in the field of view), the corresponding UIE 510 is updated to eliminate the highlighting (or just displayed without highlighting). When the UIE 510 is displayed using the default settings / values ​​of one or more visual attributes of the UIE 510, it is just depicted / displayed without highlighting. In some aspects, the setting of the visual attributes for depicting the UIE 510 in a highlighted state can be referred to as a first state / first visual attribute setting, etc., and the setting of the visual attributes for depicting the UIE 510 in a non-highlighted state can be referred to as a second state / second visual attribute setting, etc.

[0077] In some aspects, two or more of the UIEs 510 may be highlighted simultaneously. For example, if two or more structures are in view, each respective UIE 510 is highlighted. Figure 6 Another exemplary user interface 500 is depicted. The UIE 510 shown uses Figure 1 It should be understood that the examples depicted herein are illustrative, and in one or more aspects, UIE 510 may use shapes, forms, and attributes different from those depicted herein. For example, Figure 6 As shown, view 610 includes user interface 500, wherein each UIE 511 includes an ellipse symbol 515 and an annotation 516. The fill color (or pattern, etc.) of symbol 515 is used as a visual attribute for the graphic overlay 502 depicting the corresponding structure (e.g., prostate, seminal vesicle, bladder, etc.). In other aspects, another attribute may be used.

[0078] View 620 depicts another example of user interface 500. In this example, UIE 511 includes annotation 516, but does not include symbol 515 (as in other examples). Instead, the visual attributes of UIE 511 itself are used to represent the graphic overlay 502 corresponding to the structure associated with UIE 511. For example, here, the fill color (or fill pattern, etc.) of UIE 511 is used to represent the graphic overlay 502. (For example, the graphic overlay 502 has the same color as the fill color). It should be understood that the description herein is not limiting, and other attributes of UIE 511 can be used to indicate the relationship with the corresponding (multiple) graphic overlays 502.

[0079] Various aspects of the technical solutions described herein improve surgical procedures by increasing the safety of surgery. Further, the technical solutions described herein contribute to improving computing technology, particularly computing technology used during surgical procedures. Various aspects of the technical solutions described herein contribute to one or more machine learning models (such as computer vision models) using spatiotemporal information to process images obtained from a real-time video feed of a surgical procedure in real time. The machine learning model uses information from a real-time video feed and (if available) a robotic sensor platform using techniques such as neural networks to detect and distinguish one or more features (such as anatomical structures, surgical instruments) in an input window of a real-time video feed, and further depicts predictions / identifications to the user in a non-invasive, informative, intuitive, configurable, and aesthetically pleasing manner. It should be noted that the output of a machine learning model may be referred to as a "prediction" unless otherwise specified.

[0080] A graphical overlay is generated and displayed for the surgeon and / or other user in an enhanced visualization of the surgical view using the predictions. The graphical overlay may mark key anatomical structures, surgical instruments, surgical staples, scar tissue, results of previous surgical actions, etc. The graphical overlay may further illustrate the relationship between the surgical instrument(s) in the surgical view and one or more anatomical structures, and thus guide the surgeon and other users during surgery. The graphical overlay is adjusted according to the user's preferences and / or according to the confidence score of the prediction.

[0081] By using machine learning models and computational techniques to predict and mark various features in surgical views in real time, aspects of the technical solution help surgeons replace visualization based on external contrast agents (e.g., indocyanine green (ICG), ethanethiol, etc.) that must be injected into the patient. Such contrast agents may not always be available due to patient prerequisites or other factors. Accordingly, aspects of the technical solution described herein provide practical applications in surgical procedures. In some aspects, in addition to the technical solutions described herein, contrast agents can also be used. An operator (e.g., a surgeon) can turn on / off one (or both) of the contrast agent-based visualization or the graphic overlay 502.

[0082] Furthermore, aspects of the technical solutions described herein address the technical challenges of predicting complex features in a live video feed of a surgical view in real time. These technical challenges are addressed by using real-time analysis and enhanced visualization of the surgical view.

[0083] Now go to Figure 4 , according to one aspect, a computer system 800 is generally shown. As described herein, the computer system 800 may be an electronic computer framework including and / or employing any number of computing devices and networks utilizing various communication technologies and combinations thereof. The computer system 800 may be easily expanded, extended, and modularized, and may have the ability to change to different services or reconfigure some features independently of other features. The computer system 800 may be, for example, a server, a desktop computer, a laptop computer, a tablet computer, or a smart phone. In some examples, the computer system 800 may be a cloud computing node. The computer system 800 may be described in the general context of computer system executable instructions (such as program modules) executed by the computer system. Typically, a program module may include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. The computer system 800 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located in local and remote computer system storage media including memory storage devices.

[0084] like Figure 4As shown, the computer system 800 has one or more central processing units (CPUs) 801a, 801b, 801c, etc. (collectively or generally referred to as (multiple) processors 801). The processor 801 can be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The processor 801 is also referred to as a processing circuit, which is coupled to a system memory 803 and various other components via a system bus 802. The system memory 803 may include one or more memory devices, such as a read-only memory (ROM) 804 and a random access memory (RAM) 805. The ROM 804 is coupled to the system bus 802 and may include a basic input / output system (BIOS) that controls certain basic functions of the computer system 800. The RAM is a read-write memory coupled to the system bus 802 for use by the processor 801. The system memory 803 provides temporary storage space for the operation of the instructions during operation. The system memory 803 may include a random access memory (RAM), a read-only memory, a flash memory, or any other suitable memory system.

[0085] Computer system 800 includes an input / output (I / O) adapter 806 and a communication adapter 807 coupled to system bus 802. I / O adapter 806 may be a small computer system interface (SCSI) adapter that communicates with hard disk 808 and / or any other similar components. I / O adapter 806 and hard disk 808 are collectively referred to herein as mass storage device 810.

[0086] Software 811 for execution on computer system 800 may be stored in mass storage device 810. Mass storage device 810 is an example of a tangible storage medium readable by processor 801, wherein software 811 is stored as instructions executed by processor 801 to cause computer system 800 to operate as described below with reference to the various figures. Examples of computer program products and execution of such instructions are discussed in greater detail herein. Communications adapter 807 interconnects system bus 802 with network 812, which may be an external network that enables computer system 800 to communicate with other such systems. In one aspect, a portion of system memory 803 and mass storage device 810 together store an operating system, which may be a coordinating Figure 4 The various components shown function as any suitable operating system.

[0087] Additional input / output devices are shown connected to the system bus 802 via a display adapter 815 and an interface adapter 816. In one aspect, adapters 806, 807, 815, and 816 may be connected to one or more I / O buses that are connected to the system bus 802 via an intermediate bus bridge (not shown). A display 819 (e.g., a screen or display monitor) is connected to the system bus 802 by a display adapter 815, which may include a graphics controller and a video controller for improving the performance of graphics-intensive applications. A keyboard, mouse, touch screen, one or more buttons, speakers, and the like may be interconnected to the system bus 802 via an interface adapter 816, which may include, for example, a super I / O chip that integrates multiple device adapters into a single integrated circuit. Suitable I / O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols such as the Peripheral Component Interconnect (PCI). Thus, as shown in FIG. Figure 4 As configured, the computer system 800 includes processing capabilities in the form of a processor 801, storage capabilities including a system memory 803 and a mass storage device 810, input devices such as buttons and a touch screen, and output capabilities including a speaker 823 and a display 819.

[0088] In some aspects, the communication adapter 807 can use any suitable interface or protocol (such as Internet Small Computer System Interface, etc.) to transmit data. The network 812 can be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, etc. An external computing device can be connected to the computer system 800 through the network 812. In some examples, the external computing device can be an external network server or a cloud computing node.

[0089] It should be understood that Figure 4 The block diagram is not intended to indicate that the computer system 800 will include Figure 4 Instead, the computer system 800 may include Figure 4 800 (e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.). Further, the aspects described herein with respect to computer system 800 may be implemented with any appropriate logic, wherein, in various aspects, the logic as mentioned herein may include any appropriate hardware (e.g., a processor, an embedded controller, or an application specific integrated circuit, etc.), software (e.g., an application, etc.), firmware, or any appropriate combination of hardware, software, and firmware.

[0090] The reports / views / notes and other information described herein are added to the electronic medical record (EMR) of one or more cases. In some aspects, information about a particular surgical procedure can be stored in a patient record associated with a patient undergoing surgery during the surgical procedure. Alternatively or additionally, the information is stored in a separate database for later retrieval. The retrieval can be associated with the patient's unique identifier, such as an EMR identifier, a social security number, or any other unique identifier. The stored data can be used to generate patient-specific reports. In some aspects, information can also be retrieved from the EMR to enhance one or more operations described herein. In one or more aspects, surgical notes can be generated, which include one or more outputs from a machine learning model. The surgical notes can be stored as part of the EMR.

[0091] Figure 5 A surgical system 900 is depicted according to one or more aspects. Figure 5 The example depicts a surgical support system 902 configured to communicate with a surgical scheduling system 930 via a network 920. The surgical support system 902 may include or may be coupled to the system 100. The surgical support system 902 may use one or more cameras 904 to acquire image data, such as images. The surgical support system 902 may also interface with a plurality of sensors 906 and effectors 908. The sensor 906 may be associated with a surgical support device and / or patient monitoring. The effector 908 may be a robotic component or other device that may be controlled by the surgical support system 902. The surgical support system 902 may also interact with one or more user interfaces 910 (such as various input and / or output devices). While a surgical operation is being performed, the surgical support system 902 may store, access and / or update surgical data 914 associated with a training data set and / or real-time data. The surgical support system 902 may store, access and / or update surgical objectives 916 to help train and guide one or more surgical operations.

[0092] The surgical scheduling system 930 can access and / or modify scheduling data 932 for tracking planned surgical operations. Scheduling data 932 can be used to schedule physical resources and / or human resources to perform planned surgical operations. Based on the surgical manipulation predicted by one or more machine learning models and the current operation time, the surgical support system 902 can estimate the expected time for the end of the surgical operation. This can be based on previously observed similar complex cases recorded in the surgical data 914. Changes in the predicted end of the surgical operation can be used to notify the surgical scheduling system 930 to prepare the next patient, who can be identified in the record of the scheduling data 932. The surgical support system 902 can send an alarm to the surgical scheduling system 930, which triggers a scheduling update associated with a later surgical operation. Scheduling changes can be captured in the scheduling data 932. Predicting the end time of the surgical operation can improve the efficiency of running parallel sessions in the operating room because resources can be allocated between operating rooms. Based on the scheduling data 932 and the predicted surgical manipulation, a request to enter the operating room can be transmitted as one or more notifications 934.

[0093] As surgical maneuvers and steps are completed, progress can be tracked in surgical data 914 and status can be displayed via user interface 910. Status information can also be reported to other systems via notification 934 when a surgical maneuver is completed or any problems (such as complications) are observed.

[0094] The present invention may be a system, method and / or computer program product at any possible level of integrated technical detail. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon to cause a processor to perform various aspects of the present invention.

[0095] Computer readable storage medium can be a tangible device that can retain and store instructions for use by instruction execution devices. Computer readable storage medium can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage medium includes the following: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device (such as punch card or the convex structure in the groove with instructions recorded thereon) and any suitable combination of the above. Computer readable storage medium used herein should not be interpreted as instantaneous signal itself, such as radio waves or other free propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses transmitted by fiber optic cables), or electrical signals transmitted by wires.

[0096] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or downloaded to an external computer or external storage device. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0097] The computer-readable program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as "C" programming language or similar programming languages. The computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, and partially on a remote computer, or completely on a remote computer or server. In the latter scenario, a remote computer may be connected to a user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., via the Internet, using an Internet service provider). In some respects, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may be personalized electronic circuits to execute computer-readable program instructions using the state information of the computer-readable program instructions, so as to perform various aspects of the present invention.

[0098] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to various aspects of the present invention. It will be understood that each frame of the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions.

[0099] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a way to implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can instruct a computer, a programmable data processing device, and / or other equipment to operate in a specific manner, so that the computer-readable storage medium in which the instructions are stored includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0100] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operating steps to be performed on the computer, other programmable apparatus, or other device, thereby producing a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0101] The flowcharts and block diagrams in the accompanying drawings show the architecture, functions and operations of possible implementations of systems, methods and computer program products according to various aspects of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, fragment or part of an instruction, which includes one or more executable instructions for implementing (multiple) specific logical functions. In some alternative embodiments, the functions marked in the box may not occur in the order marked in the accompanying drawings. For example, depending on the functions involved, two boxes shown in succession can actually be executed simultaneously, or these boxes can sometimes be executed in reverse order. It will also be noted that each box of the block diagram and / or flowchart illustration and the combination of boxes in the block diagram and / or flowchart illustration can be implemented by a dedicated hardware-based system that performs a combination of a specified function or action or performs dedicated hardware and computer instructions.

[0102] The description of various aspects of the present invention is presented for the purpose of illustration, but is not intended to be exhaustive or limited to the disclosed aspects. Various modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described aspects. The terms used herein are selected to best explain the principles of various aspects, practical applications or technical improvements over existing technologies on the market, or to enable those of ordinary skill in the art to understand the various aspects described herein.

[0103] Various aspects of the present invention are described herein with reference to the accompanying drawings. Without departing from the scope of the present invention, alternative aspects of the present invention can be designed. Various connections and positional relationships (e.g., above, below, adjacent, etc.) between elements are set forth in the following description and the accompanying drawings. Unless otherwise indicated, these connections and / or positional relationships may be direct or indirect, and the present invention is not intended to be limited in this respect. Accordingly, the coupling of entities may refer to direct or indirect coupling, and the positional relationship between entities may be direct or indirect positional relationships. In addition, the various tasks and process steps described herein may be incorporated into a more comprehensive program or process with additional steps or functions not described in detail herein.

[0104] The following definitions and abbreviations are used to interpret the claims and specification. As used herein, the terms "comprises / comprising / includes / including", "has / having", or "contains / containing" or any other variation thereof are intended to cover a non-exclusive inclusion. For example, a composition, mixture, process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such composition, mixture, process, method, article, or apparatus.

[0105] Additionally, the term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs. The terms "at least one" and "one or more" may be understood to include any integer greater than or equal to one, i.e., one, two, three, four, etc. The term "plurality" may be understood to include any integer greater than or equal to two, i.e., two, three, four, five, etc. The term "connected" may include both indirect "connections" and direct "connections."

[0106] The terms "about," "substantially," "approximately," and variations thereof are intended to include the degree of error associated with measuring a particular quantity based on the equipment available at the time the application was filed. For example, "about" may include a range of ±8% or 5% or 2% of a given value.

[0107] For the sake of brevity, conventional techniques related to making and using various aspects of the present invention may or may not be described in detail herein. In particular, various aspects of computing systems and specific computer programs for implementing the various technical features described herein are well known. Accordingly, for the sake of brevity, many conventional implementation details are only briefly mentioned herein or omitted entirely, without providing well-known system and / or process details.

[0108] It should be understood that the various aspects disclosed herein may be combined in combinations different from those specifically presented in the specification and drawings. It should also be understood that, according to examples, certain actions or events of any process or method described herein may be performed in a different order, may be added, combined, or omitted entirely (e.g., all described actions or events may not be necessary to implement these techniques). In addition, although for clarity, certain aspects of the present disclosure are described as being performed by a single module or unit, it should be understood that the technology of the present disclosure may be performed by a combination of units or modules associated with, for example, a medical device.

[0109] In one or more examples, the described techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include non-transitory computer-readable media, which corresponds to tangible media such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer).

[0110] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term "processor" as used herein may refer to any of the above structures or any other physical structures suitable for implementing the described techniques. In addition, these techniques may be fully implemented in one or more circuits or logic elements.

Claims

1. A computer-implemented user interface comprising: Multiple user interface elements, each user interface element corresponds to a structure in a list of structures expected in a surgical video, and each user interface element has a visual attribute, wherein in response to detecting that a first structure is in a field of view of the surgical video, the visual attribute of the first user interface element is set to a first state, and in response to detecting that the first structure is not in the field of view, the visual attribute of the first user interface element is set to a second state.

2. The computer-implemented user interface of claim 1, wherein: The visual attribute of the first user interface element is based on marking the first structure with an overlay using the visual attribute.

3. A computer-implemented user interface as claimed in claim 1 or 2, wherein: The first structure is detected to be in the field of view using machine learning.

4. A computer-implemented user interface as claimed in claim 1, 2 or 3, wherein: The plurality of user interface elements are grouped in a single menu user interface.

5. The computer-implemented user interface of claim 4, wherein: The menu user interface is a toolbar that is displayed as a graphical overlay on the surgical video.

6. A computer-implemented user interface as claimed in any preceding claim, wherein: The surgical video is a real-time video stream.

7. The computer-implemented user interface of claim 5, wherein: The position where the toolbar is rendered can be fixed or user configurable.

8. A computer-implemented user interface as claimed in any preceding claim, wherein: The structure list includes a predetermined structure list based on the type of surgery in the surgery video.

9. A computer-implemented user interface as claimed in any preceding claim, wherein: The structure list includes a dynamic structure list.

10. A computer-implemented user interface as claimed in any preceding claim, wherein: The visual attribute is one of color, pattern, shape, image, and animation.

11. A computer-implemented user interface as claimed in any preceding claim, wherein: The user interface element includes a label.

12. A computer-implemented method comprising: using machine learning by one or more processors to identify structures in a video of a surgical procedure; In response to at least a portion of the structure being visible in the field of view, representing, by the one or more processors, a user interface element corresponding to the structure using a first visual attribute; as well as In response to the structure not being visible in the field of view, the one or more processors represent, using a second visual attribute, the user interface element corresponding to the structure.

13. The computer-implemented method of claim 12, wherein: The structure is one of anatomical structure and surgical instrument.

14. The computer-implemented method of claim 13, wherein: The anatomical structure is one of an organ, artery, tube, surgical item, and anatomical landmark.

15. The computer-implemented method of claim 13 or 14, wherein: The surgical instrument is one of a clamp, a stapler, a knife, a scalpel, a sealer, a divider, a dissector, a tissue fusion instrument, a monopolar instrument, a Maryland instrument, and a fenestrated instrument.

16. The computer-implemented method of any one of claims 12 to 15, wherein: The structure is one of a list of predefined structures.

17. The computer-implemented method of any one of claims 12 to 16, wherein: The user interface element includes a geometric shape that is displayed at a predetermined location during the video display of the surgical procedure.

18. The computer-implemented method of any one of claims 12 to 17, wherein: The first visual attribute is used to represent the structure by displaying a graphical overlay having the same first visual attribute in response to the structure being in the field of view so as to highlight the structure.

19. The computer-implemented method of any one of claims 12 to 18, wherein: This video is a live video stream of the surgical procedure.

20. A computer program product comprising a memory device having stored thereon computer executable instructions which, when executed by one or more processors, cause the one or more processors to perform a method for generating a user interface to depict the presence of a structure in a field of view, the method comprising: Using a neural network model trained using surgical training data to identify structures in videos of surgical procedures; generating a visualization comprising a graphical overlay at the location of the structure in the video of the surgical procedure, the graphical overlay using a first visual attribute; identifying a symbol corresponding to the structure from a list of displayed symbols; as well as The symbol is updated by displaying the symbol using the first visual attribute.

Images