Devices, systems, and methods for intraoperative instrument tracking and information visualization

Abstract

Systems and methods for intra-operative tracking and visualization are disclosed. A current minimally invasive surgical (MIS) instrument pose may be determined based on a real-time intra-operative input video stream that includes a current image frame captured by a MIS camera. In addition, an instrument activation status and at least one parameter value associated with the instrument may also be determined. Intraoperative graphical visualization enhancements may be determined based on comparison of activation status and / or parameter values of the instrument to corresponding parameter thresholds. Visualization enhancement may be applied to the current graphics frame. The current graphics frame may also include a visualization enhancement relating to a proximate anatomical structure having a proximity determined by the instrument pose and the anatomical model. A current graphics frame may be mixed with a current input image frame to obtain an output mixed image frame, which may form part of an output video stream.

Description

Technical Field

[0001] The subject matter disclosed herein relates to intraoperative instrument tracking and visualization, and in particular to apparatus, systems, and methods that enhance intraoperative image display, facilitate data visualization, and improve safety. Background Art

[0002] Visualization systems are often used intraoperatively to image tissue, perform biopsies, surgical procedures, diagnose, and / or other medical procedures. As used herein, the term "intraoperative" refers to actions performed or events occurring during any medical procedure (invasive or non-invasive). In many cases, such as during endoscopic surgery, relevant medical information related to diagnosis and / or decision-making may not be readily available to the attending physician. For example, patient medical information (e.g., medical history, vital parameters, etc.), device status and device parameters (e.g., status and parameters of instruments and / or robots used to facilitate the medical procedure being performed), real-time information (e.g., related to tissue during surgery), etc. are typically part of completely different subsystems used for various operations and other reasons (e.g., privacy, data security, limited interoperability, etc.). Therefore, intraoperative access to the above medical information may involve multiple display screens and / or reliance on other personnel to provide information, which may divert attention from the medical procedure, increase the physician's cognitive load / mental effort, and interrupt the surgical process, thereby increasing the risk of errors, extending the surgical time, and increasing costs. Furthermore, when a head mounted display (HMD) and / or a stereo headset is used (e.g., during endoscopic and / or robotic-assisted surgery), it may be cumbersome to remove, put on, and / or otherwise adjust the HMD and / or headset to view other display screens. Accordingly, some embodiments disclosed herein facilitate the performance of medical procedures including endoscopic and / or robotic-assisted surgery, in part by facilitating intraoperative instrument tracking and enhancing information visualization. Summary of the invention

[0003] In some embodiments, a processor-implemented method may include: determining a current instrument posture associated with an instrument of a surgical device based on a real-time intraoperative input video stream captured by a MIS camera, wherein the input video stream includes a current input image frame, the current input image frame including an image of the instrument; determining an activation state of the instrument and at least one parameter value associated with the instrument; determining one or more first intraoperative graphic visualization enhancements based on (a) the activation state of the instrument, and (b) a comparison of at least one parameter value with at least one corresponding parameter threshold, wherein the first intraoperative graphic visualization enhancement is applied to the current graphic frame; and initiating display of a mixed image frame included in a mixed output video stream, wherein the mixed image frame is obtained by mixing the current input image frame with the current graphic frame.

[0004] In another aspect, a surgical device system may include: a camera for obtaining a real-time intraoperative video stream; an instrument; a memory; and a processor coupled to the camera, the instrument, and the memory. In some embodiments, the processor may be configured to: determine a current instrument posture associated with the instrument based on a real-time intraoperative input video stream, wherein the input video stream includes a current input image frame, the current input image frame including an image of the instrument; determine an activation state of the instrument and at least one parameter value associated with the instrument; determine one or more first intraoperative graphic visualization enhancements based on (a) the activation state of the instrument, and (b) a comparison of at least one parameter value with at least one corresponding parameter threshold, wherein the first intraoperative graphic visualization enhancement is applied to the current graphic frame; and initiate display of a mixed image frame included in a mixed output video stream, wherein the mixed image frame is obtained by mixing the current input image frame with the current graphic frame.

[0005] In yet another aspect, a system may include: a component for determining a current instrument posture associated with an instrument of a surgical device based on a real-time intraoperative input video stream captured by an imaging component of the surgical device, wherein the input video stream includes a current input image frame that includes an image of the instrument; a component for determining an activation state of the instrument and at least one parameter value associated with the instrument; a component for determining one or more first intraoperative graphic visualization enhancements based on (a) the activation state of the instrument, and (b) a comparison of at least one parameter value with at least one corresponding parameter threshold, wherein the first intraoperative graphic visualization enhancement is applied to the current graphic frame; and a component for initiating display of a mixed image frame included in a mixed output video stream, wherein the mixed image frame is obtained by mixing the current input image frame with the current graphic frame.

[0006] In some embodiments, a computer-readable medium may include instructions to configure a processor to: determine a current instrument posture associated with a surgical device instrument based on a real-time intraoperative input video stream captured by a surgical device camera, wherein the input video stream includes a current input image frame that includes an image of the instrument; intraoperatively identify one or more proximity anatomical structures within a distance threshold of the instrument in the current input image frame based on the current instrument posture and a three-dimensional (3D) anatomical model; determine one or more intraoperative graphic visualization enhancements in the current input image frame based on (a) an activation state of the instrument, and (b) a determination that one or more parameters associated with the instrument fall outside of one or more corresponding parameter thresholds, and (c) one or more proximity anatomical structures; and initiate display of a mixed image frame included in a mixed output video stream, wherein the mixed image frame is obtained by mixing the current input image frame with the graphic visualization enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings.

[0008] Figure 1A An example diagram is shown illustrating an example system for intraoperative visualization according to certain embodiments disclosed herein.

[0009] Figure 1B A process flow diagram is shown illustrating an example method for intraoperative visualization according to certain embodiments disclosed herein.

[0010] Figure 2A and Figure 2B A schematic block diagram of an example embodiment of an apparatus for facilitating instrument tracking and intraoperative visualization is shown.

[0011] Figure 3A A schematic block diagram 300 is shown illustrating example high level signal and data flows between functional blocks to facilitate intraoperative visualization.

[0012] Figure 3B Aspects of example blocks for receiving and processing sensor input are detailed.

[0013] Figure 3C Some example functional blocks associated with blocks for determining proximity and MIS instrument state based feedback according to certain embodiments disclosed herein are shown.

[0014] FIG. 4A to FIG. 4I Example image frames are shown that may form part of a hybrid output stream and be viewed by a surgeon or another medical professional.

[0015] Figure 5 A flow chart of an example method for intraoperative tracking and visualization is shown.

[0016] Figure 6 An example software architecture associated with a system for intraoperative visualization according to certain embodiments disclosed herein is shown.

[0017] According to some example embodiments, similar reference numerals and symbols in each figure indicate similar elements. In addition, multiple instances of functional elements can be indicated by adding letters or hyphens and second digits after the first digit of the element. For example, multiple instances of element 160 can be indicated as 160-1, 160-2, 160-3, etc. In some cases, the suffix may refer to the same element but in different states and / or at different times. When only the first digit is used to refer to such an element, it should be understood that any instance of the element (for example, the element 160 in the previous example will refer to element 160-1, 160-2 and / or 160-3). DETAILED DESCRIPTION

[0018] Some disclosed embodiments relate to devices, systems and methods for facilitating intraoperative instrument tracking, enhancing intraoperative image display, facilitating data visualization, and improving the safety of medical procedures. In some embodiments, the disclosed technology can be applied during minimally invasive surgery (MIS) procedures, including procedures involving robots / robotic surgery.

[0019] While intraoperative visualization facilitates imaging of tissue, during medical procedures such as performing biopsies, surgical procedures, diagnostic procedures, etc., relevant medical information related to the intraoperative procedure may not be present and / or involve the use of multiple different display systems, thereby interrupting the surgical flow and increasing the likelihood of errors.

[0020] Some disclosed embodiments relate to systems, apparatuses, and methods for synchronizing a video stream from a minimally invasive surgical (MIS) device with information about an energy device (e.g., which may form part of and / or be coupled to a MIS system) and / or other medical subsystems in real time. Minimally invasive surgery refers to medical surgery performed using endoscopic and / or laparoscopic devices. In some cases, the MIS device may form part of a robotic surgical system. In some embodiments, alpha blending (or α blending) techniques may be used to overlay and / or blend a video stream from an endoscopic device with energy device information, end effector information, and / or information from other medical subsystems. The term energy device is used to refer to an instrument that emits energy, which may be used for cutting (dissection, transection, resection, ablation, etc.), performing a biopsy, cauterization, electrocoagulation, etc. Energy may be used and transmitted by an energy device in various forms, such as high-frequency radiation, ultrasonic energy, microwave radiation, direct heat application, mechanical energy (e.g., to cut tissue), etc. The transmitted energy interacts with living tissue, which may be used to cut tissue and organs or stop bleeding (depending on the surgery being performed). One problem with using energy devices is that the energy transmission may affect tissues or structures that are unrelated to the actual goal of the surgery, which may lead to medical complications and prolong recovery time. The term energy device is also used herein to refer to an end effector. An end effector refers to an instrument (e.g., a catcher, grasper, soldering knife, etc.) that can be coupled to an endoscope or a robotic device arm and transmits or uses energy to manipulate tissue. End effectors can be of various types and can be specific to the medical procedure being performed.

[0021] The disclosed embodiments facilitate mixing energy device related information onto an input real-time video stream, thereby facilitating real-time and intraoperative (during a surgical procedure) seamless visualization of energy device information, thereby reducing interruptions, facilitating visualization of device energy related information and / or other parameters, and reducing surgical cognitive load. In some embodiments, an auditory interface and / or a tactile interface may also be used to provide additional feedback to the surgeon.

[0022] For example, some disclosed embodiments provide visual and / or audible warnings during laparoscopic surgery when the blade temperature of the energy device is above a temperature threshold and the distance to one or more anatomical structures is below a predetermined safety distance or safety threshold. As used herein, the term threshold may refer to an upper limit (upper limit), a lower limit (lower limit), or a range (e.g., between an upper limit and a lower limit). If the parameter value is above an upper limit, below a lower limit, or outside a range, it may be determined that the parameter value falls outside a threshold.

[0023] In some embodiments, when the proximity of the anatomical structure to the energy device is below a certain distance threshold, the anatomical structure can be automatically identified (e.g., based on surgery) and can be visually enhanced (e.g., using augmented reality (AR) technology). The term "augmented reality or AR" refers to a computer-generated visual enhancement or enhancement of an object in a camera view. The enhancement or visual enhancement can be based on the camera pose (the position and orientation of the camera or imaging device) so that it appears to be a natural part of the scene (and is not seen as an extraneous artifact). AR can also involve the enhancement or addition of other sensory information (such as auditory and tactile information). In addition, in some embodiments, visual enhancement (e.g., using AR) can also be used to highlight the energy device when it is above a temperature threshold. In some embodiments, the endoscopic video stream can be enhanced with a heat map and / or a proximity map to indicate the sensitivity of various tissue regions to energy transmission based on their proximity. In some embodiments, supervised learning (e.g., in an offline phase) can be used to train an artificial intelligence (AI) and / or machine learning (ML) ("AI / ML") model to identify specific anatomical structures (e.g., blood vessels, organs, etc.) in the input video stream. The AI / ML model can be trained using preoperative images, 3D models (e.g., from computed tomography (CT) scans), etc., as well as real-time video streams. During the training phase, the model can be trained to recognize anatomical structures that are common between surgeries (e.g., blood vessels) and / or anatomical structures that are specific to a class of surgery (e.g., gastrointestinal, respiratory, urological, gynecological, etc.).

[0024] In some embodiments, during surgery, an anatomical structure of interest in a field of view (FOV) can be identified in real time in a real-time intraoperative video stream, the state of the energy device can be determined, and the distance of the anatomical structure of interest to the energy device can be determined. The state of the energy device and / or the distance of the anatomical structure of interest can be used to alert the medical practitioner. In some embodiments, the alert can occur by one or more of the following: (a) enhancing the visualization of the identified anatomical structure of interest; (b) enhancing the visualization of the energy device to indicate the state of the energy device; (c) enhancing the real-time intraoperative video stream with a proximity map and / or a heat map based on the proximity of the anatomical structure of interest to the energy device; (d) providing an auditory and / or visual indication of the device state when a safety margin (e.g., based on energy and / or proximity and / or activation time length) is violated; (e) providing other (e.g., textual or graphical) information related to one or more of the above (a) to (d) and / or other surgical information, and / or information synthesized with the real-time intraoperative video stream from one or more additional medical subsystems. For example, with respect to (e) above, patient medical information (e.g., medical history, preoperative and / or intraoperative vital parameters, etc.), real-time tissue status information, etc. (which, for various reasons, may typically be part of a completely different subsystem) may be synthesized and displayed along with the real-time intraoperative video stream. The information to be displayed may be determined based on a user profile (e.g., associated with the medical professional performing the surgery), system settings (e.g., set by the medical professional), surgery-specific settings (e.g., based on the type of surgery being performed), and / or some combination of the above. In some embodiments, the real-time intraoperative video stream may be a high-definition (HD) stereoscopic (3D) video stream that may be viewed by a medical practitioner using enhancements on an HMD. Thus, the embodiments disclosed herein facilitate the performance of medical procedures, including endoscopic surgery and / or robotic-assisted surgery, and improve surgical safety, in part by facilitating intraoperative instrument tracking and enhancing information visualization.

[0025] In some embodiments, inputs from multiple graphics and video streams may be received. The graphics stream may include instrument sensor data, robot information and parameters (e.g., during robot-assisted surgery), information provided by an artificial intelligence (AI) engine, which may be based on analysis of the video stream, etc. The information provided by the AI ​​engine may include intraoperative guidance, suggestions, enhancements (e.g., enhancements to the video stream), warnings, and / or assistance. In some embodiments, AI information may be provided in an enhanced form (e.g., using augmented reality (AR) technology on a real-time intraoperative video stream). A real-time intraoperative video stream may include 3D video obtained by a medical device (such as an endoscope) during an intraoperative procedure, etc. In some embodiments, one or more intraoperative video streams in the intraoperative video stream may be mixed with one or more graphics streams (e.g., using alpha blending or alpha blending) and displayed to medical personnel (e.g., surgeons) on one or more of a monitor, HMD, and / or a portable computing device to facilitate the execution of intraoperative surgery. For clarity, real-time video obtained during surgery by a medical device (such as an endoscope) used to perform a medical procedure is also referred to herein as a real-time video stream or an intraoperative real-time video stream. The term "composite stream" or "mixed stream" is used to refer to the result of mixing a first real-time video stream and one or more graphics streams.

[0026] Figure 1A An example diagram is shown illustrating an example system for intraoperative visualization according to certain embodiments disclosed herein. Figure 1A Patient 130 is shown undergoing a medical procedure involving an endoscope ( Figure 1A The use of Figure 1A In some embodiments, hub 150 (including base unit 155 and tablet or mobile computer housing touch screen 165), HMD 140, displays 135-1 and 115, tablet computer 145, together with surgical equipment (e.g., endoscope, laparoscope, robotic equipment, etc. - Figure 1A ), an endoscope controller 110 and a camera and instruments coupled to the surgical device may form a MIS system 108 (not shown in Figure 1A The endoscope controller 110 may control the endoscope and / or surgical / robotic device based on input from a medical professional 140.

[0027] like Figure 1AAs shown, device 150 (hereinafter referred to as "hub 150") can receive data streams via network 165 (e.g., also referred to as hospital network 170), which can be coupled to devices local to the medical facility. Hub 150 can take the form of or include the functionality of a computing device capable of graphics processing and video processing. Hub 150 can also receive multimedia streams, such as from an endoscope ( Figure 1A The hub 150 may be integrated with the MIS system 108 (e.g., be a part or subsystem of the MIS system), or in other embodiments, may be a separate system that is operably coupled (e.g., docked to the MIS system 108 via a docking port / station, and / or coupled to the MIS system using a wired and / or wireless interface). Similarly, in some embodiments, the HMD 140 may also be a part and / or subsystem of the MIS, or operably coupled (e.g., using a wired or wireless interface) to the MIS system. As previously described, the MIS system 108 may take the form of an endoscopic system, a laparoscopic system, and / or a robotic medical device.

[0028] In some embodiments, the video from the endoscope may be received by the hub 150, processed, and then wirelessly transmitted to the HMD 140. In some embodiments, the endoscope (first) input video stream seen by the medical professional 120-2 may be mixed with one or more second graphics streams (such as graphics, AI input, sensor data, etc.) based on the configuration of the hub 150. The sensor data may include data from sensors associated with the endoscope or other instrument (including energy devices) being used by the medical professional 120, which may provide and / or be used to determine camera and / or instrument pose, instrument status, and other information. The HMD 140 may be a custom or commercially available HMD.

[0029] The term "pose" (e.g., "camera pose" or "instrument pose") refers to the position and orientation of the appropriate element (camera or instrument). In some embodiments, the pose can be determined for 6 degrees of freedom (6DOF), which refers to three translational components (which can be given by X, Y, Z coordinates) and three angular components (e.g., roll, pitch, and yaw) relative to a reference frame. In some embodiments, the pose (camera or instrument) can be determined and / or tracked by a processor using a visual tracking solution based on images captured by the camera, which for a 3D video stream can include depth information. The instrument pose can be determined based on a combination of: input from a sensor coupled to an actuator associated with the instrument (where the actuator state can provide an indication of the instrument position), and / or known shape and size parameters associated with the instrument, and / or an identification of the instrument in the current FoV of the camera (which can be used to determine the instrument position based on the known or determined position of the camera and / or MIS device geometry).

[0030] In some embodiments, the camera may include an image sensor, such as a CCD, CMOS, and / or optical sensor, and may convert an optical image into an electronic or digital image, and may send the captured digital image to a processor. In some embodiments, the camera may be a color camera that provides "color information", which may be considered to include 1 to N channels, where N is some integer depending on the color space used to store the image. For example, an RGB image includes three channels, one for each red, blue, and green information. Color images may also be obtained in various other color spaces, such as hue saturation lightness (HSL), hue saturation value (HSV), or hue saturation brightness (HSB), etc.

[0031] In some embodiments, the camera may also include a depth sensor that can provide "depth information". The term "depth sensor" is used to refer to a functional unit that can be used to obtain per-pixel depth (D) information independently and / or in combination with the capture of a color (e.g., RGB) image by a camera. The depth sensor can capture depth information of a scene in the field of view of the camera. Therefore, each color image frame can be associated with a depth frame that can provide depth information of an object in the color image frame. In one embodiment, the camera can be stereoscopic and capable of capturing 3D images. For example, the depth sensor can take the form of a passive stereo vision sensor that can use two or more cameras to obtain depth information of a scene. In the captured scene, the pixel coordinates of points common to the two cameras can be used together with camera pose information and / or triangulation techniques to passively obtain per-pixel depth information. In another embodiment, the camera may include a color camera with an active depth sensor that can capture per-pixel depth information in addition to a color image when the active depth sensor is enabled. As another example, in some embodiments, the active depth sensor can take the form of a time-of-flight (ToF) sensor. The term ToF sensor is used to refer to sensors that use light such as structured light, strobe, etc. to determine depth information. For example, a strobe light coupled to a 3D camera can illuminate an object in the scene, and the reflected light can be captured by a CCD / CMOS or other image sensor. Depth information can be obtained by measuring the time it takes for a light pulse to travel to the object and return to the sensor. As another example, a light source can project a structured or textured light pattern (which may be composed of one or more narrow light bands) or a checkerboard pattern onto an object in the scene. Depth information can then be obtained by exploiting the geometric distortion of the projected pattern caused by the shape of the object surface. In one embodiment, depth information can be obtained from a combination of an infrared structured light projector and an infrared camera registered to an RGB camera.

[0032] In some embodiments, model-based tracking techniques (such as visual simultaneous localization and mapping (VSLAM) algorithms) can use image and depth information in (or derived from) images captured by the camera to determine a 6DOF camera pose estimate. In some embodiments, the MIS system may include other sensors, such as an inertial measurement unit (IMU) and / or accelerometer, etc., which may also provide input for camera pose determination. SLAM / VSLAM-based techniques can facilitate the generation of a map of an unknown environment while locating the position of the camera. In VSLAM, the images (and any depth information) obtained by the camera can be used to model the unknown scene with relatively low computational overhead, which can facilitate real-time and / or near real-time modeling. VSLAM may therefore involve tracking the 6DOF pose of the camera while also determining or refining the 3-D structure of the surrounding scene.

[0033] For example, in some embodiments, VSLAM technology can detect significant feature patches in one or more captured image frames and store the captured imaged frames as key frames or reference frames. In keyframe-based SLAM, the pose of the camera can then be determined, for example, by comparing the current captured image frame with one or more key frames and determining the pose of the camera based in part on the positions and other characteristics of common features in the current frame relative to the key frames.

[0034] In some embodiments, the scale-invariant feature transform (SIFT) technique can be used to identify features in an image frame, such as corner points, edges, etc. SIFT-based descriptors and their variants (including a variety of computationally efficient modified SIFT descriptors) facilitate robust tracking and feature correspondence between images. In some embodiments, key points in an image frame can be identified and local descriptors corresponding to individual key points can be constructed and used to track image features. A "patch" or "image patch" is a pixel area around a feature point that can be used for tracking. The techniques outlined above are only examples, and in some embodiments, various other techniques can be used for feature detection. For example, techniques based on accelerated segmented test features ("FAST") or their variants, accelerated robust features ("SURF") or their variants, oriented FAST and rotational BRIEF ("ORB") or their variants, hybrid point and / or edge detection techniques, gradient-based techniques, etc. can be used to perform feature detection with appropriate modifications, as will be apparent to those skilled in the art. When VSLAM is used in conjunction with augmented reality (AR) technology, virtual objects can be inserted into a user's view of the real world and displayed on a mobile device. Virtual objects may include highlights, emphasis, and / or other visual enhancements of real objects.

[0035] As another example, a dense SLAM method can be used with a monocular endoscope, where video frame clusters are segmented according to a parallax criterion and sparse SLAM feature matching is used to accurately estimate the cluster frame pose. Dense matches between cluster frames can be calculated in parallel by combining a variational method of zero-mean normalized cross-correlation and a gradient Huber norm regularizer. The Huber norm is a mixed error measure that is robust to outliers. The Huber norm can be minimized with a gradient-based algorithm and is used as a regularization term for optimization problems in computer graphics problems. The above-mentioned dense SLAM method is robust in the face of challenging lighting, poor textures, and deformations commonly encountered in endoscopic examinations. In addition, the above-mentioned dense SLAM method can run in real time on a processor (e.g., a GPU), and facilitates parallel endoscope position tracking and 3-D scene reconstruction without the need for external markers or tracking devices, and is easily integrated into the surgical workflow. The above method is described in “Live tracking and dense reconstruction for handheld monocular endoscopy” (N. Mahmoud et al., IEEE transactions on medical imaging (2018), pp. 79-89), the entire content of which is incorporated herein by reference.

[0036] As another example, DefSLAM can be used, which facilitates real-time tracking and mapping of deformable scenes. DefSLAM combines template shape (SfT) and motion non-rigid structure (NRSfM) techniques during exploratory SLAM sequences. Deformation tracking is used to determine the camera's pose and the deformation of the observed map at the frame rate by SfT processing of a template that models the shape of the scene at rest. The deformation mapping thread runs in parallel with tracking to update the template at the key frame rate through isometric NRSfM processing of a batch of full-perspective keyframes. DefSLAM produces an accurate 3D model of the scene relative to a moving monocular camera. DefSLAM is described in "DefSLAM: Tracking and Mapping of Deforming Scenes from Monocular Sequences", J. Lamarca et al., IEEE Transactions on Robotics, (2020): pp. 291-303, which is incorporated herein by reference in its entirety.

[0037] Figure 1B A process flow diagram is shown that illustrates an example method 170 for intraoperative tracking and visualization according to certain embodiments disclosed herein. Figure 1BIn some embodiments, the hub 150 may access a preoperative scenario-specific 3D model of an anatomical structure 180 (hereinafter referred to as a "preoperative 3D model"). The preoperative 3D model 180 may be specific to the current (intraoperative) surgery being performed and may be patient-specific. The preoperative 3D model 180 may include information about anatomical structures that may be encountered during surgery, including anatomical risk structures (e.g., organs, blood vessels, sensitive tissues, etc.) of the patient 130. In some embodiments, a preoperative CT scan 178A and / or a preoperative MRI image 178B may be used to obtain the preoperative 3D model 180. The preoperative 3D model 180 may include information about anatomical structures (e.g., including risk structures), including: (a) the location and extent of the structure, (b) biomechanical / mobility parameters of the anatomical risk structures, and (c) safety margins for the risk structures. Biomechanical / movement parameters can be patient-specific (e.g., for patient 130 based on the patient's medical / physical condition) and / or typical (e.g., based on a population sample having similar characteristics to the patient or based on a population sample having similar characteristics to the relevant anatomical structure).

[0038] In some embodiments, in box 176, intraoperative 3D reconstruction may be performed (e.g., by the MIS system 108 and / or the hub 150). For example, the MIS system 108 and / or the hub 150 may be able to perform real-time 3D reconstruction of the intraoperative environment based on the MIS system / endoscopic video feed. The term 3D reconstruction refers to the calculation of a continuous surface representation from a set of discrete points (e.g., point clouds, polygonal meshes, etc.) that retains important surface features and provides an acceptable approximation of the sampled surface in the presence of noise. When depth information is available (e.g., from a stereo MIS system), 3D reconstruction may be performed using stereo computer vision techniques. When passive stereo images are obtained using a pre-calibrated camera, 3D reconstruction may be performed based on multiple images of the intraoperative environment by determining the stereo correspondence of feature points in the image and triangulation based on known camera parameters (e.g., geometry, focal length, etc.). Structural refinement may also be performed using filtering or priors on the shape of the object. Once a collection of 3D points or point clouds is obtained (actively or passively), the points (in the point cloud) can be used to determine a volume scalar function to describe the surface, or the points in the point cloud can be connected in a consistent way to create a mesh surface representation, or a collection of shape primitives can be combined so that their boundaries after combination resemble a collection of points, or a collection of patches can be fit to a collection of points, and so on.

[0039] Various techniques can be used for 3D reconstruction (including when the camera is monocular (i.e., non-stereo)), such as shape from shading (SfS), shape from motion (SfM), deformable SfM, shape from template (SfT), shape from texture (SfTx), or a combination of the above techniques. Other techniques include ORBSLAM, which can estimate the 3D structure of the endoscope position and the intraoperative environment. ORBSLAM can be used to reconstruct 3D maps of soft tissue organs.

[0040] In some embodiments, the MIS system 108 / hub 150 may also run and / or receive input from a trained AI / ML model 172 for 3D reconstruction during surgery, which was previously trained offline. For example, in some embodiments, the MIS system 108 / hub 150 may implement an AI / ML-based method and / or run a previously trained AI / ML model 172 for 3D reconstruction. In some embodiments, the AI / ML model 172 for 3D reconstruction may use machine learning techniques for pairwise patch matching. For example, multi-view (from multiple images) reconstruction may be performed using a 3D recurrent neural network (based on long short-term memory (LSTM)) based model. LSTM has feedback connections that can process single images and image sequences (e.g., videos). As another example, the AI / ML model 172 for 3D reconstruction may use a convolutional neural network (CNN) for 3D reconstruction such as SurfaceNet. SurfaceNet pre-computes the cost volume based on voxel-by-voxel view selection and uses a trained 3D CNN-based model to regularize and infer surface voxels. The term voxel refers to a data point in a 3D model that contains a position and color value. As another example, the AI / ML model 172 for 3D reconstruction may be based on an unsupervised depth estimation framework, such as END flow. END flow is based on an unsupervised optical flow network and is trained offline on uncorrected stereo video without calibrated camera parameters. In some embodiments, a 3D model of an intraoperative anatomical structure (hereinafter referred to as an "intraoperative 3D model") 174 may be output as a result of performing intraoperative 3D reconstruction (e.g., in box 180).

[0041] In some embodiments, in box 186, intraoperative 3D registration may be performed. For example, the hub 150 may also register the intraoperative 3D model 174 (e.g., obtained from the intraoperative 3D surface reconstruction in box 180) with the preoperative 3D model 180 (e.g., obtained preoperatively from the preoperative CT scan 178A and / or the preoperative MRI image 178B). The registration of the intraoperative 3D model 174 and the preoperative 3D model 180 provides a unified reference system, which can facilitate navigation and tracking of MIS system cameras and instruments. As an example, volume information from the preoperative preoperative 3D model 180 (e.g., from the CT scan 178A and / or the MRI image 178B) can be used to facilitate surface-based registration. As another example, a surface mesh can be used for registration based on geometric shape information. As an example, the shape of the surface of an organ (or a portion of the surface of an organ) can be determined intraoperatively (e.g., based on a 3D surface reconstruction) and matched with the shape of the same organ in the preoperative 3D model. In some embodiments, anatomical landmarks or anatomical feature points common to both the intra-operative 3D model 174 and the pre-operative 3D model 180 may also be used to facilitate registration.

[0042] In some embodiments, the MIS system 108 / hub 150 may implement and / or use a context-aware system that, based on context (e.g., the type of surgical procedure), may identify organs based on: (a) segmentation by extracting homogeneous regions from an image (e.g., the left image) of a stereoscopic image pair, (b) segmentation by extracting texture and intensity features from the segmented image, (c) merging segments based on confidence assigned during the classification process. In some embodiments, a previously trained context-specific AI / ML model 182 for semantic segmentation may be used to determine the segments. The term segmentation refers to classifying pixels in an image as belonging to one or more semantic classes, which may be predefined. The term "semantic class" refers to a category to which a pixel has been classified (e.g., kidney, instrument, shaft, background, etc.). In some cases, an output channel may be created for each class to which the pixel belongs. Each output channel may be used as a mask to restrict image operations to that class. Thus, in semantic segmentation, each pixel of an image I may be labeled with one of the S semantic classes.

[0043] In addition, in some embodiments, a first imaging system (e.g., a preoperative ultrasound or CT scanning device) and a second imaging system (e.g., an intraoperative endoscope) can be pre-calibrated to use a common reference frame. For example, a tracking system can be used to establish a spatial relationship between the two imaging systems to facilitate intraoperative 3D registration. As another example, markers (e.g., optical, electromagnetic, magneto-optical, etc.) can be used preoperatively (e.g., using a CT scan) and intraoperatively on the system (e.g., endoscope) to facilitate later (e.g., intraoperative) registration.

[0044] In some embodiments, data and images from two or more imaging platforms (which may include 2D color images without depth information) may be fused and / or combined. For example, various 2D-3D registration algorithms may be used to facilitate data / image fusion. As an example, a standard 2D-3D registration method uses a 2D projection view, an initial or estimated pose, and a 3D volume model to register images and refine the registration. In some embodiments, a trained deep learning model may be used to determine the initial or estimated pose, and to facilitate registration and refinement based on various cost metrics (such as normalized cross-correlation or binary cross entropy loss). For example, some embodiments may use an intensity-based deformable registration process based on a grid-based parameterization with B-splines or free-form deformation (FFD) for parameterization.

[0045] In some embodiments, a registered 3D anatomical model (hereinafter referred to as "registered 3D model") 184 may be output as a result of performing intraoperative 3D registration (e.g., in block 186). In some embodiments, the registered 3D model may include and / or be associated with semantic classifications of pixels / voxels (e.g., an object class associated with each pixel / voxel). As described above, semantic segmentation may include anatomical structure classifications (e.g., vessels, lesions, tissue / organ type, fat, etc.) and / or instrument-related classifications (e.g., instrument type, sheath, etc.).

[0046] In addition, in some embodiments, in block 190, 3D MIS instrument localization may be performed. For example, the hub 150 may implement real-time MIS instrument localization 190 to determine the 3D pose of the MIS instrument. In some embodiments, the pose of the MIS instrument may be determined based on the pose of the MIS system camera. As previously described, the MIS instrument may be an energy device.

[0047] In some embodiments, the instrument and / or camera pose may be determined (e.g., in box 190) using a previously trained pose estimation AI / ML model 188, which may facilitate instrument tracking and determine the instrument pose. In some embodiments, the instrument detection and tracking model may be based on a convolutional neural network (CNN). As an example, a CNN-based pose estimation AI / ML model 188 may be used to detect and segment the instrument before using mathematical (e.g., algebraic geometry) techniques for instrument pose estimation. In order to segment the instrument, a CNN-based pose estimation AI / ML model 188 may use an encoder and decoder with pixel-by-pixel classification. The encoder may include convolution and subsampling layers and perform feature extraction. The subsampling layer promotes spatial invariance by reducing the feature map, which results in extracting more features at a lower computational cost. The decoder then projects the extracted features onto the pixel space to obtain a dense pixel-by-pixel classification. In order to avoid losing spatial resolution due to subsampling, the CNN may use a skip connection, which connects the last layer of the decoder to the original (full-resolution) image via a series of depth-separable convolutions (i.e., full-resolution feature map generators) that do not perform subsampling.

[0048] CNNs for image classification typically use a "flattening layer" to vectorize arrays into a single long continuous linear vector. The flattening layer may be followed by several densely connected layers. Typically, most parameters used by CNN for image classification are derived from densely connected layers, which may lead to overfitting. In some cases, in order to prevent overfitting, a dropout layer may be used during CNN model training, which randomly sets half of the activations of the fully connected layers to zero. In addition, CNNs may use a global average pooling (GAP) layer, in which only one feature map is generated for each corresponding classification category. The GAP layer performs dimensionality reduction by reducing the height x weight x depth dimensional tensor to a 1X 1X depth vector to avoid overfitting, which facilitates classification robustness in the presence of spatial translation. In some embodiments, CNNs may use depth-separable convolutions to limit network size, reduce resource overhead, and facilitate real-time operation.

[0049] In some embodiments, the instrument posture can then be determined by extracting the edge and center line of the instrument. The edge and center line of the instrument can be determined using edge detection techniques (such as thresholding and Hough transform), while the maximum pixel value can be used to determine the instrument tip, and the direction parameter r_3 of the instrument axis can be determined from the intersection of the edge line and the center line. The edge line, the center line, and the instrument origin can form a geometric primitive that can be used to determine the posture. The instrument origin c can be determined as a 3D position where the instrument tip closely matches the extracted instrument tip when reprojected (e.g., using least squares or another appropriate error minimization technique). The above-determined posture can be refined by minimizing the pixel intensity along the geometric primitives predicted by the geometric model and its 3D posture (e.g., minimizing the sum of squared intensities). Therefore, in some embodiments, the CNN-based posture estimation AI / ML model 188 can be used to semantically segment the image and detect the instrument. In some embodiments, the CNN-based model segmentation can also be used to distinguish between parts of the instrument (such as actual tools (e.g., buckles, cauterizers, etc.)) and other parts of the instrument (e.g., shafts, etc.). In some embodiments, as a result of performing 3D MIS instrument positioning (e.g., in block 190), a 6DoF MIS instrument pose 191 (hereinafter referred to as "instrument pose") may be output. In some embodiments, a pose uncertainty or pose error margin may be output with the instrument pose 191.

[0050] In box 193, the instrument pose 191 can be used for continuous anatomical distance measurement. For example, in box 193, the hub 150 can determine the distance between the MIS instrument and one or more identified anatomical features (or portions of anatomical features). In some embodiments, the distance measurement can be scenario-specific. For example, the degree of tissue movement (e.g., maximum or typical), the uncertainty of the pose measurement can be taken into account in the distance measurement. Tissue movement can depend on the tissue or organ close to the operation being performed, organ parameters (such as organ health, organ size, etc.) and / or other patient parameters (e.g., physical parameters such as weight, height, etc., and medical parameters such as patient diagnosis, comorbidities, etc.). For example, the degree of intraoperative movement can be different for the liver relative to intestinal tissue. As another example, the intraoperative degree of a diseased liver with condition A can be different relative to a healthy liver or a liver with condition B. Therefore, the tissue movement parameters can be surgery-specific, organ-specific and / or patient-specific or a combination of the above. The pose uncertainty can be estimated based on the typical error margin of the model used to determine the instrument pose 191. In some embodiments, tissue movement parameters and pose estimation uncertainties may be factored into distance measurements of various identified anatomical features of interest.

[0051] In some embodiments, in box 190, based on the pose of the energy device, the hub 150 may determine the distance between the energy device and the anatomical risk structure (e.g., identified in the registered intraoperative 3D model 184). In some embodiments, the hub 150 may determine whether the estimated distance is below a distance threshold (e.g., a specified safety margin for the procedure, anatomical structure, and / or instrument / energy device). The estimated distance and / or safety margin may account for potential tissue movement. For example, the distance threshold may be modified for the procedure (e.g., not changing the estimated distance to the anatomical structure) to account for tissue movement. As another example, the distance estimate may be modified to account for pose estimation errors and / or tissue movement.

[0052] In block 195, a MIS instrument state and / or tissue state may be determined based on sensor input 194. MIS instrument state determination may occur continuously (e.g., at frequent intervals) and / or in parallel with other blocks. Sensor input 194 may provide information about device activation, energy usage, energy emitted, temperature, jaw state, force, force distribution, applied pressure, tissue hydration, tissue state, etc. The sensor may include a microelectromechanical sensor (MEMS) and may be integrated into a MIS instrument and / or as part of one or more MIS subsystems. The sensor may include one or more of the following: a force sensor (including a capacitive force sensor, e.g., embedded in the instrument to determine clamping force, clamping force distribution, tissue state, etc.), a microactuator, a temperature sensor, a microphone (e.g., which may provide audio input, the audio may provide an indication of tissue state, such as tissue hydration, etc., which may change as the temperature of the tissue changes), a pressure sensor (such as a photonic crystal fiber (PCF) sensor or a fiber Bragg grating (FBG) sensor), a component that uses the scattering produced by a FBG sensor, a fiber optic pressure sensor, an optical sensor (e.g., for photoplethysmography (PPG)), etc. In some embodiments, the sensor input may be used directly and / or input into a model that may determine other instrument and / or tissue parameters based on the raw sensor input. The threshold may be for one or more of the above instrument parameters or tissue parameters. For example, an upper limit may be set for the energy emitted, or a range may be set for the clamping force distribution, an upper limit may be set for energy usage, a lower limit may be set for tissue hydration, etc. Parameter thresholds may be customized by a medical professional (e.g., for some parameters) and / or predetermined based on surgery, tissue type, etc. (e.g., for some other parameters). In some cases, parameter thresholds may represent safety thresholds for surgery, tissue type, and / or instrument. In some embodiments, sensory inputs may also be received from instruments / sensors separate from the MIS (e.g., via hub 150). For example, one or more intraoperative patient parameters being monitored (e.g., blood pressure, heart rate, etc.) may be received as part of sensor input 194 (e.g., when these parameters are not directly determined by the MIS).

[0053] In some embodiments, block 195 may output information about one or more relevant MIS instrument states / tissue states 197 (e.g., instrument activation state, temperature, jaw open-closed state, energy usage, energy output, resonant frequency, tissue state, etc.). For example, in block 195, the hub 150 may determine whether one or more instrument state parameters (e.g., activation state, temperature, mechanical state, radiant energy, energy usage, etc.) exceed corresponding instrument state parameter thresholds. For example, the hub 150 may determine whether an energy device is activated and / or whether its temperature exceeds a certain threshold (e.g., 50 degrees Celsius) and / or the amount / degree by which it exceeds a corresponding threshold. In some embodiments, block 195 may be performed independently during surgery, and relevant state information may be output to block 196.

[0054] In box 196, feedback based on proximity and MIS device status can be determined and / or generated for the medical professional, which can be used to provide appropriate feedback (e.g., via HMD 140). For example, in the event that the hub 150 determines that the proximity to one or more anatomical structures is below a distance threshold and one or more device state parameters (e.g., activation state, temperature, mechanical state, radiated energy, energy usage, etc.) fall outside of corresponding parameter thresholds, then a visual and / or audible indication can be generated and provided to the medical professional operating the MIS system via HMD 140, as further outlined herein. The audible indication can change in pitch and / or in the number of times the tone is played in a given time period and / or in amplitude / volume based on the device status and / or proximity to the anatomical structure. The visual indication can use AR to highlight elements and can also be color coded to reflect the device status, as further outlined herein. In some embodiments, the output of box 196 may include one or more of the following: an anatomical structure distance map (showing the distance to relevant anatomical structures within a certain threshold), a parameter threshold heat map (showing instruments exceeding the corresponding parameter threshold and the amount by which the threshold was exceeded), and an activation status map (indicating active / inactive devices).

[0055] refer to Figure 1A In some embodiments, hub 150 may include or be coupled to an AI engine that may perform AI processing locally (including running Figure 1BIn some embodiments, AI information streams may also be received over the network 165 and / or from a cloud-based service 175. Cloud-based services may be hosted on a private cloud, a public cloud, and / or a hybrid public-private cloud. AI functionality may be distributed between the hub 150, computing devices, and other devices on the hospital network 170, and the cloud 175. For example, the hub 150 may run the AI / ML models described herein and other models to provide AI-based decision support, video annotation, etc., while AI-related data may be de-identified (e.g., for patient privacy and regulatory reasons) and stored locally and / or on a device coupled to the hospital network 170, while machine learning models may be trained and / or improved using de-identified collected data using AI-related applications provided on the cloud 175.

[0056] In some embodiments, such as Figure 1A As shown, the example hub 150 may include a touch screen display 165 that can be used to present a user interface (UI) to enable the medical professional 120 to select an input stream, select an AI model to be run, and configure other functions, such as display of endoscopic images, reception of sensor data, providing power to devices, and / or controlling instruments and / or devices coupled to the hub 150. In some embodiments, the touch screen 165 can be detachable and can be wirelessly coupled to a base unit 155 of the hub 150. In some embodiments, the touch screen 165 can form part of a tablet computer and / or other mobile computing device, and the processing power can be distributed between the base unit 155 and the tablet computer or mobile computer that houses the touch screen 165. In some embodiments, the base unit 155 can also serve as a docking station for the touch screen 165 and / or other tablet computers and / or mobile computing devices.

[0057] In some embodiments, the hub 150 can function as a wireless access point (WAP) and broadcast its own local network 160-1, which can be used by authorized devices (e.g., within the operating room and / or within range). Authorized devices can connect to the hub 150 (functioning as a WAP) via the network 160-1 and can be communicatively coupled to the hub 150 via the network 160-1. The hub-centric network 160-1 can shield various intraoperative devices in or near the operating room from wireless network outages, signal reception issues, bandwidth-related issues, etc. that may occur in a hospital-provided wireless network (e.g., hospital network 170) or a medical facility. Thus, the hub-centric network 160-1 can facilitate increased bandwidth and reduced latency between intraoperative devices coupled to the hub 150, and improve information sharing including video, images, graphics, and / or other information while maintaining privacy. For example, hub 150 may receive images from an intraoperative imaging device (e.g., such as a fluoroscope camera) in an operating room (OR) via network 160-1 and may display the images on display 135-1. As another example, hub 150 may send graphics and / or user interface (UI) information for display on tablet computer 145. Medical professional 120-1 may remotely use the UI to configure hub 150 and / or one or more devices coupled to hub 150 via network 160-1.

[0058] Figure 2A and Figure 2B 2 is a schematic block diagram of an example system 200 illustrating functional components of an embodiment of an apparatus for facilitating fault-tolerant intraoperative visualization, such as an example hub 150. As used herein, the term processor may refer to a frame processing engine (FPE) 255-1 and / or a GPU 220-1 and / or a CPU 250 (e.g., as in Figure 2A and Figure 2B any combination of .

[0059] like Figure 2AAs shown, a frame processing engine (FPE) 255-1 in the hub 150 may receive multiple input video streams 202 (e.g., high definition (HD) video streams, such as from an endoscope or ultrasound device). The input video streams 202 may include a first input video stream 202-1, which may correspond to a video stream associated with a procedure currently being performed by the medical professional 120-1. The input video streams 202 may be provided via a standard interface (e.g., such as a high-definition multimedia interface (HDMI) or a serial digital interface (SDI), etc.) or a custom interface. The HD video may include 10-bit 4K video, such as, for example, a resolution of 3840x 2160 pixels (also known as 2160p), 1920x 1080 pixels (also known as 1080p video), and 1280x 720 pixels (also known as 720p video), wherein the frame rate is 59.94 image frames per second and / or 60 image frames per second (fps).

[0060] FPE 225-1 may also be capable of outputting video stream 204. In some examples, the number of output channels may reflect the number of input channels, and output video stream 204 may reflect the resolution and frame rate of input video stream 202. Output video stream 204 may be transmitted via an HDMI interface and / or an SDI interface. In some embodiments, output video stream 204 may also be transmitted wirelessly in association with hub 150 (e.g., via a wireless local area network (WLAN)). FPE 225-1 may include and / or be coupled to memory 270. Memory 270 may include a plurality of frame buffers ( Figure 2A 2 (not shown), which may be used to store graphics frames and / or video frames. Graphics frames may be received using an "alpha" channel (e.g., as red-blue-green-alpha or "RGBalpha"), which may indicate a transparency associated with each pixel. The alpha channel may be separate or premultiplied. Graphics frames (e.g., UI) may be received from a graphics processing unit (GPU) 220-1.

[0061] In some embodiments, the memory 270 may maintain multiple frame buffers for each input video stream. In some embodiments, each frame buffer may be configured to have the same size as the input video stream being processed. The video information and graphics information in the frame buffers associated with the channel may be mixed based on the alpha information (e.g., in the graphics stream) and output by the FPE 225-1. In some embodiments, the memory 270 may also store data; program code; models including one or more of the preoperative 3D model 180, the intraoperative 3D model 174, the registered 3D anatomical model 184, the AI / ML model 182 for semantic segmentation, and the pose estimation AI / ML model 188; and / or some combination of the above.

[0062] In some embodiments, the FPE 225-1 may be implemented using an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) add-on board, and / or using some combination of hardware, software, and firmware. As an example, the FPE 225-1 may use an M2 form factor FPGA and use a peripheral component express (PCIe) interface to communicate with the GPU 220-1 processor 250 and / or other components on the hub 150. PCIe is a high-speed serial computer expansion bus standard. In the above embodiments, the FPE 225-1 may be added to an existing computing platform such as a PC as a solution. For example, the FPE 225-1 (e.g., as an additional PCIe component) may be used to add the functionality disclosed herein to an existing computing platform that may lack the features disclosed herein. In some embodiments, the FPE 225-1 and other hub components may receive power from the power supply 208, which may comply with the IEC 60601-1 standard for medical devices. IEC 60601-1 is a series of standards related to basic safety and essential performance requirements for medical electrical equipment.

[0063] The GPU 220-1 may perform graphics processing and AI processing. For example, the GPU 220-1 may include a tensor processing unit (TPU) or tensor core, which may be capable of running AI models (e.g., one or more of the preoperative 3D model 180, the intraoperative 3D model 174, the registered 3D anatomical model 184, the AI / ML model 182 for semantic segmentation, the pose estimation AI / ML model 188, and / or some combination of the above), graphics processing including 3D graphics, video rendering, etc. The GPU 220-1 may be capable of receiving and processing video input from the FPE 225-1 and preoperative and intraoperative images related to the patient 130. The AI ​​application running on the GPU 220-1 may process the received input to determine and mark blood vessels, hemorrhage, etc. in the frame. In some embodiments, GPU 220-2 may provide graphics and / or UI and / or enhancements in real time to FPE 225-1, which may use annotations / enhancements to the video to provide to medical professionals 120 intraoperatively as decision support, guidance, warnings, etc. GPU 220-1 may also receive video input from FPE 225-1, which may be displayed on displays 135-1, 115( Figure 1A ) and / or on a tablet computer 145.

[0064] Although shown as being separate from FPE 225-1, functionality associated with one or more of GPU 220-1, processor 250, and memory 270 may be combined with FPE 225-1 (e.g., as a single component, chip, or card), or external to FPE 225-1. For example, memory 270 may be distributed across hub 150 such that one or more components such as processor 250, FPE 225-1, and / or GPU 220-1 may each include local memory and may also access main memory or system memory. In addition, memory 270 may include primary memory and / or secondary memory.

[0065] Program code, data, images, configuration information, graphics, image and video frames, models including AI models, etc. may be stored in the memory 270 and may be read and / or executed by one or more of the FPE 225-1, GPU 220-1, and / or processor 250 to perform the techniques disclosed herein. As used herein, the term "memory" refers to any type of long-term memory, short-term memory, volatile memory, non-volatile memory, or other memory, and is not limited to any specific type of memory or any number of memories, or any type of medium on which memory is stored. Examples of storage media include computer-readable media encoded with databases, data structures, etc. and computer-readable media encoded with computer programs. Computer-readable media may include physical computer storage media. Storage media may be any available media that can be accessed by a computer. By way of example and not limitation, such computer-readable media may include random access memory (RAM) and its variants, including non-volatile RAM (NVRAM), read-only memory (ROM) and its variants, erasable programmable (EPROM), flash memory, etc. Computer-readable media may also include compact disc ROM (CD-ROM), memory cards, portable drives or other optical disc storage, magnetic disk storage, solid-state drives, other storage devices, or any other medium that can be used to store desired program code in the form of instructions and / or data structures and can be accessed by a computer; as used herein, disks and optical discs include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0066] The communication interface 226 may be capable of communicating with another device (e.g., HMD 140) by wire (e.g., using wired communication interface 228) or wirelessly (e.g., using wireless communication interface 230), and may be used to receive sensor inputs 194, communicate with smart devices 242, and / or external networks such as hospital network 165 (FIG. 1).

[0067] Captured images, AI models, device states, sensor inputs 194, inputs from smart devices 244, robot data, etc. can be received through the communication interface 226. The communication interface 226 can also be used to receive user input (e.g., received from the tablet computer 145) and transmit confirmation / messages. Wireless communication may include communication using one or more of the following: a wireless local area network (WLAN) interface 238 (e.g., communicating with the tablet computer 145, display 140 via Wi-Fi, etc.), which may be based on the IEEE 802.11 standard; and / or through a wireless wide area network (WWAN) interface 240 (e.g., communicating with a remote mobile device, a cloud-based edge server via the cloud 175, etc.), which may be based on a cellular communication standard such as a fifth generation (5G) network or long term evolution (LTE); and / or through a wireless personal area network (WPAN) interface 236 (e.g., communicating with an audio device, some nearby smart devices 242, etc.), which may be based on IEEE 802.11x (e.g., using Bluetooth, etc.).

[0068] In some embodiments, cloud-based applications, AI models, preoperative and intraoperative images, etc. can be downloaded by the processor using wired interface 228 from one or more devices coupled to hub 150 via USB interface 234, and / or a server coupled to hospital network 165 via Ethernet 232 and / or from a cloud-based server coupled to cloud 167 via Ethernet 232.

[0069] In some embodiments, the processor may also send commands to a control device using the device control interface 222 or through the communication interface 226 based on input received from the medical professional 120 or based on instrument status, sensor input, configuration settings, etc. The power interface 220 may be enabled by the processor 150 to supply power to one or more instruments, sensors, and / or other devices coupled to the hub 150. The hub 150 may receive power from the power source 208 when plugged in, and may include backup battery power in some embodiments.

[0070] The processor 150 may also communicate with a device (e.g., a robot or other device) via a custom interface 224, which may be dedicated. In some embodiments, the processor may also run program code to control the operation of the hub 150, record events, notify medical personnel (e.g., via messages), store data, images, sensor inputs, retrieve patient records from a database (local or networked), facilitate the maintenance of multimedia operation synchronization, provide support for medical digital imaging and communications (DICOM) and other standards, facilitate access to cloud-based services, etc. For example, based on user input and configuration information, the processor 150 may record and record a surgery performed using an endoscope for replay (e.g., for training and / or review) and / or analysis (e.g., for training a machine learning model / AI model) at a later time. As another example, the processor 150 may facilitate real-time streaming of surgery via a communication interface 226 (e.g., for educational purposes or training purposes). DICOM is a standard for the communication and management of medical imaging information and related data between devices.

[0071] The methods described herein may be implemented in hardware, firmware, software, or a combination thereof. For hardware implementation, the processor 250, FPE 225, GPU 220 may be implemented in one or more application specific integrated circuits (ASICs), TPUs, digital signal processors (DSPs), image processors, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, or any combination thereof. The processor 250 and GPU 220 may also include functionality to perform other well-known computer vision and image processing functions, such as feature extraction from images, image comparison, image matching, object recognition and tracking, image compression and decompression, blending (including line-by-line blending), alpha blending, etc. Figure 2A and Figure 2B The components and / or blocks shown in (e.g., processor 250, GPU 225, FPE 220, memory 270, communication interface 226, etc.) may be coupled using a bus, an interface (e.g., PCIe, etc.), a wired connection, etc.

[0072] Figure 2B 2 is a schematic block diagram 200 of an example system illustrating functional components of another embodiment of an apparatus for facilitating intraoperative visualization, such as the example hub 150. Like labeled components are similarly represented in FIG. Figure 2A It works as described in . Figure 2B The description describes the functional blocks and / or components relative to Figure 2A An implementation that implements some change in functionality, behavior, or interaction.

[0073] like Figure 2B As shown, encoder 218-1 may receive and encode a 3D HD video stream 208 (e.g., from a 3D camera or stereo camera associated with an endoscope), which may include a left video channel 208-L and a right video channel 208-R, each at 60 fps. Figure 2B As shown, encoder 218-2 can receive and encode 2 separate HD video streams 209 (e.g., from a C-arm extension camera and an ultrasound device), each at 60fps. Switch 290 can receive the encoded input video streams 208 and 209 and graphics input in the form of an RGB alpha stream (RGB stream with an alpha channel) from GPU 220-2. Figure 2B As shown, the switch 290 may be capable of sending 4 HD input streams at 60fps each and receiving 4 HD graphics RGBα streams at 60fps each to / from the GPU 220-2. The switch 290 may be configured to send the encoded input stream 208 and the RGBα stream to the 3D decoder 280 with alpha blending. Similarly, the switch 290 may be configured to send the encoded input stream 209 and the RGBα stream to the decoder 275 with alpha blending and / or to the compression and wireless transmission block 285.

[0074] In some embodiments, for example, when the mixed (graphics and video) frames output by the 3D decoder with alpha blending 280 are incorporated into the input video channel 208, the mixed stream can be output (e.g., to the HMD 140) as a 3D video stream 214, which can include a left video channel 214-L and a right video channel 214-R at 60 fps each. As another example, when the mixed (graphics and video) frames output by the decoder with alpha blending 275 are incorporated into the input video channel 209, the mixed stream can be output as an HD stream 212 at 60 fps each.

[0075] like Figure 2B As shown, FPE 225-2 may also include a wireless compression block 285 that may compress one or more of the input streams for wireless transmission. Figure 2B , for simplicity, stream 216 is shown as being output by wireless compression block 285. However, stream 216 may be sent by wireless compression block 285 to communication interface 226 and then output using wireless communication interface 230 and / or WLAN interface 236.

[0076] Figure 3AA schematic block diagram 300 is shown that illustrates example high-level signal and data flows between functional blocks to facilitate intraoperative visualization. In some embodiments, one or more functional blocks may form part of a device (e.g., MIS system 108 / hub 150) to process information received from various MIS subsystems (e.g., endoscopes, instruments, sensors, etc.) to facilitate providing user feedback (visual, auditory, tactile, etc.). In some embodiments, an HMD 140 may be used to provide some user feedback.

[0077] In some embodiments, the hub 150 and / or the HMD 140 may be housed separately but operably coupled to the MIS system 108. In other embodiments, the hub 150 and / or the HMD 140 may form part of (e.g., as a subsystem of) the MIS system 108. Thus, in some embodiments, the hub 150, the HMD 140 may form part of a single (functionally) integrated MIS system 108, and the MIS system 108 may include the functionality described herein with respect to the hub 150 and / or the HMD 140.

[0078] In some embodiments, block 310 may process input video data 202 (e.g., from a stereoscopic video device such as MIS video device 305 (which may include color and depth, such as RGB-D information)). In some embodiments, example block 310 may include processing performed by blocks 172 through 190 (e.g., Figure 1B ) and may output an instrument pose 191 (which may include pose uncertainty in some cases). In some embodiments, block 310 may also receive sensor data (as indicated by the dashed line from block 195 to block 310), which may be used during video processing in block 310. For example, in some cases, information from a sensor (e.g., an ultrasonic sensor, etc.) may be used to determine the instrument pose.

[0079] exist Figure 3A In some embodiments, block 195 may receive and process sensor input 194 (e.g., as described above with respect to Figure 1BThe hub 150 may determine whether (a) the instrument state, and / or (b) one or more instrument state parameters (e.g., activation state, temperature, mechanical state, radiant energy, energy usage, etc.) exceed (i) a corresponding instrument state parameter threshold and / or (ii) the amount / degree by which the corresponding parameter threshold is exceeded. The amount / degree by which the corresponding parameter threshold is exceeded can be expressed in a manner appropriate for the corresponding instrument state parameter. The instrument state and other information may be stored / recorded in the instrument state record database 315.

[0080] Figure 3B Aspects of an example block 195 for receiving and processing sensor input 194 are detailed. Figure 3B In block 360, the temperature sensor data 358 may be used to determine the temperature of one or more MIS instruments and / or the amount by which a corresponding temperature threshold of the instrument is exceeded. For example, MIS instrument I_1 may have a corresponding temperature threshold H(I_1). In addition, it may be determined that instrument I_1 is at temperature T(I_1) and / or the amount by which the corresponding temperature threshold H(I_1) is exceeded may be determined. The amount may be expressed as a difference (e.g., T(I_1)-H(I-1)), or as a ratio (e.g., (T(I_1) / H(I_1)), or represented by another mathematical function.

[0081] In some cases (e.g., when a temperature sensor associated with a MIS instrument is active only when the MIS instrument is active, or when the MIS instrument lacks a temperature sensor), the mathematical model 356 can be used to predict the temperature of the instrument after activation, even in cases where direct sensory input (e.g., from a temperature sensor) is limited or unavailable. For example, some energy devices can generate heat by oscillating an instrument (such as a blade) against tissue at various power levels (high, low, etc.). The generator and circuitry can control the power to actuate the instrument and can provide information about the input voltage, input current, resonance state, and resonance frequency at the instrument when activated at high power. At high power, the blade quickly generates heat, which can be used to seal tissue grasped by the instrument. In a low power or inactive state, the instrument cannot generate enough heat for sealing and can be in a cooling mode (e.g., paused for cooling to avoid tissue damage before the device is used again for tissue manipulation, etc.). If the instrument does not have temperature sensing technology in place, then conventionally, the actual temperature of the instrument cannot be determined. Therefore, temperature estimation and safe MIS device use can be entirely dependent on the surgeon's experience level, which can lead to errors.

[0082] In some embodiments, the mathematical model 356 can use MIS instrument physics and / or other sensory inputs such as resonance information 354 (e.g., resonance state and / or resonance frequency) to determine instrument temperature. The mathematical model 356 can determine instrument temperature (e.g., whether the MIS instrument is hot enough for sealing or cool enough to perform tissue manipulation after deactivation without risk of thermal damage), thereby improving surgical safety and surgical efficiency (by eliminating unnecessary long wait times) and surgical focus (e.g., by reducing cognitive overhead associated with peripheral tasks). The mathematical model 356 can be used to determine or estimate instrument temperature even in cases where temperature sensor data is not available.

[0083] When the MIS device is active, energy-related data (e.g., input current, input voltage, device characteristics, etc.) and / or MIS device resonance information 354 (e.g., resonance state, resonance frequency, etc.) can be used to determine the temperature. In addition, the mathematical model 356 can also determine the MIS device temperature based on the resonance information 354 (such as the resonance state and resonance frequency) after the device has been deactivated after high power sealing. For example, the mathematical model 356 can predict the device temperature based on the resonance information 354 (such as the input resonance state and resonance frequency (which is related to the device temperature)).

[0084] In some embodiments, the activation logic data 344 may be used by the activation state box 348 to determine the currently active MIS instrument. In some embodiments, a more specific activation state may be determined. For example, based on the activation logic data and / or sensor input, the detect jaw state box 346 may determine whether the grasper jaws are open or closed.

[0085] In some embodiments, other functional boxes (such as the detect tissue and / or contact state box 342) can detect the MIS instrument contact state and / or tissue state based on inputs from one or more sensors (such as contact sensors, proximity sensors, force sensors, pressure sensors, audio sensors, etc.). For example, in box 342, audio sensing input from a microphone can be used to sense a hissing sound and estimate tissue state (such as the hydration content of the tissue). As another example, audio sensing input from a microphone can be used to sense a heartbeat and / or heart rate. As another example, in box 342, changes in grip force can be compared to the degree of opening / closing of the jaws (e.g., received from box 346), which can be used to determine tissue stiffness and / or determine tissue type. Contact sensors and / or proximity sensors can be used to determine whether the MIS instrument is in contact with the tissue. Force sensors and / or pressure sensors can also be used to determine the distribution of force applied by the MIS instrument. As Figure 3BAs shown in FIG. 1 , in some embodiments, block 195 may also receive video data (as indicated by the dashed line from block 310 to block 195), which may be used during sensor state processing in block 195. For example, in some cases, video information may be used instead of or in addition to sensor data 194 to determine instrument state (e.g., jaws open / closed, etc.).

[0086] Figure 3B The blocks shown in are merely examples, and various other functional blocks may exist based on the procedure being performed and / or the MIS instruments used during the procedure. Figure 3B The additional mathematical models 356 shown in FIG. 1 are merely examples. In general, various mathematical / physical models may be used to determine other instrument states and / or tissue parameters based on the available sensor inputs 194. Figure 3B 356, but a mathematical model (such as mathematical model 356) may be integrated into any functional block to determine and / or refine parameter estimates from sensory input. MIS instrument state / tissue state 197 (which may include parameter values, corresponding parameter thresholds, tissue state, etc.) may be output by block 194.

[0087] refer to Figure 3A In block 196, feedback based on the proximity and MIS device state may be determined and / or generated (e.g., for display / output by HMD 140), and as described above with respect to Figure 1B When the proximity to one or more anatomical structures is below a distance threshold and one or more device state parameters (e.g., activation state, temperature, mechanical state, radiated energy, energy usage, etc.) fall outside of corresponding parameter thresholds, then a visual and / or audible indication and / or a tactile indication may be generated and provided (e.g., via HMD 140) to a medical professional operating the MIS system, as further outlined herein.

[0088] Figure 3C Some example functional blocks associated with block 196 are shown according to certain embodiments.

[0089] In some embodiments, the instrument state indication box 322 can determine the instrument state based on the information in the MIS instrument and the tissue state 197. For example, the instrument state indication box 322 can perform semantic segmentation associated with the registered 3D model 184 to determine pixels / voxels associated with one or more MIS instruments. In some embodiments, pixels / voxels associated with MIS instruments can be enhanced (or marked for enhancement) based on their activation state, parameters (e.g., instrument temperature), and / or the amount or degree to which the instrument state exceeds the corresponding parameter threshold. For example, pixels associated with the instrument can be color-coded or highlighted based on the activation state and temperature of the instrument. Enhancement can be performed using a mask so that other pixels / voxels (i.e., not associated with the MIS instrument) remain unaffected by the operations in box 322.

[0090] In some embodiments, the proximity enhancement block 324 based on the instrument state may determine anatomical structures of interest (e.g., pixels / voxels associated with anatomical structures such as vessels, organs, sensitive tissues, etc.) in the registered 3D model 184 that are within a certain distance threshold of the MIS instrument, where at least one parameter falls outside one or corresponding instrument state parameter thresholds. For example, block 324 may determine a set of MIS instruments having at least one parameter that falls outside one or corresponding instrument state parameter thresholds. In addition, semantic segmentation associated with the registered 3D model 184 may be used to determine a set of pixels / voxels associated with anatomical structures, and the current instrument pose 194 may be used to determine a subset of proximity pixels / voxels that are also associated with an anatomical structure in the set of anatomical structures. The proximity pixels / voxels may be located within a certain distance of the set of MIS instruments. For example, the proximity enhancement block 324 based on the instrument state may determine anatomical structures that fall within a certain distance threshold of the MIS instrument and have a temperature exceeding 50° C. In some embodiments, a subset of proximity pixels of the identified pixels / voxels may be enhanced or marked for enhancement. Enhancement may include color coding and / or highlighting pixels in a subset of close pixels based on distance. Enhancement may also distinguish pixels associated with different anatomical structures. Enhancement may be performed using a mask so that other pixels / voxels (i.e., not determined to be close to instruments in the MIS instrument set) remain unaffected by the operations in box 324. In some embodiments, based on MIS device settings or configurations (e.g., by the surgeon), anatomical structures of interest may be enhanced at different times during surgery or throughout the surgical process. When highlighting the anatomical structure of interest as above, the proximity enhancement box 324 based on instrument status may use additional differentiation enhancement to distinguish pixels in a subset of close pixels.

[0091] In some embodiments, the instrument state warning frame 330 can initiate the display of an alarm / warning based on the instrument state. For example, frame 330 can initiate the display of an alarm / warning or text message related to the instrument state (e.g., jaws open or closed).

[0092] In some embodiments, the risk parameter related warning box 332 can initiate the display of a warning based on the amount or degree to which the device state exceeds the corresponding parameter threshold. For example, the display of an alarm / warning or text message related to one or more risk parameters can be initiated (e.g., the device I_1 temperature exceeds 50°C).

[0093] In some embodiments, the audio feedback box 340 can start or enable the provision of audio feedback. For example, the audio can provide an indication of tissue status (such as tissue hydration). In addition, audio feedback indications can be provided based on the state of the instrument and / or the proximity to the anatomical structure. The audio feedback can indicate that the feedback changes in pitch and / or the number of times the tone is played in a given time period and / or in amplitude / volume based on the state of the instrument and / or the proximity to the anatomical structure. The HMD 140 can be used to provide audio feedback.

[0094] In addition, in some embodiments, the haptic feedback frame 350 can initiate the provision of tactile and / or haptic feedback. For example, a handheld control used by a surgeon / medical professional can be used to provide tactile / haptic feedback.

[0095] In some embodiments, the output of block 196 may include one or more of the following: an anatomical distance map (showing distances to relevant anatomical structures within a certain distance or proximity threshold), a parameter threshold heat map (showing instruments exceeding corresponding parameter thresholds and the amount / degree by which the thresholds were exceeded), an activation state map (indicating active / inactive devices). The above-mentioned maps may be based on the registered 3D model 184.

[0096] In some embodiments, the registered 3D model may include and / or be associated with semantic classifications of pixels / voxels (e.g., an object class associated with each pixel / voxel). Semantic segmentation may include anatomical structure classifications (e.g., vessels, lesions, tissue / organ types, fat, etc.) and / or instrument-related classifications (e.g., instrument type, sheath, etc.). Based on the MIS instrument pose 191, a set of close pixels / voxels in the registered 3D model that are within a certain distance threshold may be determined. Pixels / voxels associated with the anatomical structure of interest in the first proximity set may be identified, and an anatomical structure distance heat map may be determined. The anatomical structure distance heat map may include one or more subsets, each of which includes pixels / voxels associated with the anatomical structure of interest and located within a certain range of the instrument. In some embodiments, a mask corresponding to the pixels / voxels in each subset may be generated so that operations on the pixels / voxels in the corresponding subset may be performed without affecting any other pixels.

[0097] refer to Figure 3A In block 320 , AR enhancements may be projected into the current camera FOV based on the camera and instrument poses and the registered 3D model. In some embodiments, block 320 may output AR enhancements and other warnings / alerts as an RGB alpha stream 222 .

[0098] For example, pixels / voxels that are (i) in the current FoV of the camera, and (ii) associated with an anatomical structure of interest, and (iii) within a certain distance threshold of an instrument, where (iv) the instrument state falls outside a certain corresponding parameter threshold, may be enhanced. In addition, one or more instruments in the current FoV may be enhanced based on instrument state. For example, one or more instruments in the current FoV may be color coded based on their current state (e.g., active, active state outside a parameter threshold, inactive, etc.). In addition, pixels that are associated with an anatomical structure of interest in the current FoV and within a certain distance threshold of an instrument (where the instrument state falls outside a certain corresponding parameter threshold) may be color coded based on their distance from the instrument.

[0099] In block 325, the output of block 320 may be a 3D mixed video stream (such as Figure 2B 214). The 3D video stream 214 may be a synchronized and mixed (eg, using alpha blending) stream that feeds into a stereoscopic video stream (such as Figure 2B 208) is mixed with graphics data (e.g., RGB alpha stream 222), which may include AR enhancements as described herein. The 3D mixed video stream 214 may be output by block 320 to the HMD 140. Although the above examples refer to Figure 2Band 3D video streams, but the disclosed embodiments may also be used with an input 2D video stream (e.g., 202 or 209), where the depth information and the registered 3D model 184 are determined using monocular VSLAM (e.g., based on the input 2D input video stream 202 or 209), and the output of block 320 may also appear as a 2D video stream (such as one of streams 204, 206, or 212).

[0100] In addition, other feedback (e.g., audio, haptics, etc.) may also be provided. In some cases, alpha blending may be used with an input 3D HD video stream (such as Figure 2B The mixed 3D video stream 214 may occur together with the left and right channels of the input video stream 208 in FIG. 1 . In some embodiments, the timestamps associated with the stream 208 and other input streams may be synchronized. In some embodiments, the timestamps associated with the input streams may be synchronized using the Network Time Protocol (NTP) or the Simple Network Time Protocol (SNTP) or by other suitable methods. The mixed 3D video stream 214 may be output to the HMD 140.

[0101] In some embodiments, the RGB alpha stream 222 output by block 330 may be stored / recorded in a data repository 306. In some embodiments, each stream may be buffered. For example, each stream may have at least two frame buffers, one buffer being written to while the other buffer is read / processed, and logic may switch between the two buffers. In some embodiments, the input video stream 202 may be blended with the RGB alpha stream 222 (e.g., by alpha blending).

[0102] FIG. 4A to FIG. 4I An example image frame is shown, which may form part of a hybrid output stream and be viewed by a surgeon or another medical professional. In some embodiments, the example frame may be output stream 214 and / or 212 ( Figure 2B ) or 204 and / or 206( Figure 2A ). In some embodiments, 4A to 4H The image frames in can be displayed on HMD 140. In the case where the mixed output video stream is stereoscopic, 4A to 4H The 3D stereo version (eg, both left and right channels) of the current image of the mixed 3D stereo output video stream may be represented by the medical practitioner 140. 4A to 4H suffixes in the reference numerals refer to the same elements, but at different times and / or activation states.

[0103] Figure 4A An example image frame (eg, current image frame) is shown with an inactive MIS instrument 440-1 below a temperature threshold. Figure 4AThe intraoperative image before the MIS instrument is activated may be reflected. The visual indication may be provided by appropriate visual enhancement of the instrument 440-1, or by not visually enhancing the instrument 440-1 (e.g. Figure 4A ) and / or by visually distinguishing the above-mentioned state of the instrument 440-1 from other possible states. Figure 4A , instrument 440 - 1 is shown without visual enhancement to indicate that (a) it is inactive and (b) is below a temperature threshold.

[0104] Figure 4A Also shown is window 402, which may display patient-related information (e.g., patient name, date of birth, medical condition, etc.) configured by the surgeon or other medical professional. In some embodiments, window 404 may display information about instrument activation (e.g., the number and type of instruments currently active). In addition, window 415 may display system-specific information (e.g., MIS system name, model, etc.), while window 420 may indicate the specific procedure / sub-procedure and / or procedure code of the procedure currently being performed. In some embodiments, window 430 may include information about the relevant status of the instrument (e.g., thermal status and / or jaw open-closed status), and icons may be used to indicate the status. For example, icon 434-1 in window 430 indicates the jaw open state of instrument 440-1, while icon 432-1 indicates that all parameters associated with the instrument fall within their corresponding thresholds. Status bar 438-2 (which may also be color-coded in some cases) indicates that MIS instrument 440-1 is not activated. In some embodiments, text box 436 may be used to indicate the monitored parameters, which is displayed in Figure 4A It says "hot" (for temperature parameters).

[0105] In some embodiments, windows 402, 404, 415, 420 and / or 430 can be user-customizable such that the number of windows, the position of each window, the window slideshow, and the information shown in each window can be set by the user for each surgical type and can be further associated with a user profile.

[0106] Figure 4B Another example image frame (e.g., the current image frame) is shown with an active MIS instrument 440-2 above a temperature threshold. Figure 4B In the present invention, reference and description of some elements previously described have been omitted for simplicity. Figure 4B, the MIS instrument 440-2 is shown with a visual enhancement indicated by a shaded area on the instrument 440-2. For example, the MIS instrument 440-2 can be color coded to indicate activation status and / or temperature, where: (a) the color changes to reflect the temperature of the MIS instrument 440-2, or (b) the color changes based on the degree to which the temperature of the MIS instrument 440-2 exceeds a temperature threshold. For example, temperatures exceeding a temperature threshold can be displayed with a shade of a first color (e.g., red), while temperatures below the threshold can be indicated with a shade of a second color (e.g., yellow), while an activated (but unheated) instrument can be indicated with a shade of a third color (e.g., green), and an inactive instrument (and unheated) can be indicated with a fourth color (e.g., gray).

[0107] like Figure 4B As shown, window 430 includes information about the relevant status of the instrument. In window 430, icon 434-2 has been enhanced to indicate that the instrument is active; icon 432-2 and shaded status bar 438-2 can indicate that the thermal state is below the temperature threshold; and icon 434-2 indicates the "jaw open" state.

[0108] As previously outlined, the CNN-based pose estimation AI / ML model 188 can be used to detect and segment the instrument 440 in the current image frame. The pixels / voxels associated with the instrument can then be enhanced (e.g., using an appropriate mask to prevent any other pixels / voxels from being affected by the enhancement operation) and projected into a graphics frame based on the current MIS system camera pose 189. The instrument-related visual enhancement can be used with windows 402, 404, 415, 420, and 430 to compose a graphics frame that can be synchronized and blended with the input video frame (e.g., using alpha blending) to obtain Figure 4B The mixed image frame shown in .

[0109] Figure 4C Another example image frame (e.g., the current image frame) is shown with an active MIS instrument 440-2 above a temperature threshold. Figure 4C In the present invention, reference and description of some elements previously described have been omitted for simplicity. Figure 4C , MIS instrument 440-2 is shown with a visual enhancement indicated by a cross-hatched area on instrument 440-3. For example, MIS instrument 440-3 may be color coded to indicate activation status and / or temperature, where: (a) the color changes to reflect the temperature of MIS instrument 440-3, or (b) the color changes based on the degree to which the temperature of MIS instrument 440-3 exceeds a temperature threshold.

[0110] like Figure 4CAs shown, window 430 includes information about the relevant status of the instrument. In window 430, icon 434-3 has been further enhanced to indicate that the thermal state exceeds the threshold and also indicates the "jaw open" state. In addition, icon 432-3 indicates an alarm / warning related to the "thermal" parameter shown in text box 436 because the temperature of MIS instrument 440-3 exceeds the temperature threshold. In addition, in some embodiments, status bar 438-3 (which can be color-coded to reflect the MIS instrument temperature) can be used to indicate the current thermal (or another monitoring) state of instrument 440-3.

[0111] As previously outlined, the CNN-based pose estimation AI / ML model 188 may be used to detect and segment the instrument 440-3 in the current image frame. The pixels / voxels associated with the MIS instrument 440-3 may then be enhanced (e.g., using an appropriate mask and without affecting any other pixels / voxels) and projected onto a graphics frame based on the current MIS system camera pose along with windows 402, 404, 415, 420, and 430. The graphics frame may be synchronized and blended with the input video frame to obtain Figure 4C The mixed image frame shown in .

[0112] Figure 4D Another example image frame (e.g., the current image frame) is shown with an active MIS instrument 440-4 exceeding a temperature threshold along with an adjacent anatomical structure. Figure 4D In the present invention, reference and description of some elements previously described have been omitted for simplicity. Figure 4D As shown in , the MIS instrument 440-4 is shown with visualization enhancements indicated by the cross-hatched area on the instrument 440-4. Figure 4D , the cross-hatching indicates that instrument 440-4 is activated and the temperature of instrument 440-4 exceeds a predetermined temperature threshold. As previously outlined, in some embodiments, the CNN-based pose estimation AI / ML model 188 can be used to detect and segment instrument 440 in the current image frame and determine the MIS system camera pose 189 and the MIS instrument pose 191.

[0113] In addition, if Figure 4D As shown, one or more identified anatomical structures 452-1 and 452-2 that are proximate to the MIS instrument 440-2 are also enhanced. The proximate anatomical structures may be located within a certain distance threshold of the MIS instrument 440-4. Figure 4D, the visually enhanced proximal anatomical structure shown is a blood vessel. The anatomical structure may be identified based on semantic segmentation. For example, semantic segmentation may be associated with the registered 3D model 184. As previously outlined, semantic segmentation may include anatomical structure classification (e.g., blood vessels, lesions, tissue / organ types, fat, etc.). The proximity of the identified anatomical structure to the MIS instrument 440-2 may be determined based on the 6-DoF MIS instrument pose 191 using the registered 3D model 184. For example, pixels / voxels associated with the anatomical structure (or portion of the anatomical structure) may be determined (e.g., based on the registered 3D model 184) to be proximal because they are within a certain distance threshold of the current MIS instrument pose 191. In some embodiments, portions of the current image (such as image portions 450-1 and 450-2) may be color coded based on their distance from the MIS instrument 440-4.

[0114] In some embodiments, an alarm or warning message may be displayed in window 460 to alert the surgeon that one or more anatomical structures are in proximity to a heating instrument that is outside of a safe range (eg, above a temperature threshold).

[0115] The proximate pixels / voxels (e.g., associated with the proximate anatomical structure 452) and / or the image portion 450 may then be enhanced (e.g., using an appropriate mask). Visual enhancements of the MIS instrument 440-4, the proximate anatomical structure 452, and / or the proximate image portion 450 may be projected onto the graphics frame based on the current MIS system camera pose 189 along with the windows 402, 404, 415, 420, 430, and 460. The graphics frames may be synchronized and blended (e.g., using alpha blending) to obtain Figure 4D The mixed image frame shown in , which can be part of a mixed output video stream.

[0116] Figure 4E Another example intraoperative image (similar to Figure 4A ), which may be part of a mixed video stream. For example, Figure 4E It can reflect the intraoperative images before the MIS instrument is activated. Figure 4E , (a) patient related information is shown in window 402, (b) the MIS instrument activation status in window 404 indicates that no (zero) instruments are activated, (c) system information is indicated as "Harmonic 1100" in window 415, the procedure is indicated as "Advanced Hemostasis" in window 420 with procedure code 5, and window 430 shows that the "thermal" parameter 436 is within the temperature threshold, as indicated by the check icon 432-1 and the green bar / bar 438. In addition, icon 434-2 indicates that the jaw status of instrument 440-1 is open. Figure 4E, instrument 440 - 1 is not visually enhanced because instrument 440 - 1 is not activated and has not exceeded a monitored parameter threshold (eg, a temperature threshold).

[0117] Thus, in the mixed video stream output, the surgeon or medical practitioner may see (e.g., displayed on HMD 140): (1) patient-related information (e.g., patient details in window 402, etc.), (2) MIS instrument activation status information (e.g., number of activated MIS instruments in window 404, etc.), (3) instrument-related information (e.g., make / model, etc. in window 415), (4) surgery-related information (e.g., surgery type, code, etc. in window 420), (5) MIS instrument monitoring parameters (e.g., "heat" in text box 436), MIS instrument status in window 430 (e.g., icon 432-1 and color bar 438 showing that the monitored parameters - such as temperature - are within thresholds, icon 434-2 "indicating jaw open" status). In addition, in some embodiments, (6) a visual indication may be provided that (a) MIS instrument 440-1 has not been activated, and (b) all monitored parameters (e.g., heat) fall within corresponding thresholds (e.g., MIS instrument temperature is below a temperature threshold). The visual indication may be provided by appropriate visual reinforcement of the instrument 440-1, or by not visually reinforcing the instrument 440-1 (e.g., Figure 4E The information in (1) to (6) above may form part of a graphics frame that may be synchronized and alpha-blended (e.g., in Figure 3A 325 in the example) to an input video stream (e.g., Figure 2A 202, or Figure 2B 208 and / or 209) in order to obtain an output mixed video stream (eg, Figure 2A 204 and / or 206, or Figure 2B 212 and / or 214).

[0118] Figure 4F Another example intraoperative image (similar to Figure 4B ), where MIS device 440-2 has been activated. Figure 4F In the present invention, reference and description of some elements previously described have been omitted for simplicity. Figure 4F As shown, window 404 includes text stating "1 Instrument Activated," indicating that MIS instrument 440-2 has been activated.

[0119] Figure 4FThe MIS instrument monitoring parameters (e.g., temperature) are also shown within the thresholds in window 430. In addition, in some embodiments, a visual indication may be provided that (a) MIS instrument 440-1 has been activated, and (b) all monitored parameters (e.g., heat) fall within corresponding thresholds (e.g., the MIS instrument temperature is below the temperature threshold). The visual indication may be provided by appropriate visual reinforcement of MIS instrument 440-2. Figure 4F , the MIS instrument 440-2 is now highlighted in green to indicate the new state (e.g., the MIS instrument 440 is active and all monitored parameters are within corresponding parameter thresholds). In some embodiments, auditory and / or other feedback may also be provided along with the indication of the instrument state. In some embodiments, a CNN-based pose estimation AI / ML model 188 may be used to detect and segment the instrument 440 in the current image frame. The pixels / voxels associated with the instrument may then be enhanced (e.g., using a mask to prevent other pixels / voxels from being affected by the enhancement operation) and projected into the current graphics frame based on the current MIS camera pose 189. In addition to the instrument-related visual enhancement, the current graphics frame may also include windows 402, 404, 415, 420, and 430. The current graphics frame may be blended with a current input image frame (e.g., a portion of an input video stream) to obtain Figure 4F The mixed image frame shown in .

[0120] Figure 4G Another example intraoperative image (similar to Figure 4C ), wherein the MIS device 440-4 has been activated and a temperature threshold associated with a thermal / temperature parameter has been exceeded. Figure 4G In the present invention, references and descriptions of some elements previously described have been omitted for simplicity.

[0121] Figure 4G A MIS instrument monitoring parameter (e.g., temperature) exceeding a threshold value is also shown in window 430. In addition, in some embodiments, a visual indication may be provided that (a) the MIS instrument 440-4 has been activated, and (b) at least one monitored parameter (e.g., heat) exceeds a corresponding threshold value (e.g., the MIS instrument temperature is above a temperature threshold value). The visual indication may be provided by appropriate visual reinforcement of the MIS instrument 440-4. Figure 4F, MIS instrument 440-2 is now highlighted in red to indicate a new state (e.g., MIS instrument 440 is active and at least one monitored parameter exceeds a corresponding parameter threshold). In some embodiments, a CNN-based pose estimation AI / ML model 188 may be used to detect and segment instrument 440 in the current image frame, determine MIS system camera pose 189 and MIS instrument pose 191. Pixels / voxels associated with the instrument may then be enhanced (e.g., using a mask to prevent other pixels / voxels from being affected by the enhancement operation) and projected into the current graphics frame based on the current MIS system camera pose. In addition to instrument-related visual enhancements, the current graphics frame may also include windows 402, 404, 415, 420, and 430. The current graphics frame may be blended with a current input image frame (e.g., a portion of an input video stream) to obtain Figure 4G The mixed image frame shown in . In some embodiments, auditory and / or other feedback may also be provided to indicate the status of the instrument.

[0122] Figure 4H Another example intraoperative image (similar to Figure 4D ), wherein the MIS instrument 440-4 has been activated and a temperature threshold associated with a thermal / temperature parameter has been exceeded. Figure 4H Also shown are visualization-enhanced anatomical structures (e.g., 452-1 and 452-2) proximate to the instrument. The proximate anatomical structures may be located within a certain distance threshold of the MIS instrument 440-4. Figure 4H In the present invention, references and descriptions of some elements previously described have been omitted for simplicity.

[0123] like Figure 4H As shown in , the MIS instrument 440-4 is shown with a visual enhancement indicated by the red color of the instrument 440-4. Figure 4H , the red color of instrument 440-4 indicates that instrument 440-4 is activated and the temperature of instrument 440-4 exceeds a predetermined temperature threshold. As previously outlined, in some embodiments, the CNN-based pose estimation AI / ML model 188 can be used to detect and segment instrument 440 in the current image frame and determine the MIS system camera pose 189 and the MIS instrument pose 191.

[0124] In addition, if Figure 4H As shown, one or more identified anatomical structures 452-1 and 452-2 that are proximate to the MIS instrument 440-4 are also enhanced. The proximate anatomical structures may be located within a certain distance threshold of the MIS instrument 440-4. Figure 4H, the visually enhanced proximate anatomical structures shown are blood vessels. Anatomical structures may be identified based on semantic segmentation and associated with the registered 3D model 184. The proximity of the identified anatomical structures to the MIS instrument 440-2 may be determined based on the 6-DoF MIS instrument pose 191 using the registered 3D model 184. For example, pixels / voxels associated with anatomical structures (or portions of anatomical structures) may be determined (e.g., based on the registered 3D model 184) to be proximate because they are within a certain distance threshold of the current MIS instrument pose 191. Figure 4H 4, portions of the current image, such as image portions 450-1 and 450-2, are colored red to indicate their proximity to the MIS instrument 440-4. In addition, window 460 also displays a warning message related to the proximity of the anatomical structure, indicating "low distance to risk structure". The MIS instrument visual enhancement and the anatomical structure visual enhancement can be projected into the current graphics frame based on the current MIS camera pose 189. In addition, the current graphics frame can also include windows 402, 404, 415, 420, 430 and 460. The current graphics frame can be blended with the current input image frame (e.g., a portion of the input video stream) to obtain Figure 4H The mixed image frame shown in .

[0125] Fig. 4I Another example intraoperative image (similar to Figure 4H ), where MIS instrument 440-5 has been deactivated, as indicated in window 404-5, which shows "0 Instrument Activations". However, the temperature of instrument 440-5 remains above the temperature threshold. For example, MIS instrument 440-5 may be cooled (or in the process of being cooled) before being used for another purpose (e.g., grasping tissue). Fig. 4I Also shown are visualization-enhanced anatomical structures (e.g., 452-1 and 452-2) proximate to the instrument. The proximate anatomical structures may be located within a certain distance threshold of the MIS instrument 440-4. Fig. 4I In the present invention, references and descriptions of some elements previously described have been omitted for simplicity.

[0126] In some embodiments, even after deactivation, a temperature sensor or mathematical model (e.g., based on the resonant state and resonant frequency) can be used to determine the temperature of the instrument 440-5, and visual enhancement can be applied to the instrument 440-5 as long as the temperature of the instrument 440-5 remains above a temperature threshold. Fig. 4IAs shown, instrument 440-5 is colored red to indicate that its temperature is above the temperature threshold. In addition, window 460 continues to display a warning message related to anatomical structure proximity, indicating "low distance to risk structure". In addition, icon 432-5 (!) and status bar 438-5 (red) indicate that the temperature of MIS instrument 440-5 exceeds the temperature threshold (icon 438-5). In some embodiments, as the MIS instrument cools, the visualization enhancement (e.g., via color coding) can reflect the lower temperature.

[0127] Figure 5 A flow chart of an example method 500 for intraoperative instrument tracking and visualization is shown. In some embodiments, the method 500 may be used to enhance portions of intraoperative images for data visualization. The method 500 may reduce the cognitive load of medical professionals performing minimally invasive surgery and / or other medical procedures and improve safety. In some embodiments, the method 500 may be performed by a MIS system, which may include components such as a camera, one or more instruments, one or more sensors, a display, and at least one processor coupled to the above components. In some embodiments, the display may include an HMD 140 and / or a monitor 135. In some embodiments, the camera may be a stereo camera, and the HMD 140 may be capable of displaying stereo images. In some embodiments, at least one processor may form a part of a hub 150, and the method 500 may be performed by the hub 150. In some embodiments, the hub 150 may form a part of a MIS system. In some cases, the method 500 may be performed in real time during surgery. For example, augmentation and / or other visual, auditory, and / or tactile enhancements perceived by a user (eg, a surgeon) may occur synchronously with a real-time input video stream and with minimal delay.

[0128] In box 510, a current instrument pose (e.g., 6 degrees of freedom (6-DoF)) associated with a minimally invasive surgical (MIS) instrument may be determined. The current instrument pose may be determined based on a real-time intraoperative input video stream captured by a camera, wherein the input video stream may include a current input image frame that includes an image of the instrument (or a portion of the instrument). In some embodiments, the input video stream may be stereoscopic and / or include depth information. The depth information may be obtained (actively) from a depth sensor that forms part of the camera, or (passively) from a stereo camera. In some embodiments, the camera may be monocular (non-stereoscopic) and depth information is obtained from the input video stream using monocular visual simultaneous localization and mapping (VSLAM) technology. In some embodiments, the current instrument pose may be determined based on a 6-DoF current camera pose associated with a MIS system camera. In some embodiments, the current instrument pose may be determined using a trained machine learning (ML) model. For example, an AI / ML model may use a convolutional neural network (CNN) to detect and segment instruments in the current image before determining the instrument pose.

[0129] In block 520, an activation state of the instrument and at least one parameter value associated with the instrument may be determined. For example, the instrument activation state may be an activated state or an inactivated state. In some embodiments, a standby mode may be included in which sensory input from the instrument is available. For example, an instrument in standby mode may continue to output sensory information even when the instrument has been deactivated. As another example, an activated instrument may provide sensory input in standby mode without the functionality associated with the actuated instrument.

[0130] For example, the parameters monitored or determined may include thermal parameters or temperature parameters associated with the instrument, and the thermal parameters may be associated with corresponding high temperature parameter thresholds. As a further example, the parameters monitored or determined may include radiant energy parameters associated with corresponding radiant energy thresholds; or input energy parameters based on the input energy of the instrument associated with corresponding input energy parameter thresholds; or some combination of the above. Other parameters that may be monitored may also include applied force, force and / or pressure distribution, tissue hydration status, blood pressure, heart rate, etc. Each parameter monitored may be associated with a threshold value, which may be customized by a medical professional based on the type of surgery, tissue type, and / or instrument.

[0131] These parameters may be determined based on information received from various sensors that may be part of or coupled to the MIS system. For example, the temperature of the instrument may be determined based on at least one of: information from a temperature sensor; or an input voltage and input current of the instrument; or a resonant state and resonant frequency associated with the instrument; or a combination thereof. In addition, although the parameters may be received from the instrument or associated with a sensor coupled to the MIS system, the parameters may reflect tissue state (e.g., hydration level, elasticity, deformation, etc.) or patient state (e.g., blood pressure, blood flow, oxygenation level, etc.). Tissue-related parameters or patient state parameters may affect instrument activation and / or use. In some embodiments, parameter thresholds may be determined based on (or reflect) safety considerations.

[0132] In block 530, one or more first intraoperative graphic visualization enhancements may be determined based on (a) the activation state of the instrument, and (b) a comparison of at least one parameter value to at least one corresponding parameter threshold. In some embodiments, the first intraoperative graphic visualization enhancement may be applied to the current graphic frame 535 (e.g., being composed). The parameters / parameter values ​​compared to the corresponding thresholds may include any of the parameters described herein (including the instrument, tissue state, patient state, etc.), and the first intraoperative graphic visualization enhancement may be based on the parameters / parameter values ​​and / or the thresholds.

[0133] The first intraoperative graphical visualization enhancement may be based on: the activation state of the instrument; or at least one parameter value (e.g., temperature); or the degree to which at least one parameter value falls outside of at least one corresponding parameter threshold (e.g., the difference or ratio of the instrument temperature to the temperature threshold); or a combination thereof. For example, the first intraoperative graphical visualization enhancement may include: color coding the instrument based on at least one parameter value (e.g., the color may change with the instrument temperature); or color coding the instrument based on the degree to which at least one parameter value falls outside of at least one corresponding parameter threshold (e.g., based on the temperature difference or ratio relative to the temperature threshold); or color coding the instrument based on the activation state of the instrument (e.g., whether it is active, inactive, standby, etc.); or a combination thereof. In some embodiments, the first intraoperative graphical visualization enhancement may include initiating the generation of an alarm message when one or more parameters associated with the instrument fall outside of one or more corresponding parameter thresholds.

[0134] In some embodiments, the first visualization enhancement may include one or more information windows (such as those described above with respect to FIG. 4A to FIG. 4I167) and / or a cloud-based service (e.g., from a cloud-based server coupled to the cloud 167).

[0135] In block 540, display of a blended image frame 545 included in a blended output video stream (e.g., one or more of output streams 204 and / or 206, or 212 and / or 214 and / or 216) may be initiated. For example, the blended image frame 545 may be obtained by blending a current input image frame with a current graphics frame 535. In some embodiments, blending may include: projecting one or more first intraoperative graphics visualization enhancements into a current camera field of view (FoV) based on a camera pose to obtain (or compose) the current graphics frame 535; and synchronizing and blending the current input image frame with the current graphics frame 535 to obtain the blended output frame 545. In some embodiments, the blended output video stream may be transmitted to a stereo head mounted display (e.g., HMD 140) and / or another monitor (e.g., monitor 135). The blended output video stream with all visual enhancements (including any AR enhancements) may be displayed on one or more appropriate devices of appropriate devices (e.g., HMD 140 and / or monitor 135).

[0136] In some embodiments, method 500 may also include intraoperatively identifying one or more proximal anatomical structures (e.g., blood vessels or other organs / tissues) located within a distance threshold of the instrument in the current input image frame based on the current instrument pose and the 3-dimensional (3D) anatomical model. For example, the 3D anatomical model used to identify the anatomical structure may include semantic segmentation and may be obtained by registering a semantically segmented intraoperative 3D anatomical model obtained from the input video stream with a semantically segmented preoperative 3D anatomical model (e.g., obtained from a preoperative imaging device or image storage subsystem).

[0137] In some embodiments (e.g., upon identification of an approaching anatomical structure), when at least one parameter value falls outside of at least one corresponding parameter threshold (e.g., instrument temperature exceeds a temperature threshold), auditory feedback may be initiated, wherein the auditory feedback changes based on the distance of the instrument to at least one of the one or more approaching anatomical structures. The auditory feedback may change in pitch and / or in the number of times the tone is played in a given time period and / or in amplitude / volume based on the instrument state and / or the proximity of the instrument to the anatomical structure. In some embodiments, the identified anatomical structures may be limited to critical anatomical structures (e.g., which may be sensitive and / or considered more likely to cause surgical complications when inadvertently mishandled).

[0138] In some embodiments (e.g., when identifying an approaching anatomical structure), method 500 may also include determining one or more second intraoperative graphical visualization enhancements for one or more approaching anatomical structures. The second intraoperative graphical visualization enhancements may be applied to the current graphical frame prior to initiating display of the blended image frame. For example, the second intraoperative graphical visualization enhancement may be used to compose the current graphical frame prior to blending with the current input image frame such that the blended output reflects both the first visualization enhancement and the second visualization enhancement.

[0139] In some embodiments, one or more second intraoperative graphic visualization enhancements applied to the proximate anatomical structures may be based on: at least one parameter value falling outside of at least one corresponding parameter threshold (e.g., when the temperature exceeds a temperature threshold); or based on the corresponding distance of one or more proximate anatomical structures from the instrument (e.g., proximity-based enhancement of anatomical structures); or a combination thereof. For example, one or more second intraoperative graphic visual enhancements may include: increasing the color intensity of one or more proximate anatomical structures; or increasing the contrast of one or more proximate anatomical structures relative to the background (or relative to any other visible anatomical structure); or highlighting one or more proximate anatomical structures; or color coding a sub-portion of the current graphic frame that includes the proximate anatomical structures; or color coding one or more proximate anatomical structures in the current graphic frame based on the corresponding distance of one or more proximate anatomical structures from the instrument (e.g., proximity-based color mapping); or a combination thereof. In addition, in some embodiments, one or more second intraoperative graphic visualization enhancements may include: initiating the generation of an alarm message based on the presence of the proximate anatomical structure when one or more parameters associated with the instrument fall outside of one or more corresponding parameter thresholds. The second intraoperative visualization enhancement may include information about FIG. 4A to FIG. 4I Example anatomical enhancements are described.

[0140] Figure 6An example system 600 is shown, which illustrates the associated software architecture supporting intraoperative tracking and visualization according to certain embodiments disclosed herein. In some embodiments, Figure 6 Some or all of the software architecture shown may be implemented on the MIS system and / or hub 150.

[0141] like Figure 6 As shown, the software architecture 600 may include: (a) a communication and peripheral services layer 660, which may support interfaces and communications with external devices; (b) a connection layer 650, which may enable the capture of incoming information and formatting and packaging of information for output via the communication and peripheral services layer 660; (c) a data layer, which may enable intraoperative event logging, event notification, and data management, including storage and uploading of data to online web-based services and / or cloud-based services and / or portals; (d) a multimedia processing and synchronization layer 630, which may process and synchronize incoming streams. For example, the incoming streams may include video from a stereo endoscope, video from an intraoperative imaging device (e.g., an ultrasound probe, a camera in an operating room, etc.), sensor data, etc. In some embodiments, the multimedia processing and synchronization layer 630 may appropriately encode (if necessary), process the incoming streams, and synchronize the incoming streams to a common reference time.

[0142] The UI layer 610 (which may allow a user to authorize the connection of one or more devices) configures coupled devices, selects streams, selects AI models to run, selects preoperative and intraoperative image sources, configures surgical guidance (e.g., to be provided to medical professionals 120 and / or support personnel), specifies notifications (e.g., to be provided to personnel when certain events occur), configures remote coaching, etc. For example, the UI layer 610 may be used to select and configure a trained AI / ML model 182 for semantic segmentation, a trained AI / ML model 188 for pose estimation, etc. for use. The AI / ML models and other models may be context specific to the instrument being used, the procedure being performed, and / or the anatomical environment.

[0143] The service layer 620, which may be modular, may provide the ability to enable advanced applications (including AI-based applications) to be used in an intraoperative environment. For example, the service layer 620 may provide a DICOM service function 626 that may catalog and list available sources for relevant preoperative images and may retrieve and store selected images / models locally. For example, the service layer 620 may catalog, list, and retrieve relevant preoperative 3D models 180 (which may be registered with intraoperative 3D models to obtain registered 3D models 184). The service layer 620 may also provide various medical team assistance services. For example, the service layer 620 may catalog and list available AI models for selection and operation on real-time input video streams. AI / ML models may perform or assist in 3D registration, semantic segmentation, and the like, such as by identifying and marking pixels / voxels associated with blood vessels or other anatomical structures, marking pixels / voxels associated with MIS instruments, and the like. The service layer 620 may also include an event service function 622 that can record and log events, and cloud support services that can allow access to cloud-based applications, including AI / ML models and other models, data service providers, etc., and also allow uploading of data for training / refining AI models, etc. The service layer 620 may also provide various other functions, such as a UI service function that can allow interaction between the UI layer and the service.

[0144] In some embodiments, the team assistance service 624 can facilitate human guidance to set up surgical guidelines for a particular surgery preoperatively, and then use these guidelines intraoperatively to follow the steps outlined in the guidelines. The machine learning model can be trained offline based on recorded videos and surgical settings for a particular surgery to automatically identify the phases and steps outlined as part of the surgery. In some embodiments, the trained model can then process the live endoscopic video feed in real time to automatically identify the steps and phases, and automatically advance the on-screen guide to the next step in the surgical guide by changing the content.

[0145] Team support services 624 may also include notification-related services to facilitate messaging from within the operating room (OR) to personnel outside the OR based on the current status of the surgery. The messages may include one or more notifications, such as: the surgery has begun, the surgery may be completed or exceed the scheduled time, the OR is ready, etc. These notifications help with planning and more efficient use of resources.

[0146] In some embodiments, the team assistance service 624 may include configuring and enabling image viewing and image-based planning functions, which may facilitate the processing, review, analysis, communication, and media exchange of multi-dimensional digital images acquired preoperatively from imaging devices (e.g., from computed tomography (CT), magnetic resonance imaging (MRI), diagnostic ultrasound, X-ray, positron emission tomography (PET) scans, etc.). The image viewer and image-based planning may also facilitate preoperative surgical planning and intraoperative display of digital images by enabling 3D rendering and visualization and picture archiving and communication system (PACS) integration. In some embodiments, the image viewing and image-based planning functions may also make the above information available to authorized personnel on a network or on a web page, and facilitate the deployment of an online digital surgical portal.

[0147] The integrated anatomy, 3D modeling, and patient outcome simulation capabilities provided by the team assistance service 624 can use available preoperative and intraoperative 3D models and facilitate the display and manipulation of 3D models for preoperative planning and intraoperative reference, and can improve the visualization of anatomical structures and tumors and facilitate the performance of patient-specific and surgery-specific simulations.

[0148] The multimedia processing and synchronization layer 630 may facilitate the management and synchronization of connected input devices, enable data, video and audio fusion and processing, and also provide services for data management (storage, retrieval, tagging, indexing, etc.).

[0149] The data layer 640 may facilitate scalable multi-service and device communications, provide notifications to all stakeholders involved in the intraoperative workflow, and facilitate systematic recording and storage of intraoperative information over the hospital network 165 and / or through a cloud-based data service provider.

[0150] The connectivity layer 650 may include functionality to facilitate multimedia data exchange and capture multimedia input received in various formats and through various hardware interfaces such as video interfaces, audio interfaces, custom interfaces, etc., and may also output information to a connected display and tablet computer 145.

[0151] The communications and peripheral interface layer 660 may include software for configuring and managing WLAN (e.g., Wi-Fi), WPAN (e.g., Bluetooth), WWAN (e.g., 5G), Ethernet, USB, custom peripheral port connections, etc.

[0152] like Figure 6As shown, the communication and peripheral interface layer 660 can facilitate interaction with various external devices and peripheral devices. For example, the video input and output 670 can use an HDMI interface and / or an SDI interface to receive video from devices such as an endoscope 670-1, an ultrasound device 670-2, etc. The video input and output 670 can also output the video to an external display 135 ( Figure 6 not shown).

[0153] The wired and wireless networking interfaces 160 may include a wired network connection such as an Ethernet connection 160-3 that may be used to: connect to servers, storage, services, etc. on the hospital network 170; facilitate data exchange with the digital surgical portal; and may provide connectivity to cloud-based storage and / or services 175. The wireless network 160-1 may include an interface for WLAN connectivity with an external WLAN device 174 such as a tablet computer 145, which may allow a user to interact with, configure, and manage the system 600. In some embodiments, a local (e.g., operating room) wireless network 160 may be provided by the hub 150, which may be used by authorized devices by accessing the network to communicate with the hub 150 and / or between themselves (e.g., improving bandwidth, latency, and reducing reliance on the hospital network). The wireless network 160-1 may include an interface for WPAN connectivity and communication with external WPAN (e.g., Bluetooth) devices such as audio devices, smart instruments, robotic instruments, sensors, etc.

[0154] Peripheral interfaces 680 may include standard interfaces, such as USB, which may provide connectivity to an OR camera, provide power, and support surgical instruments (such as surgical scissors, sealers, etc.). Custom interfaces 683 may include manufacturer-specific interfaces (such as connectors for insufflation devices, electromechanical devices such as pumps, aspirators, irrigators, etc.) Figure 6 not shown).

[0155] Although the present disclosure has been described in conjunction with specific embodiments for illustrative purposes, the present disclosure is not limited thereto. Various adjustments and modifications may be made to the present disclosure without departing from the scope of the present disclosure. Therefore, the essence and scope of the appended claims should not be limited by the foregoing description.

Claims

1. A processor-implemented method, include: determining a current instrument pose associated with a minimally invasive surgery (MIS) instrument based on a real-time intraoperative input video stream captured by a MIS camera, wherein the input video stream includes a current input image frame including an image of the instrument; determining an activation state of the instrument and at least one parameter value associated with the instrument; determining one or more first intraoperative graphic visualization enhancements based on (a) an activation state of the instrument and (b) a comparison of the at least one parameter value to at least one corresponding parameter threshold, wherein the first intraoperative graphic visualization enhancements are applied to a current graphic frame; as well as Display of a mixed image frame included in a mixed output video stream is initiated, wherein the mixed image frame is obtained by mixing the current input image frame with the current graphics frame.

2. The method according to claim 1, in, The one or more first intraoperative graphic visualization enhancements are determined based on at least one of: the activation state of the instrument; or the at least one parameter value; or the extent to which the at least one parameter value falls outside of the at least one corresponding parameter threshold; or a combination thereof.

3. The method according to claim 1, in, The one or more first intraoperative graphical visualization enhancements include at least one of the following: color coding the instrument based on the at least one parameter value; or color coding the instrument based on the extent to which the at least one parameter value falls outside of the at least one corresponding parameter threshold; or color coding the instrument based on the activation state of the instrument; or A combination of them.

4. The method according to claim 1, further comprising: include: intraoperatively identifying one or more proximal anatomical structures located within a distance threshold of the instrument in the current input image frame based on the current instrument pose and a 3-dimensional (3D) anatomical model; as well as When the at least one parameter value falls outside of at least one corresponding parameter threshold, generation of auditory feedback is initiated, wherein the auditory feedback varies based on a distance of the instrument to at least one of the one or more proximate anatomical structures.

5. The method according to claim 1, further comprising: include: intraoperatively identifying one or more proximal anatomical structures located within a distance threshold of the instrument in the current input image frame based on the current instrument pose and a 3-dimensional (3D) anatomical model; as well as One or more second intraoperative graphical visualization enhancements are determined for the one or more proximate anatomical structures, wherein the second intraoperative graphical visualization enhancements are applied to the current graphical frame prior to initiating the display of the hybrid image frame.

6. The method according to claim 14, in, The one or more second intraoperative graphical visualization enhancements include at least one of the following: Increase the intensity of one or more colors close to the anatomical structure; or increasing the contrast of one or more proximal anatomical structures relative to the background; or highlighting the one or more proximal anatomical structures; or color coding a sub-portion of the current graphics frame including the proximate anatomical structure; or color coding the one or more proximate anatomical structures in the hybrid image frame based on their corresponding distances from the instrument; or A combination of them.

7. The method according to claim 6, in, The one or more second intraoperative graphic visualization enhancements are determined based on at least one of: the at least one parameter value falling outside of the at least one corresponding parameter threshold; or the corresponding distance of the one or more proximal anatomical structures from the instrument; or a combination thereof.

8. The method according to claim 6, in, Determining the one or more second intraoperative graphic visualization enhancements includes: When the one or more parameters associated with the instrument fall outside of the one or more corresponding parameter thresholds, generation of an alarm message is initiated.

9. The method according to claim 1, in, The at least one parameter and the associated at least one corresponding threshold value include: the temperature of the device associated with a corresponding high temperature threshold; or a radiant energy parameter associated with a corresponding radiant energy threshold; or an input energy parameter based on the input energy of the instrument associated with a corresponding input energy threshold; or A combination of them.

10. The method according to claim 9, in, The temperature of the device is determined based on at least one of: Information from a temperature sensor; or the input voltage and input current of the device; or a resonant state and resonant frequency associated with the instrument; or A combination of them.

11. The method according to claim 1, in, The 3D anatomical model is semantically segmented and obtained by registering a semantically segmented intraoperative 3D anatomical model obtained from the input video stream with a semantically segmented preoperative 3D anatomical model.

12. The method according to claim 1, in, The current instrument pose is determined based on a 6-DoF current camera pose associated with the MIS camera, and blending the current input image frame with the current graphics frame comprises: projecting the one or more first intraoperative graphics visualization enhancements into a current camera field of view (FoV) based on the camera pose to obtain the current graphics frame; and The current input image frame is synchronized and mixed with the current graphics frame.

13. The method according to claim 12, in, The current instrument pose is determined using a trained machine learning (ML) model, wherein the ML model uses a convolutional neural network (CNN) to detect and segment the instrument in the current image prior to determining the instrument pose.

14. The method according to claim 1, further comprising: include: The mixed output video stream is transmitted to a stereoscopic head mounted display.

15. A minimally invasive surgery (MIS) system, include: A camera for obtaining a real-time intraoperative video stream, instrument, Memory, and a processor coupled to the camera, the instrument, and the memory, wherein the processor is configured to: determining a current instrument pose associated with the instrument based on the real-time intraoperative input video stream, wherein the input video stream includes a current input image frame, the current input image frame including an image of the instrument; determining an activation state of the instrument and at least one parameter value associated with the instrument; determining one or more first intraoperative graphic visualization enhancements based on (a) an activation state of the instrument and (b) a comparison of the at least one parameter value to at least one corresponding parameter threshold, wherein the first intraoperative graphic visualization enhancements are applied to a current graphic frame; as well as Display of a mixed image frame included in a mixed output video stream is initiated, wherein the mixed image frame is obtained by mixing the current input image frame with the current graphics frame.

16. The MIS system according to claim 15, in, To determine the one or more first intraoperative graphical visualization enhancements, the processor is configured to: determining the one or more first intraoperative graphical visualization enhancements associated with the instrument based on the activation state of the instrument; or determining one or more first intraoperative graphical visualization enhancements associated with the instrument based on the at least one parameter value; or determining one or more first intraoperative graphical visualization enhancements associated with the instrument based on the extent to which the at least one parameter value falls outside of the at least one corresponding parameter threshold; or Perform a combination of them.

17. The MIS system according to claim 15, in, The processor is further configured to: intraoperatively identifying one or more proximal anatomical structures located within a distance threshold of the instrument in the current input image frame based on the current instrument pose and a 3-dimensional (3D) anatomical model; as well as When the at least one parameter value falls outside of at least one corresponding parameter threshold, generation of auditory feedback is initiated, wherein the auditory feedback varies based on a distance of the instrument to at least one of the one or more proximate anatomical structures.

18. The MIS system according to claim 15, in, The processor is further configured to: intraoperatively identifying one or more proximal anatomical structures located within a distance threshold of the instrument in the current input image frame based on the current instrument pose and a 3-dimensional (3D) anatomical model; as well as One or more second intraoperative graphical visualization enhancements are determined for the one or more proximate anatomical structures, wherein the second intraoperative graphical visualization enhancements are applied to the current graphical frame prior to initiating the display of the hybrid image frame.

19. The MIS system according to claim 18, in, The one or more second intraoperative graphical visualization enhancements are based on at least one of: the at least one parameter value falling outside of the at least one corresponding parameter threshold; or the corresponding distance of the one or more proximate anatomical structures from the instrument; or a combination thereof.

20. A non-transitory computer readable medium comprising instructions to configure a processor to: determining a current instrument pose associated with a minimally invasive surgery (MIS) instrument based on a real-time intraoperative input video stream captured by a MIS camera, wherein the input video stream includes a current input image frame including an image of the instrument; intraoperatively identifying one or more proximal anatomical structures located within a distance threshold of the instrument in the current input image frame based on the current instrument pose and a 3-dimensional (3D) anatomical model; determining one or more intraoperative graphical visualization enhancements in the current input image frame based on (a) an activation state of the instrument, and (b) a determination that one or more parameters associated with the instrument fall outside of one or more corresponding parameter thresholds, and (c) the one or more adjacent anatomical structures; and Display of a mixed image frame included in a mixed output video stream is initiated, wherein the mixed image frame is obtained by mixing the current input image frame with a graphic visualization enhancement.

Images