System and method for surgical telementoring and training with virtualized telestration and haptic holograms, including metadata tagging, encapsulation and saving multi-modal streaming medical imagery together with multi-dimensional [4-d] virtual mesh and multi-sensory annotation in standard file formats used for digital imaging and communications in medicine (DICOM)

Abstract

The invention relates generally to a medical apparatus and method of using the same for receiving and transmitting streaming medical imagery and audio signals in real time, and allowing remote operators to annotate and telestrate with same. The invention acquires streaming medical imagery and audio signals through a telestreamer input device, allowing users to electronically collaborate, generally by telestrating, annotating, and sketching image overlays on streaming medical imagery. Video images of streaming imagery data displayed on a monitor are superimposed onto a virtual mesh projected via computer graphics. The vertices of the mesh move according to equations of motion based on a computational physics engine. Virtual tools are projected above the mesh via computer graphics. These virtual tools interact with the virtual mesh according to the physics engine. The superposition of the video images onto the virtual mesh makes it appear that points within the video image are moving in a realistic manner and reacting to the virtual tools with a realistic response. The invention allows for recursive superposition of mesh layers, also known as ‘surgi-skins’, and creation of a multi-layered virtual mesh. Multi-layered surgi-skins synthesized from multi-modal streaming medical imagery and saved together with multi-dimensional [4-D] virtual mesh and multi-sensory annotation in single file format as DICOM files are also known as ‘DICOM mesh’ or ‘haptic holograms’.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS[0001]This application claims the benefit of:[0002]U.S. Provisional Application No. 61 / 745,383 filed Dec. 21, 2012 entitled “SYSTEM AND METHOD FOR SURGICAL TELEMENTORING USING VIRTUALIZED TELESTRATION,” naming as inventors, G. Anthony Reina and James Omer L'Esperance, which is incorporated herein by reference in its entirety.[0003]This application may be related to the following commonly assigned and commonly filed U.S. patent applications, each of which is incorporated herein by reference in its entirety:[0004]1. U.S. patent application Ser. No. US 2011 / 0282141 A1 entitled “METHOD AND SYSTEM OF SEE-THROUGH CONSOLE OVERLAY”, naming as inventors Itkowitz et al., filed on 17 Nov. 2011.[0005]2. U.S. patent application Ser. No. US 2011 / 0282140 A1 entitled “METHOD AND SYSTEM OF HAND SEGMENTATION AND OVERLAY USING DEPTH DATA”, naming as inventors Itkowitz et al., filed on 17 Nov. 2011.[0006]3. U.S. patent application Ser. No. US 2010 / 0164950 A1 entit...

Images