Systems and methods for detecting abnormal tissue using vascular features

Abstract

An imaging system includes a camera, a display, and a processor. The camera has color video acquisition capability and is mounted to the distal end of an interventional instrument that can be inserted into an object, the camera providing image frames for imaging the vascular system of the object, each image frame including a plurality of pixels providing corresponding signals. The processor is programmed to: receive signals; amplify changes in at least one of color and motion of the signals corresponding to the plurality of pixels; respectively determine spatial phase variability, frequency, and signal characteristics of at least some of the amplified signals corresponding to the plurality of pixels; identify pixels indicating abnormal vascular distribution based on spatial phase variability, frequency, and / or signal characteristics; create a vascular map corresponding to each, wherein each vascular distribution map includes a portion of the object having abnormal vascular distribution; and operate the display to display each vascular distribution map.

Description

Background Art

[0001] Detection of tumors using vascularization is currently used in medical imaging. That is, tumors can be detected by analyzing the blood vessels within the tumor because these tumor blood vessels are often dilated with irregular shapes, disordered connections, and leaky capillaries compared to normal blood vessels. For example, dynamic imaging of contrast agents injected into the venous system can be used to detect areas of increased vascularity, angiogenesis, and vascular leakage, all of which indicate cancerous tumors. However, cameras and optical imagers have not yet been used for cancerous tumor detection, even though video-linked surgical procedures are becoming more and more common. Minimally invasive surgical procedures often achieve improved patient outcomes.

[0002] Differentiation of normal tissue from abnormal tissue during surgery is often a challenge for surgeons. This differentiation can be particularly difficult for non-palpable / non-tactile tumors that are often seen in brain cancer (e.g., neurosurgery), lung cancer (e.g., ground-glass opacities on computed tomography (CT) scans), liver cancer (e.g., diffuse liver disease), and other cancers in soft tissue. Moreover, during robotic surgery, surgeons cannot use tactile features to distinguish tumors. Difficulties may also arise when surgical planning information acquired from CT scans, magnetic resonance imaging (MRI) scans, and / or positron emission tomography (PET) scans fails to help determine the location of the tumor (or lesion) due to tissue deformation. For example, this often occurs during thoracic surgery due to pneumothorax and during neurosurgery due to loss of cerebrospinal fluid (CSF).

[0003] Furthermore, the use of surgical robots and / or steerable devices for vision-assisted minimally invasive surgery limits direct visual access to potentially abnormal tissue. Examples of surgical robots include multi-arm systems such as Robots, or soft robots, such as Medrobotics Robotic systems. These robotic systems are controlled by the surgeon (or user) using various interface mechanisms, including hand controllers, input handles, and / or heads-up display (HUD) devices (e.g., head-mounted display (HMD) devices) for operating the robotic system and image displays for capturing endoscopic video and displaying various control modes of the robotic system.

[0004] It would therefore be desirable to be able to accurately detect tumors during surgery or other interventional procedures using images of vascularization obtained from a camera insertable within the body.

[0005] Amir-Khalili, Alborz et al., "Automatic segmentation of occluded vasculature via pulsatile motion analysis in endoscopic robot-assisted partial nephrectomy video" (Medical Image Analysis 25.1 (2015): 103-110) describes the automatic segmentation of occluded vasculature. It shows that there is a method that can adjust the phase-based video magnification to "marking rather than magnification".

[0006] US 2017 / 0294015 A1 describes a method for analyzing OCT angiography. In an embodiment, a binary skeleton, a vascular skeleton, and a vascular perimeter map are generated from vascular image data and then processed to detect multiple vascular metrics (e.g., vascular area density) and flow impairment. These metrics can be used to detect and evaluate vascular abnormalities. Summary of the invention

[0007] According to an illustrative embodiment, an imaging system includes a camera, a display, and a processor. The camera has a color video acquisition function and is mounted to a distal end of an interventional instrument that can be inserted into an object. The camera provides image frames for imaging the vascular system of the object, each image frame including a plurality of pixels providing corresponding signals. The processor is programmed to: receive the signal; amplify the change of at least one of the color and motion of the signal corresponding to the plurality of pixels to provide an amplified signal; respectively determine at least one of the spatial phase variability, frequency, and signal characteristics of at least some of the amplified signals corresponding to the plurality of pixels; identify pixels indicating abnormal vascular distribution in the plurality of pixels based on at least one of the spatial phase variability, frequency, and signal characteristics of at least some of the amplified signals; use the identified pixels to create a vascular map corresponding to each of the at least one of the spatial phase variability, frequency, and signal characteristics of at least some of the amplified signals, each vascular distribution map including a portion of the object having abnormal vascular distribution; and operate the display to display each vascular distribution map.

[0008] According to another illustrative embodiment, a method is provided for identifying abnormal vascularity indicative of abnormal tissue in a subject using images from a camera having color video acquisition capability and mounted to a distal end of an interventional instrument insertable into a subject. The method includes: receiving signals corresponding to a plurality of pixels in each of a plurality of image frames provided by a camera of a target portion in the subject for imaging the vasculature of the target portion; and respectively amplifying changes in at least one of color and motion of the signals corresponding to the plurality of pixels in each of the plurality of image frames to provide amplified signals; respectively determining at least one of spatial phase variability, frequency, and signal characteristics of at least some of the amplified signals corresponding to the plurality of pixels; identifying pixels indicating abnormal vascularity in the plurality of pixels in each of the plurality of image frames based on at least one of spatial phase variability, frequency, and signal characteristics of at least some of the amplified signals; using the identified pixels to create a vascularity map corresponding to each of at least one of spatial phase variability, frequency, and signal characteristics of at least some of the amplified signals, each vascularity map including a portion of the subject having abnormal vascularity; and displaying each vascularity map. The method may also include combining at least two of the vessel maps corresponding to each of at least one of spatial phase variability, frequency, and signal characteristics of at least some of the amplified signals to create an aggregated vessel map; and displaying the aggregated vessel map.

[0009] According to another illustrative embodiment, a computer readable medium is provided, the computer readable medium containing software executable by a computer processor to use images from a camera having color video acquisition capability and mounted to a distal end of an interventional instrument insertable into a subject to identify abnormal vascular distribution indicative of abnormal tissue in a subject. The computer readable medium includes: receiving code for receiving signals corresponding to a plurality of pixels in each of a plurality of image frames of a target portion in the subject provided by the camera via the interventional instrument to image the vascular system of the target portion; amplifying code for respectively amplifying at least a change in motion of the signals corresponding to the plurality of pixels in each of the plurality of image frames to provide amplified signals; determining code for respectively determining at least one of spatial phase variability, frequency, and signal characteristics of at least some of the amplified signals corresponding to the plurality of pixels; identifying code for identifying the plurality of pixels in each of the plurality of image frames based on at least one of spatial phase variability, frequency, and signal characteristics of at least some of the amplified signals in the amplified signals. a pixel indicative of abnormal vascular distribution in the pixels; a creating code for creating, using the identified pixels, a vascular map corresponding to each of at least one of spatial phase variability, frequency, and signal characteristics of at least some of the amplified signals, each vascular map including a portion of the object having abnormal vascular distribution; a combining code for providing an aggregated vascular map by combining at least two of the vascular maps corresponding to each of at least one of spatial phase variability, frequency, and signal characteristics of at least some of the amplified signals; and a display code for causing the aggregated vascular map to be displayed on a display, the displayed aggregated vascular map indicating the portion of the object having abnormal vascular distribution by enhanced contrast with the portion of the object having normal vascular distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present invention will be more easily understood according to the detailed description of the following exemplary embodiments considered below in conjunction with the accompanying drawings. Like numbered elements in these figures are equivalent elements or perform the same function. If the functions are equivalent, the elements previously discussed will not need to be discussed in the following figures.

[0011] Figure 1A An example of healthy (or normal) tissue with normal vascularity is shown.

[0012] Figure 1B Example unhealthy (or cancerous) tissue with abnormal vascularity, such as in a tumor, is shown.

[0013] Figure 2 is a simplified schematic block diagram illustrating a vasculature imaging system for imaging vasculature in a region of interest, according to a representative embodiment.

[0014] Figure 3 is a simplified schematic diagram illustrating a magnified image and corresponding signals captured in multiple image frames according to a representative embodiment.

[0015] Figure 4 is a simplified schematic diagram illustrating the use of a magnified image and the frequency and signal strength of corresponding signals captured in multiple image frames in accordance with a representative embodiment.

[0016] Figure 5 is a simplified schematic diagram illustrating a phase shift between a pixel within a zoomed-in image and a corresponding phase signal captured in one or more image frames, according to a representative embodiment.

[0017] Figure 6 is a diagram showing a typical embodiment of the present invention. Figure 3-5 Simplified schematic diagram of vessel maps obtained by the described techniques, which are combined to provide a polymerized vessel map with enhanced contrast.

[0018] Fig. 7A is a schematic block diagram showing a surgical robotic system according to an illustrative embodiment of the invention.

[0019] Figure 7B is a schematic block diagram showing a surgical robotic system according to an illustrative embodiment of the invention. DETAILED DESCRIPTION

[0020] Embodiments of the present invention will now be described more fully below with reference to the accompanying drawings in which embodiments of the present invention are shown. However, the present invention can be implemented in different forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided as teaching examples of the present invention.

[0021] As described above, intraoperatively distinguishing abnormal tissue from normal tissue for resection and biopsy is a common challenge for surgeons performing various interventional procedures. According to various embodiments, vascular information extracted after creating one or more Euler video magnifications indicating vascular abnormalities or excessive vascular abnormalities can be used to help surgeons distinguish abnormal tissue (e.g., cancerous tumors) from normal tissue. In addition, various embodiments can be used to screen suspected melanoma (skin cancer) lesions to improve accuracy and potentially aid in early diagnosis.

[0022] In general, the hallmarks of cancer include six biological capabilities acquired during the multi-step development of a tumor. These hallmarks constitute an organizing principle for rationalizing the complexity of neoplastic diseases. Induced angiogenesis is one of the six hallmarks. Angiogenesis enables a tumor including cancer cells to have its own blood supply so that it can grow. Unlike normal blood vessels (e.g., blood vessels that extend through parts of the body without interference from cancerous tumors), tumor blood vessels illustrate various abnormalities, such as expansion with irregular shapes, having chaotic connections and including capillary leaks, resulting in irregular blood flow. Such abnormalities can be referred to as excessive vascularization. Figure 1A and 1B Examples of healthy (or normal) tissue with normal vascularity and unhealthy or abnormal (eg, cancerous) tissue with abnormal vascularity, such as in a tumor, are shown respectively.

[0023] According to various embodiments, camera-based detection of pulsatile flow in video is used to detect these and other anomalies. The camera can be mounted to an interventional instrument or tool, such as an endoscope or catheter, or mounted to a robotic arm to assist in performing an interventional procedure on an object. Throughout this disclosure, "mounted to" means connected to, integrated with, and / or contained therein. Often, images including videos of seemingly stationary parts of the body actually contain subtle changes that are invisible to the naked eye. However, these subtle changes can be measured via the use of algorithms to amplify the changes, for example, including the use of Euler video magnification. For example, it is known that a human pulse can be measured from color changes or small movements of pulsatile flow caused by a pulse captured by a conventional video, and such measurements are called remote photoplethysmography (rPPG). See, for example, "Eulerian Video Magnification for Revealing Subtle Changes in the World" by Wu et al. (ACM Transactions on Graphics Vol. 31, No. 4 (Proc. SIGGRAPH, 2012)).

[0024] Typically, a series of images can be acquired from a data stream provided by a camera of the vascular system, where the camera can be mounted to an interventional instrument and the data stream may have been available during the performance of an interventional procedure. Changes in color and / or motion are amplified in images indicated by pixel signals from a camera that images the vascular system (e.g., including normal and abnormal tissue). Up to three vascular maps can then be created and / or combined to distinguish between areas of normal tissue and abnormal tissue. As described below, the vascular map can include, for example, a frequency map, a signal characteristic map, and a phase variability map based on information different from the signal corresponding to the image of the target portion or ROI.

[0025] An aggregated vessel map combining two or more of the three vessel maps may be generated for enhanced contrast, i.e., better contrast than any single vessel map, thereby better distinguishing regions of normal and abnormal tissue. The vessel map and / or aggregated vessel map may be displayed using an augmented reality (AU) device including a heads-up display (HUD), for example, to enable easy visualization so that a surgeon can maintain focus on a surgical field where abnormal tissue (e.g., a tumor) is visually apparent in the vessel map and / or aggregated vessel map. One of the vessel maps may be enhanced by training a fully convolutional neural network (FCN) to segment abnormal regions in a target portion or ROI.

[0026] Camera-based vascular information can also be used to screen for suspected vascular melanoma lesions to improve accuracy and potentially aid in early diagnosis. That is, a camera (in this case, located externally) can provide an image of a lesion that is readily apparent with vasculature. Again, changes in color and / or motion from the pixel signals are amplified, and the pixel signals image the vasculature (e.g., including normal and abnormal tissue). A vascular map can then be created and / or combined to distinguish between areas of normal and abnormal tissue within and around the lesion. The vascular information can be combined with other visual cues indicative of malignancy (e.g., morphology, color, size, and temporal changes) to further improve the accuracy of camera-based screening.

[0027] For example, for a patient with many lesions, information from a color video can help clinicians triage higher risk lesions for biopsy. Early diagnosis of melanoma is a key determinant of a positive prognosis and survival. Moreover, surgical removal of skin lesions in the early stages is typically minimally invasive, while treatment of melanoma in later stages (e.g., aggressive and difficult-to-treat metastatic melanoma) results in a poorer prognosis. Convolutional neural networks (CNNs) trained with data other than conventional images can further improve accuracy and potentially aid in early diagnosis of melanoma. Such camera-based determination of vascularity can be provided at a subject's home and / or in a dermatology clinic, as the determination can be accomplished using a camera, a processor, and a memory, as described below with reference to Figure 2 In particular, while the present disclosure focuses on imaging and vascularity determination using a camera for interventional procedures, it should be understood that the process of identifying normal and abnormal vasculature, including the determination and display of vascular maps and / or aggregated vascular maps, is substantially the same for determining the vascularity of vascular melanoma lesions.

[0028] Moreover, an AR device can be provided to a user (e.g., a surgeon) to control the robot through the user's body movements (e.g., in addition to using the user's hands to manipulate conventional manual controls). The user can identify and select a target portion for the robot based on a live image received by the AR device from a camera. The live image can be an endoscopic image, for example, provided by a camera mounted to the distal end of an endoscope, for example, the endoscope can be operated by a dedicated endoscope controller; or provided by a forward-looking camera, for example, the forward-looking camera can be operated by the robot. For example, the AR device can include a heads-up display (HUD) (AR-HUD device), which displays (one or more) images on a display in a head piece worn by the user. The AR-HUD device also includes one or more sensors configured to detect movements of the user, such as head movements or eye movements, wherein the detected movements are processed to select a target portion in the surgical site shown in the image on the display for the robot.

[0029] The use of live images of the user and head motion and / or eye movement detection improves the usability of the robot by simulating the experience during conventional surgery, where the user moves his or her head and / or directs eye movements toward the area (target portion) that the user is focusing on, while retaining the improved dexterity of the surgical robot. In addition, live images can be used to provide a data stream of the vasculature, as described above. The color and / or motion from these images are amplified to amplify the vasculature signals and provide a vascular map depicting areas of abnormal tissue (e.g., tumors).

[0030] It should be understood that the terms used herein are only for the purpose of describing specific embodiments and are not intended to be limiting. Any defined terms are in addition to the technical and scientific meanings of the defined terms as generally understood and accepted in the technical field of this teaching. References to "one embodiment" or "embodiment" and other variations thereof in the specification mean that the specific features, structures, characteristics, steps, etc. described in conjunction with the embodiment are included in at least one embodiment of this teaching. Therefore, the phrases "in one embodiment" or "in an embodiment" that appear in various positions throughout the specification and the appearance of any other variations do not necessarily all refer to the same embodiment.

[0031] First, it should be noted that the images may include two-dimensional (2D) or three-dimensional (3D) images obtained as a stream of video or still images. The 2D or 3D images may be obtained via a camera mounted to the distal end of the endoscope or via a forward-looking camera provided at the distal end of a robotic arm (e.g., as an end effector). Moreover, the images may be live images captured by the camera during a minimally invasive procedure. In various configurations, the images may be captured by multiple cameras. Other medical imaging, such as images obtained by X-ray, ultrasound, and / or magnetic resonance, may be incorporated during surgery, for example, for a wider view of the surgical site and surrounding area, but may not necessarily be used directly to detect abnormal vascular distribution during an interventional procedure.

[0032] Figure 2 is a simplified schematic block diagram illustrating a vasculature imaging system for imaging vasculature in a region of interest, according to a representative embodiment.

[0033] refer to Figure 2 , an imaging system 200 is provided for detecting abnormal tissue within a subject using vascular features. The imaging system 200 includes a camera 212, which is attached to the distal end of an interventional instrument 242, such as an end effector of an endoscope, a catheter, and / or a robotic arm. In an embodiment, the camera 212 has color video acquisition capabilities and can provide digital still images and / or videos. The camera 212 can be inserted into the subject to image a target portion (e.g., or ROI) to determine the associated vascular distribution. Analysis of vascular distribution enables differentiation between normal tissue and abnormal tissue (e.g., cancerous tumors), as described below. More specifically, the camera 212 provides a vasculature signal corresponding to a plurality of pixels, including color image data and / or motion data (when the camera 212 provides a digital video signal).

[0034] Without departing from the scope of the present teachings, the camera 212 may be a monocular camera for 2D images or a stereo camera that captures 3D images. For example, a stereo camera or a pair of cameras may include multiple lenses or a lens assembly with a separate sensor for each lens that forms an image on a digital detector array (e.g., a charge coupled device (CCD) imaging array, a complementary metal oxide semiconductor (CMOS) imaging array, etc.). The camera 212 may have color video capabilities, for example, by having an imaging array with pixels that are sensitive to red, green, and blue light (or another set of colors that substantially spans the visible spectrum). The camera 212 may optionally include other typical features, such as a built-in flash (not shown) and / or an ambient light sensor (not shown) for setting the exposure time.

[0035] The imaging system 200 also includes a processing unit 230, a display 220, a user input 226, and an input / output circuit 238. The processing unit 230 includes at least one processor 232 and at least one memory 234. The memory 234 is a non-transitory storage medium, such as, for example, a random access memory (RAM), a read-only memory (ROM), a disk, and / or a solid-state memory. The memory 234 includes instructions that, when executed by the processor 232, cause the processor 232 to receive and process vasculature signals of an image provided by the camera 212. In general, the processing identifies pixels of the camera 212 indicating abnormal vasculature distribution in the target portion, the abnormal vascular distribution corresponding to abnormal tissue. The processing also includes creating a vascular map depicting abnormal vascular distribution with improved contrast, for example, using various techniques including frequency, signal characteristics, and phase variability discussed below. The vascular map can be superimposed on the image of the target portion to help the user identify and treat (e.g., remove, biopsy) the abnormal tissue. The processing unit 230 is shown as having a wired connection to the camera 212 via the interventional instrument 242 (and the I / O circuit 238). However, it should be understood that the processing unit 230 can be connected to the camera 212 via a wireless communication link or network (e.g., Wi-Fi or 4G) without departing from the scope of the present teachings.

[0036] The processed image and / or vascular map can be displayed on a display 220 (e.g., an LCD display, an OLED display, and / or a touch-sensitive screen). A user (e.g., a surgeon) can control the display 220 via a user input 226 (e.g., a mouse, a keyboard, a trackball, and / or a touch-sensitive screen). For example, the user can isolate portions of the image, zoom in and out, etc. When the camera 212 provides a video signal, for example, the user can pause at various frames and move forward and backward in the video stream. In this way, the user can overlay (one or more) vascular maps on selected image frames on the display 220. The user can also manually control the positioning of the camera 212 and / or the interventional instrument 242 via the user input 226, as discussed further below.

[0037] In an embodiment, the user input 226 and the display 220 may be incorporated into the AR-HUD device. The AR-HUD device may include a left-eye and a right-eye display as the display 220, but alternatively, the display 220 may be a single large window across both eyes. In some examples, the camera 212 provides a "first-person view", aligning the AR content with the actual view seen through the (one or more) displays of the AR-HUD device. In some examples, the AR-HUD device may be configured as a helmet, headband, glasses, goggles, or other suitable embodiments to be worn on the user's head.

[0038] As described above, a vascularity map may be created to identify pixels indicative of abnormal vascularity from among a plurality of pixels in the camera 212. Two or more of any combination of vascularity maps may be combined to provide an aggregated vascularity map with enhanced contrast of region(s) having abnormal vascularity. The vascularity map and / or aggregated vascularity map may be converted into a portion of an object including abnormal tissue (e.g., a tumor). Once identified, the portion of the object having abnormal vascularity may be removed from the object or otherwise contained or eliminated by various treatments, such as, for example, radiation therapy, cryotherapy, and high intensity focused ultrasound (HIFU) therapy.

[0039] To obtain a vascular map, changes in the signal corresponding to the vasculature image provided by the camera 212 are amplified by, for example, the processor 232. In an embodiment, an Euler video amplification algorithm may be used to amplify the signal changes, but other amplification techniques may be incorporated without departing from the scope of the present teachings. Thus, subtle signal changes in still images or video images provided by the camera 212 that are not easily detected from the original video images are enhanced.

[0040] Figure 3 is a simplified schematic diagram illustrating a magnified image and corresponding signals captured in multiple image frames according to a representative embodiment.

[0041] refer to Figure 3 , image stream 310 includes a set of image frames 311, 312, and 313 provided by camera 212 over time, as indicated by the t-axis. Image frames 311, 312, and 313 contain images of a target portion of an object. Representative pixels P1 and P2 of the pixel array are shown in image frame 311 (and are also included in subsequently acquired image frames 312 and 313, although they are not shown in the same image frame). Figure 3 The orientation shown is obscured from view by image frame 311). Below image stream 310 is a graph showing signals 301 and 302 as a function of time, signals 301 and 302 corresponding to pixels P1 and P2, respectively.

[0042] With respect to image frames 311, 312, and 313, subtle signal changes over time for representative pixels P1 and P2 are amplified. In various embodiments, amplifying the signal includes, for example, amplifying the motion changes using known Eulerian motion amplification indicated by arrow 315. See, for example, "Eulerian Video Magnification for Revealing SubtleChanges in the World" by Wu et al. (ACM Transactions on Graphics Vol. 31, No. 4 (Proc. SIGGRAPH, 2012)). For example, changes in pixel values ​​over time are amplified in a spatial multi-scale manner to effectively provide motion amplification. The use of Eulerian motion amplification can, for example, rely on exaggerating motion by amplifying temporal color changes at fixed locations.

[0043] The resulting image stream 320 includes images with magnified variations, which for the purposes of discussion may be referred to as "magnified images". The image stream 320 includes signals from pixels with enlarged or magnified motion variations, which for the purposes of discussion may be referred to as "magnified pixels", thereby enabling easier detection. Typically, variations in signals 301 and 302 (together with signals corresponding to other pixels in the pixel array) are amplified over a desired time period, thereby obtaining signals 301' and 302', as described below. The image stream 320 includes a set of magnified image frames 321, 322, and 323 (with corresponding magnified pixels) corresponding to image frames 311, 312, and 313, respectively. Pixels P1' and P2' corresponding to pixels P1 and P2, respectively, are included in the magnified image frames 321, 322, and 323 (although only shown in the magnified image frame 321). It should be understood that signals with magnified variations corresponding to other pixels of the pixel array are provided, but for ease of description, this specification focuses on representative pixels P1' and P2'.

[0044] Below the image stream 320 is a graph showing signals 301' and 302' as a function of time, where signals 301' and 302' correspond to pixels P1' and P2', respectively. Due to signal variation amplification, the traces and / or waveforms of signals 301' and 302' show subtle variations that are not apparent in signals 301 and 302. Thus, in this example, it is apparent that signal 301' is substantially sinusoidal, with smooth transitions within the expected frequency range, consistent with an arterial waveform (e.g., pulsating or corresponding to a cardiac cycle or pulse rate). Thus, with respect to blood flow through normal tissue (normal vascular distribution), the portion of the image provided by pixel P1' is consistent with normal physiology. In contrast, signal 302' is not sinusoidal, but has an irregular pattern. Thus, the portion of the image provided by pixel P2' does not have a waveform or frequency consistent with normal physiology. In other words, based at least on frequency and shape, pixel P2' can be identified as indicating abnormal vascular distribution (consistent with blood flow through abnormal tissue and / or tumors). In other words, the portion of the image shown by pixel P2 ′ corresponds to abnormal tissue, such as, for example, a cancerous tumor.

[0045] Figure 4 is a simplified schematic diagram illustrating the use of a magnified image and the frequency and signal strength of corresponding signals captured in multiple image frames in accordance with a representative embodiment.

[0046] refer to Figure 4 , the image stream 320 includes enlarged image frames 321, 322, and 323 provided over time, and enlarged pixels P1' and P2' included in the enlarged image frames 321, 322, and 323, as described above. Below the image stream 320 is a graph showing a signal 301' and another graph showing a signal 302' as a function of time, the signals 301' and 302' corresponding to pixels P1' and P2', respectively, as shown in reference Figure 3 As discussed. Figure 4 As shown, a fast Fourier transform (FFT) is performed on each of the signals 301' and 302' to decompose the signals 301' and 302' into their respective frequency components. That is, performing an FFT on the signal 301' in the time domain provides a frequency spectrum 401 in the frequency domain corresponding to the amplified pixel P1', and performing an FFT on the signal 302' in the time domain provides a frequency spectrum 402 in the frequency domain corresponding to the amplified pixel P2'. The frequency spectrum 401 includes a peak 403 within a first frequency range f1, and the frequency spectrum 402 includes a peak 404 within a second frequency range f2 that is higher than the first frequency range f1. For example, the frequency of the horizontal axis (frequency) can be scaled to the number of heart beats per minute.

[0047] By identifying peaks 403 and 404, it can be determined whether the components of the peaks (e.g., the apex and the area under each peak 403 and 404 in frequency ranges f1 and f2) are within a physiologically feasible passband PW, which is a frequency passband corresponding to a credible range of the subject's pulse rate. For example, the credible range may have a lower limit of 40 beats / minute and an upper limit of 200 beats / minute, but the credible range may vary between other minimum and / or other maximum values ​​in reality without departing from the scope of the present teachings. For example, decomposing the signal into their respective frequency components and identifying the largest peak component can be achieved by windowing in the frequency domain followed by slope inversion and local peak search.

[0048] refer to Figure 4 , it is apparent that peak 403 is within the physiologically feasible passband PW, and thus indicates that the portion of the image provided by pixel P1' is consistent with normal physiology with respect to blood flow through normal tissue (normal vascularity). Peak 404 is outside the physiologically feasible passband PW (i.e., at a higher frequency), and thus indicates that the portion of the image provided by pixel P2' is consistent with abnormal physiology with respect to blood flow through abnormal tissue (abnormal vascularity).

[0049] The FFT may be performed on signals corresponding to all pixels of the image stream 320, or on at least an appropriate sampling of the pixels of the image stream 320, to determine whether the peak in the resulting frequency spectrum is within the physiologically feasible passband PW, as discussed above with reference to pixels P1' and P2'. A frequency map 330 may then be derived based on this information as indicated by arrow 415. A normal portion 331 of the frequency map 330 includes the locations of pixels (including pixel P1') having a frequency spectrum with a peak within the physiologically feasible passband PW. An abnormal portion 333 of the frequency map 330 includes the locations of pixels (including pixel P2') having a frequency spectrum with a peak outside the physiologically feasible passband PW. As can be seen, the abnormal portion 333 covers the vicinity where the enlarged pixel P2' is located. The larger the abnormal portion 333, the more pixels it covers, indicating a larger area of ​​abnormal vascular distribution, and therefore a larger area of ​​abnormal tissue. Of course, the resulting frequency map may display multiple portions indicating abnormal vascular distribution without departing from the scope of the present teachings.

[0050] Additionally, the characteristics of the signals 301' and 302' and the changes in the underlying frequency components may be extracted by performing an FFT on each of the signals 301' and 302' and finding the area under the curve (AUC) and / or the full width at half maximum (FWHM) of the strongest frequency components in the resulting frequency spectra 401 and 402, respectively. Figure 4, AUC 405 for peak 403 is greater than AUC 406 for peak 404. A larger AUC 405 indicates a more normal vascular distribution associated with pixel P1' compared to pixel P2'. To use AUC 405 to determine whether peak 403 actually indicates the presence of abnormal tissue, AUC 405 can be compared to a predetermined AUC threshold, as a pixel can have a smaller corresponding AUC compared to another pixel in the array without necessarily indicating the presence of abnormal tissue. For example, a range of "normal" AUCs can be determined empirically.

[0051] The signal characteristic graph 430 (in Figure 6 ). A normal portion 431 of the signal characteristic graph 430 includes the locations of pixels (including pixel P1') having an AUC within the normal range (e.g., below a predetermined AUC threshold), and an abnormal portion 433 of the signal characteristic graph 430 includes the locations of pixels (including pixel P2') having an AUC outside the normal range (e.g., above a predetermined AUC threshold). Basically, the same type of graph can be derived using FWHM values. As can be seen, the abnormal portion 433 covers the vicinity of where abnormal tissue is located, while the normal portion 431 covers the vicinity of where normal tissue is located. The signal characteristic graph 430 can be similar in appearance to the frequency graph 330 because it is derived from the same FFT of signals 301' and 302'. Of course, the signal characteristic graph can display multiple portions indicating abnormal vascular distribution without departing from the scope of the present teachings.

[0052] In particular, Figure 4 In the example shown, pixel P2' can already be identified as indicating abnormal tissue because its peak 403 is outside the physiologically feasible passband PW, as described above. Therefore, determining the characteristics of the corresponding signal 302' to determine whether the pixel P2' shows normal tissue or abnormal tissue can be used as an alternative method. Alternatively, these two methods can be used and combined for redundant confirmation (e.g., creating an aggregate enhanced contrast map, as described below with reference to Figure 6 discussed).

[0053] Moreover, higher level features can be extracted from signals 301' and 302' indicating signal shape and pulsatility using a long short-term memory (LTSM) network or a recurrent neural network (RNN) to understand the corresponding shape of the signal in the time domain. Each signal value will be treated as a word. The LTSM network and the RNN take into account the timing information. The LTSM network can be particularly useful for achieving associations between more distantly separated time points in signals 301' and 302'. As described above, signal 301' is substantially sinusoidal with smooth transitions consistent with normal tissue in the expected frequency range, while signal 302' is non-sinusoidal, but has an irregular pattern consistent with abnormal tissue.

[0054] Figure 5 is a simplified schematic diagram showing phase shifts between neighboring pixels within an amplified image and corresponding phase signals captured in one or more image frames according to a representative embodiment. Phase shifts (or phase differences) between neighboring pixels among multiple pixels of the amplified image stream (or streams) are identified and compared to each other to determine spatial phase variability. Phase shifts of the main sinusoidal signal between neighboring pixels are used to understand the directionality and organization of flow in the vasculature.

[0055] refer to Figure 5 , the enlarged image frame 521 includes representative enlarged pixels P3' and P4', which are neighboring (e.g., adjacent) pixels in the pixel array. The enlarged image frame 521 can be obtained by amplifying changes in the signal corresponding to the vascular system image provided by the camera 212, such as discussed above with respect to the enlarged image frame 321. However, for the purpose of explanation, in Figure 5 501 , the amplified pixels P1' and P2' are not shown in the figure because they are not neighboring pixels. Below the amplified image frame 521 is a graph showing signals 503 and 504 as a function of time, where signals 503 and 504 correspond to pixels P3' and P4', respectively. Signals 503 and 504 are offset from each other, indicating a phase shift.

[0056] The phase shift information is collected to provide a plurality of phase maps corresponding to the magnified image frames, as shown by Figure 5 , which corresponds to, for example, enlarged image frame 521. That is, a series of images in an image frame can be compressed into multiple phase maps so that multiple enlarged image frames can be provided over a period of time. Phase map 522' corresponds to representative phase map 522 and depicts illustrative arrows or vectors that indicate phase shifts between various neighboring pixels (e.g., pixels P3' and P4'), and flow direction can be estimated.

[0057] The mean and standard deviation of the phase shifts are determined for the phase maps 522, 523, and 524. Typically, the standard deviation corresponding to pixels showing regions with irregular vascular flow (indicative of abnormal tissue) is higher than the standard deviation corresponding to pixels showing regions with regular vascular flow. A phase variability map 530 can then be derived based on the phase maps 522, 523, and 524. A normal portion 531 of the phase variability map 530 identifies the locations of pixels having phase shifts with low standard deviations, and an abnormal portion 533 of the phase variability map 530 identifies the locations of pixels having phase shifts with high standard deviations. The standard deviation can be compared to a predetermined threshold to identify a value at which the standard deviation begins to indicate irregular vascular flow. For example, a range of "abnormal" and / or "normal" standard deviations can be determined empirically. Of course, the resulting standard deviation map can display multiple portions indicating abnormal vascular distribution without departing from the scope of the present teachings.

[0058] More specifically, multiple phase maps (e.g., three or four) are acquired for different time periods, such as phase maps 522, 523, and 525. The mean and standard deviation at each pixel of phase maps 522, 523, and 525 are determined, and the information from phase maps 522, 523, and 524 is combined into phase variability map 530. Thus, information about the degree of irregularity of the flow (if any), the directionality of the flow, and the organization of the vasculature can be obtained.

[0059] Figure 6 is a diagram showing a typical embodiment of the present invention. Figure 3-5 Simplified schematic diagram of vessel maps obtained by the described techniques, which are combined to provide an aggregated vessel map with enhanced contrast.

[0060] refer to Figure 6 , Figure 610 is a raw vascular map of the target portion, which has virtually no contrast. That is, the camera image does not provide sufficient pixel-to-pixel contrast between areas of normal vascularity (normal tissue) and areas of abnormal vascularity (e.g., abnormal tissue or tumor). In contrast, the aggregated vascular map 630 shows a sharp contrast between a normal portion 631 and an abnormal portion 633 of the target portion or ROI.

[0061] In the depicted embodiment, the aggregated vessel map 630 is a combination of three other vessel maps obtained by different techniques, with improved contrast compared to any of the other vessel maps viewed on their own. The three other vessel maps include the frequency map 330, signal characteristic map 430, and phase variability map 530 discussed above. The vessel features may be displayed as the frequency map 330, signal characteristic map 430, and / or phase variability map 530, as well as the aggregated vessel map 630 (with increased contrast in abnormal tissue or tumor regions).

[0062] Embodiments include training a simple logistic regression (or other classifiers may be used) for each pixel or each pixel cluster classifier to determine the best combination of the frequency map 330, the signal characteristic map 430, and / or the phase variability map 530 to improve the contrast of malignant lesions. Embodiments include providing different weights to pixels or portions of abnormal vascular distribution according to whether the abnormality appears in one, two, or all three of the other vascular maps. The highest weight is assigned to pixels or portions of the image that show abnormalities in all three other vascular maps, and the lowest weight is assigned to pixels or portions of the image that show abnormalities in only one of the other vascular maps. The weights can then be used to create an aggregated vascular map 630. Of course, other techniques for combining vascular maps into an aggregated vascular map 630 may be incorporated without departing from the scope of the present teachings. Moreover, the aggregated vascular map 630 may combine less than all three other vascular maps without departing from the scope of the present teachings. Therefore, in various embodiments, the aggregated vascular map 630 may be based on at least one of the frequencies, signal characteristics, and spatial phase variability of at least some of the vasculature signals corresponding to multiple pixels in the pixel array, respectively.

[0063] As described above, the camera can provide images during an interventional procedure. Thus, the camera can be mounted to the distal end of an interventional instrument or tool, such as an endoscope or catheter, and / or mounted to a robotic arm to guide the camera to a target portion or ROI within a subject.

[0064] Fig. 7A is a schematic block diagram illustrating an example of a surgical robotic system that may be used to obtain images so that interventional procedure(s) can be performed, according to a representative embodiment of the present disclosure. Fig. 7A A surgical robotic system 700 is depicted that can be used for medical procedures, which generally involve manipulation of surgical instruments and other tools that can be operated by the robot at a surgical site located within a patient's body. Such medical procedures include, but are not limited to, for example, minimally invasive cardiac surgery, abdominal surgery (laparoscopy) (e.g., prostatectomy or cholecystectomy), and natural orifice endoscopic surgery.

[0065] refer to Fig. 7A, the surgical robot system 700 includes at least one robot 701, a control unit 706, and an augmented reality (AR) device 720. The robot 701 is configured to operate one or more end effectors to be positioned on a surgical site S within an object P, including at least one camera 712 (or similar image acquisition device) configured to capture live images at the surgical site S and at least one instrument 713, such as a surgical tool for performing an internal surgical procedure. The internal surgical procedure may include minimally invasive surgery or natural orifice surgery, for example, involving a target portion T within the surgical site S, examples of which are mentioned above.

[0066] The AR device 720 is configured to display live images captured by the camera 712 on an AR display unit 721 for viewing by a user, and to detect tracking data using an AR tracking system 722 that captures the user's movements (in response to the displayed images) via one or more motion sensing circuits indicated by a representative motion sensor 724. For example, as discussed further below, the AR device 720 may be a head-mounted head-up display (HUD) device (AR-HUD device) including a head component worn by a user, the head component including a display unit 721 and a motion sensor 724. The display unit 721 may be a single display, or it may be divided into two eye displays (not shown) corresponding to the user's eyes. The two eye displays enable three-dimensional viewing when the camera 712 is stereoscopic, or enable a pseudo-three-dimensional view when the camera 712 is monocular, but the two eye displays are configured to partially overlap. The motion sensor 724 may be a plurality of accelerometers, for example, for detecting the movement of the user's head around three axes, so that the tracking data includes head tracking data corresponding to the user's head movement. Alternatively, the motion sensor 724 may include rear-facing camera(s) for detecting movement of the user's eyes, such that the tracking data includes eye tracking data corresponding to the user's eye movements.

[0067] The AR device 720 may also include one or more input devices (e.g., user interfaces) indicated by the input device 726 for receiving instructions from the user. The input device 726 as used herein is an interface that allows the user to interact with the AR device 720 and the surgical robot system 700. The input device 726 may include one or more of a touch screen, a keyboard, a mouse, a trackball, a touchpad, or a voice command interface. In this embodiment, the user can use the input device 726 to input a specific command, such as transmitting a confirmation signal to the processor 730 to confirm the selected target portion T as determined by the processor 730, or transmitting an activation signal to the AR tracking system 722 and / or the processor 730 to activate one of the head tracking modes including detection of head movement by the HUD device or the eye tracking mode including detection of eye movement by the HUD device. In an alternative embodiment, the input device 726 may be outside the AR device 720. For example, the input device 726 may be integrated into the control unit 706 or may be a separate unit configured to communicate with both the AR device 720 and the processor 730 via the I / O circuit 708.

[0068] The control unit 706 is configured to control and otherwise coordinate the overall operation of the surgical robotic system 700. The control unit 706 includes input / output (I / O) circuits 708 and a processor 730. The processor 730 includes an associated memory 734 for enabling processing and a computer readable medium (CRM) 736. The processor 730 and the memory 734 / CRM 736 may be substantially similar to those described above with reference to Figure 2 The processing unit 230 of the imaging system 200 discussed. The processor 730 is generally configured to receive the acquired live images from the camera 712 via the I / O circuit 708, and process and store the acquired live images in, for example, the memory 734 and / or the CRM 736, so that the processor 730 can build a database, such as an electronic medical record (EMR) database and / or a picture archiving and communication system (PACS) database, essentially visually mapping the interior portion of the object P traversed by the camera 712. Once the target portion T has been selected, the database can then be used to determine a path to the target portion T, as described below. The processor 730 sends the acquired live images to the AR device 720 via the I / O circuit 708 for display on the display unit 721. In an alternative configuration, after the acquired live images are received by the I / O circuit 708, the acquired live images can be sent from the I / O circuit 708 to the AR device 720 without any imaging processing being performed by the processor 730.

[0069] The processor 730 is also configured to receive tracking data determined by the AR tracking system 722 from the AR device 720 via the I / O circuit 708, and process the determined tracking data to select a target (e.g., target portion T) at the surgical site S within the object P. The processor 730 is also configured to determine a path for one or more end effectors (e.g., including the illustrative camera 712 and the instrument 713) of the robot 701 to the selected target portion T based on the acquired live image and the processed determined tracking data. The processor 730 sends a robot control signal to the robot 701 via the I / O circuit 708 to move the one or more end effectors to the selected target portion T along the determined path.

[0070] Therefore, the I / O circuit 708 receives the acquired live image from the camera 712, and provides the acquired live image to the processor 730 and / or the AR device 720 (directly or forwarded from the processor 730 after image processing) for display on the display unit 721. The I / O circuit 708 also receives input data (including tracking data) from the AR device 720, provides the input data to the processor 730, and transmits the robot control signal determined by the processor 730 in response to the tracking data to the robot 701.

[0071] The robot 701 may have a rigid proximal portion 702 (e.g., a robotic arm or portion of a robotic arm) to be positioned at an entry point E within the body of a subject P, and a flexible distal portion 703 to be positioned at a surgical site S within the subject P. The robot 701 is broadly defined herein as any robotic device that is structurally configured with electric control of one or more joints and two or more corresponding links to manipulate the flexible distal portion 703 as desired for a particular robotic procedure. The flexible distal portion 703 may include one or more end effectors, including a camera 712 and an instrument 713. For example, the instrument 713 may be a grasper or tool holder, a laparoscopic instrument, a laparoscope, a tool for screw placement (e.g., in spinal fusion surgery), a needle for biopsy or treatment, an ultrasound transducer for tissue or tumor imaging and ablation, or other surgical or interventional tools. For purely exploratory interventional procedures, the flexible distal portion 703 may include only a camera 712.

[0072] In practice, as will be appreciated by those skilled in the art, the robot 701 may have a minimum of three (3) degrees of freedom, and advantageously six (6) or more degrees of freedom. The robot 701 may have a remote center of motion (RCM) mechanism with two intersecting motor axes. Moreover, the robot 701 may already be associated with a light projection device (not shown). An example of a manipulable or dexterous surgical robotic device that may be incorporated is a seven-degree-of-freedom universal wrist, such as described in U.S. Pat. No. 7,121,781 (October 17, 2006) to Sanchez, which is incorporated herein by reference. The universal wrist includes a plurality of joints and jaws at the distal portion, and a drive system at the proximal portion. The device may also be positioned using a robot positioner and a computer controller. Robot positioning allows tracking of device motion relative to the anatomical structure. An end effector may be coupled to the wrist, which provides two separate degrees of freedom about the same pivot point. The end effector may be moved and actuated by a pin, thereby allowing a compact minimally invasive medical instrument.

[0073] Despite Fig. 7A 701 for operating a single rigid proximal portion 702 and a single corresponding distal portion 703, it should be understood that multiple robots having corresponding multiple rigid proximal portions and / or multiple rigid distal portions may be incorporated without departing from the scope of the present teachings. That is, the robot 701 may include multiple robotic arms (not shown) that utilize multiple cameras 712 to control different end effectors in the surgical site S. For example, one robotic arm may include a forward-looking camera, such as camera 712 for providing a live image of the surgical site S, while another robotic arm may include an instrument 713 and (one or more) additional instruments.

[0074] Camera 712 defines part of an imaging system having control unit 706 of surgical robotic system 700, which imaging system is similar to that described above with reference to Figure 2The imaging system 200 discussed is substantially similar. The camera 712 may include one or more of any type of camera with a forward optical view or an oblique optical view, and may be capable of acquiring a sequence of two-dimensional digital video frames at a predefined frame rate (e.g., 30 frames per second), and may be capable of providing each digital video frame to the control unit 706 via the input / output circuit 708. In particular, the camera 712 may be a forward-looking camera positioned and oriented so that it can acquire live images of the surgical site S and the target portion T (e.g., an organ) from the flexible distal portion 703 of the robot 701 within its field of view. In an embodiment, the rigid proximal portion 702 may be an endoscope-like device controlled by the robot 701, wherein a single rigid proximal portion 702 includes a plurality of channels, one of the plurality of channels enabling control and / or communication with the camera 712, and at least one other of the plurality of channels enabling access and control of an instrument 713 or other end effector. In an alternative embodiment, the camera 712 is mounted to the flexible distal portion 703 corresponding to the rigid proximal portion 702, while other end effectors may be in the flexible distal portions corresponding to other rigid proximal portions (not shown), all of which are controlled via the robot 701. In yet another embodiment, the camera 712 is mounted to the flexible distal portion of a separate endoscope that is not part of the robot 701, but is under the control of a separate endoscope controller (operating in conjunction with the robot 701), as described below with reference to Figure 7B As discussed. Moreover, the camera 712 may be a stereo camera or stereo endoscope that provides images that can be perceived in three dimensions. An example of a stereo endoscope is described in U.S. Pat. No. 6,139,490 (October 31, 2000) by Breidenthal et al., which is incorporated herein by reference. Alternatively, the camera 712 may be monocular and the display unit 721 may be divided into two eye displays corresponding to the user's eyes, in which case the monocular image is split in two overlapping halves and presented separately on each of the two eye displays to create a pseudo-stereoscopic view.

[0075] An external medical imaging device may also be included. The medical imaging device may be configured to capture one or more live images of the flexible distal portion 703 of the surgical robot 701 and the target portion T at the surgical site S, thereby providing a more comprehensive overview. Such a medical imaging device may include a C-arm (not shown) that at least partially surrounds a portion of the object P. The C-arm has radiographic capabilities and can be used for fluoroscopic imaging during the surgical procedure, as known to those skilled in the art. An example of a C-arm implementing an X-ray system is described in U.S. Patent US 9095252 (August 4, 2015) by Popovic, which is incorporated herein by reference.

[0076] Figure 7B is a schematic block diagram illustrating a surgical robotic system including a separate imaging endoscope and a corresponding endoscope controller according to a representative embodiment. Figure 7B Depicted is a surgical robotic system 700 ′ that may be used for medical procedures, generally involving the manipulation of surgical instruments and other tools operable by a robot at a surgical site located within a subject's body, as described above.

[0077] refer to Figure 7B In the depicted embodiment, the surgical robotic system 700' is similar to the surgical robotic system 700, in that it includes at least one robot 701, a control unit 706, and an AR device 720. The robot 701 is configured to operate one or more end effectors to be positioned at a surgical site S within an object P, including at least one instrument 713, so as to have surgical tools for performing internal surgical procedures.

[0078] In addition, the surgical robot system 700' includes an endoscope controller 740 configured to control an endoscope 742 separated from the robot 701. The endoscope 742 includes a camera 712 mounted to the distal end. The endoscope 742 and the camera 712 are configured to collect live images in the body at the surgical site S. The AR device 720 is configured to display the live images collected by the camera 712, as described above. In various embodiments, the endoscope 742 may include a rigid or flexible tube, a light delivery system for illuminating an organ or object under inspection (e.g., the light source is usually outside the body, and the light is usually guided via an optical fiber system), and a lens system that sends the image from the objective to the observer, typically a relay lens system in the case of a rigid endoscope, or a bundle of optical fibers in the case of a fiberscope. A video oscilloscope without an eyepiece is also considered, in which the camera sends the image to a screen for image capture. Moreover, in various embodiments, the camera 712 may be stereoscopic, for example, where the image data enables display of an apparent three-dimensional image, for example, on the AR display unit 721 .

[0079] In the depicted example, the robot 701 has a rigid proximal portion 702 to be positioned at a first entry point E1 within the body of a subject P, and a flexible distal portion 703 to be positioned at a surgical site S within the subject P. An endoscope 742 includes a sheath to be positioned at a second entry point E2 within the body of the subject P. The endoscope 742 may include a plurality of channels, one of which provides image acquisition, and at least one other of which enables access and control of another instrument to be controlled by the endoscope controller 740, such as a medical instrument or manipulator and / or a light source for illuminating the surgical site S.

[0080] The control unit 706 is configured to control and otherwise coordinate the overall operation of the surgical robotic system 700', including the control and coordination of the robot 701 and the endoscope 742. The control unit 706 includes an I / O circuit 708', which is different from the I / O circuit 708, in that it has (one or more) additional interfaces for communicating with the endoscope controller 740. Thus, the processor 730 is generally configured to receive the acquired live images from the camera 712 via the endoscope controller 740 and the I / O circuit 708', and process and store the acquired live images in, for example, the memory 734 and / or the CRM 736, so that the processor 730 can build a database that essentially visually maps the interior portion of the object P passed by the endoscope 742. Once the target portion T has been selected, the database can then be used to determine a path to the target portion T, as described below. The processor 730 sends the acquired live images to the AR device 720 via the I / O circuit 708' for display on the display unit 721. In an alternative configuration, after receiving the acquired live image from the endoscope controller 740 via the I / O circuit 708 ′, the acquired live image may be sent from the I / O circuit 708 ′ to the AR device 720 without any imaging processing performed by the processor 730 .

[0081] The processor 730 is also configured to receive tracking data determined by the AR tracking system 722 from the AR device 720 via the I / O circuit 708', and process the determined tracking data to select a target (e.g., target portion T) at the surgical site S within the object P. The processor 730 is also configured to determine a path for one or more end effectors (e.g., including the illustrative instrument 713) of the robot 701 to reach the selected target portion T based on the acquired live image and the processed determined tracking data. The processor 730 sends a robot control signal to the robot 701 via the I / O circuit 708' to move the one or more end effectors to the selected target portion T along the determined path.

[0082] Therefore, the I / O circuit 708′ receives the acquired live image from the camera 712 via the endoscope 742 and the endoscope controller 740, and provides the acquired live image to the processor 730 and / or the AR device 720 (directly or forwarded from the processor 730 after image processing) for display on the display unit 721. The I / O circuit 708′ also receives input data (including tracking data) from the AR device 720 (which provides the input data to the processor 730), and transmits the robot control signal determined by the processor 730 in response to the tracking data to the robot 701.

[0083] Reference AR device 720 (in Fig. 7A and 7B ), the display unit 721 includes one or more displays that can be co-located near the user. For example, when the AR device 720 is implemented as a HUD device, the display unit 721 can be attached to the front of the head component, directly in front of the user's eyes, such as goggles. The display unit 721 is configured to display a live image of the surgical site S, as described above, and the user moves his or her head and / or eyes to observe the live image displayed on the display unit 721 and track the target for selection, thereby simulating an actual surgical procedure, wherein the surgeon moves his or her head and / or eyes to observe the surgical site and perform the procedure. When the camera 712 provides a stereoscopic image, the live image displayed on the display unit 721 can appear to be three-dimensional. In an embodiment, the display unit 721 can also display preoperative images. Therefore, images from at least one camera 712 are captured and displayed in the HUD device.

[0084] The HUD device may be referred to as an augmented reality head-up display (AR-HUD) device having one or more AR-HUD displays. The illustrative design employs left-eye and right-eye displays, but alternatively, the display may be a single large window across both eyes. In some examples, the AR-HUD device may be configured as a helmet, headband, glasses, goggles, or other suitable embodiments to be worn on the user's head.

[0085] The processor 730 can identify or select the target portion T by processing the tracking data to determine the angle of the user's head relative to the display unit 721 (and the live image displayed thereon), wherein the head angle indicates the area or point of the live image that constitutes the target portion T within the surgical field S that the user is looking at. Once the user is satisfied with the location of the target portion T in the surgical field S on the display unit 721, a confirmation signal can be transmitted to the processor 730 in the control unit 706 (e.g., via a foot pedal, button, or voice command). The processor 730 can assign (or address) coordinates in two or three dimensions corresponding to the selected target portion T. Knowing the location of the entry point E within the body of the subject P and the coordinates of the selected target portion T and the database of live images acquired from the camera 712, the processor 130 can determine the path (or (one or more) vectors) of at least one instrument 713 that can be operated by the robot 701 to reach the selected target portion T. An example of determining a path for a robotically controlled instrument to reach a target in a surgical site of a patient is provided in U.S. Pat. No. 8,934,003 (January 13, 2015) by Popovic et al., which is incorporated herein by reference. The processor 730 can then send a robot control signal to the robot 701, causing the robot 701 to guide the distal end of at least one instrument 713 to the target portion T via the determined path. That is, the control unit 706 issues a command to the robot 701 to move to the position of the selected target portion T. The compensatory (requited) motion of each joint of the robot 701 is implemented by the processor 701 using methods known in the art. For example, the robot 701 can utilize visual servoing methods known in the art.

[0086] Reference again Fig. 7A and 7B, the control unit 706 may be broadly defined herein as any controller that is structurally configured to provide one or more control commands to control the acquisition and processing of images from the camera 712 at the surgical site S and the target portion T (e.g., live, exploratory, preoperative), and to determine a path to the target portion T using tracking information related to the selection of the target portion T from the AR device 720 and also control the flexible distal portion 703. In general, the I / O circuit 708 controls communications between elements and devices external to the control unit 706. The I / O circuit 708 serves as an interface that includes necessary logic for interpreting input and output signals or data to / from the processor 730, the AR device 720, and the robot 701. The I / O circuit 708 may include: a first input portion configured to receive the acquired live image from the camera 712; and a first output portion configured to provide the acquired live image to at least one of the processor 730 and the AR device 720 for ultimate display. The I / O circuit 708 may also include: a second input unit, which is configured to receive determined input data including tracking data from the AR device 720; a second output unit, which is configured to provide the determined input data to the processor 730, and the processor 730 processes the input data in response to identify and select the target part T; and a third output unit, which is configured to provide a robot control signal to the robot 701.

[0087] The processor 730 may use a combination of hardware, software, and firmware to perform the described functions and operations. The processor 730 is configured to process medical images (e.g., from the camera 712 or an external medical imaging device) related to the flexible distal portion 703 of the surgical robot 701 at the surgical site S to align the flexible distal portion 703 with the corresponding anatomical structure at the surgical site S. The processor 730 may be configured to process additional position tracking information of the rigid proximal portion 702 of the surgical robot 701 from a position tracking system (not shown) to determine the movement of the rigid proximal portion 702. The position tracking system separate from the tracking system 722 may be one or more of: an optical tracking system, a mechanical tracking system, and an electromagnetic tracking system, as will be appreciated by those skilled in the art. A sensor or tag (e.g., a light emitting diode (LED), a passive marker, a reflective marker, etc.) may be included at the rigid proximal portion 702 of the surgical robot 701 to cooperate with the position tracking system. In an embodiment, a position compensation mode may be provided, according to which the processor 730 is configured to generate a motion compensation signal for the flexible distal portion 703 of the surgical robot 701 based on the determined position and movement of the rigid proximal portion 702.

[0088] In practice, the control process discussed may be implemented by a module implemented by any combination of hardware, software and / or firmware installed on any platform (e.g., a general purpose computer, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.). In addition, the process may be executed by the processor 730 of the control unit 706.

[0089] As used in the specification and the appended claims, the terms "a", "an", and "the" include both singular and plural references unless the context clearly dictates otherwise. Thus, for example, "a device" includes one device and a plurality of devices. Moreover, it should be understood that the use of any of " / ", "and / or", and "at least one of..." below, such as in the case of "A / B", "A and / or B", and "at least one of A and B", is intended to cover the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of both options (A and B).

[0090] As used herein, the statement that two or more parts or components are "coupled" shall mean that the parts are connected or operate together either directly or indirectly (ie, through one or more intermediate parts or components), so long as a link occurs.

[0091] Directional terms / phrases and relative terms / phrases may be used to describe the relationship of various elements to each other as illustrated in the figures. These terms / phrases are intended to encompass different orientations of the device and / or elements in addition to the orientation depicted in the figures.

[0092] As used herein, "computer-readable storage medium" encompasses any tangible storage medium that can store instructions that can be executed by a processor of a computing device. The computer-readable storage medium may be referred to as a non-transitory computer-readable storage medium to distinguish it from transient media such as transient propagating signals. Computer-readable storage media may also be referred to as tangible computer-readable media.

[0093] In some embodiments, memory and / or computer-readable storage medium may also be able to store data that can be accessed by a processor of a computing device. Examples of memory and computer-readable storage medium include, but are not limited to, floppy disks, magnetic hard disk drives, solid-state hard disks, flash memory, USB thumb drives, random access memory (RAM), read-only memory (ROM), optical disks, magneto-optical disks, and register files of processors. Examples of optical disks include compact disks (CDs) and digital versatile disks (DVDs), such as CD-ROMs, CD-RWs, CD-Rs, DVD-ROMs, DVD-RWs, or DVD-R disks. The term computer-readable storage medium also refers to various types of recording media that can be accessed by a computer device via a network or communication link. For example, data can be retrieved via a modem, via the Internet, or via a local area network. References to computer-readable storage media should be interpreted as possibly being multiple computer-readable storage media. Various executable components of one or more programs may be stored in different locations. Computer-readable storage media may, for example, be multiple computer-readable storage media within the same computer system. Computer-readable storage media may also be computer-readable storage media distributed among multiple computer systems or computing devices.

[0094] "Memory" is an example of a computer-readable storage medium. Computer memory is any memory that is directly accessible to a processor. Examples of computer memory include, but are not limited to, RAM memory, registers, and register files. References to "computer memory" or "memory" should be interpreted as potentially being multiple memories. The memory may be, for example, multiple memories within the same computer system. The memory may also be multiple memories distributed among multiple computer systems or computing devices.

[0095] A computer storage device is any non-volatile computer-readable storage medium. Examples of computer storage devices include, but are not limited to, hard drives, USB thumb drives, floppy disk drives, smart cards, DVDs, CD-ROMs, and solid-state hard drives. In some embodiments, a computer storage device may also be a computer memory, or vice versa. References to "computer storage devices" or "storage devices" should be interpreted as potentially including multiple storage devices or components. For example, a storage device may include multiple storage devices within the same computer system or computing device. A storage device may also include multiple storage devices distributed among multiple computer systems or computing devices.

[0096] As used herein, "processor" encompasses an electronic component capable of executing a program or machine executable instructions. References to a computing device that includes a "processor" should be interpreted as possibly including more than one processor or processing core. A processor may, for example, be a multi-core processor. A processor may also refer to a collection of processors within a single computer system or distributed among multiple computer systems. The term computing device should also be interpreted as possibly referring to a collection or network of computing devices that each include one or more processors. Many programs have instructions that are executed by multiple processors that may be within the same computing device or may even be distributed across multiple computing devices.

[0097] As used herein, a "user interface" or "user input device" is an interface that allows a user or operator to interact with a computer or computer system. The user interface can provide information or data to the operator and / or receive information or data from the operator. The user interface enables input from the operator to be received by the computer, and output can be provided from the computer to the user. In other words, the user interface can allow the operator to control or manipulate the computer, and the interface can allow the computer to indicate the effect of the user's control or manipulation. Displaying data or information on a display or graphical user interface is an example of providing information to the operator. The reception of data by a touch screen, keyboard, mouse, trackball, touchpad, pointing stick, graphic input tablet, joystick and headset (e.g., AR-HUD) is a full example of a user interface component that enables receiving information or data from a user.

[0098] As used herein, a "display" or "display device" or "display unit" encompasses an output device or user interface suitable for displaying images or data. A display may output visual, audio, and / or tactile data. Examples of displays include, but are not limited to, computer monitors, television screens, touch screens, tactile electronic displays, Braille screens, cathode ray tubes (CRTs), memory tubes, bi-stable displays, electronic paper, vector displays, flat panel displays, vacuum fluorescent displays (VFs), light emitting diode (LED) displays, electroluminescent displays (ELDs), plasma display panels (PDPs), liquid crystal displays (LCDs), organic light emitting diode displays (OLEDs), projectors, and head mounted displays.

[0099] While various embodiments have been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.

Claims

1. An imaging system (200), comprising: a camera (212) having color video acquisition capability, the camera being mounted to a distal end of an interventional instrument (242) insertable into an object, the camera providing a plurality of image frames (311, 312, 313) for imaging a vascular system of the object, each image frame comprising a plurality of pixels (P1, P2) providing corresponding signals; a display (220); and A processor (232) programmed to: receiving the signal (301, 302); amplifying (315) a change in at least one of color and motion of the signal corresponding to the plurality of pixels to provide an amplified signal (301', 302', 503, 504); respectively determining at least one of a spatial phase variability (522'), a frequency (401, 402) and a signal characteristic of at least some of the amplified signals corresponding to the plurality of pixels; identifying (415) pixels in the plurality of pixels indicative of abnormal vascularization based on the at least one of spatial phase variability, frequency, and signal characteristics of the at least some of the amplified signals; using the identified pixels to create a vascularity map (330, 430, 530) corresponding to each of the at least one of spatial phase variability, frequency, and signal characteristics of the at least some of the amplified signals, each vascularity map including a portion of the object having the abnormal vascularity (333, 433, 533); and The display is operated to display each blood vessel map.

2. The imaging system (200) according to claim 1, wherein: The processor is also programmed to create an aggregated vessel map (630) comprising a combination of at least two of the vessel maps corresponding to each of the at least one of spatial phase variability, frequency, and signal characteristics of the at least some of the amplified signals.

3. The imaging system (200) according to claim 2, wherein: The aggregated vessel map (630) provides a sharper contrast between the portion (633) of the object having the abnormal vascular distribution and another portion (631) of the object having a normal vascular distribution than a contrast provided by any one of the vessel maps corresponding to each of the at least one of spatial phase variability, frequency, and signal characteristics of the at least some of the amplified signals.

4. The imaging system (200) according to claim 1, wherein: The change in at least one of color and motion of the signal is amplified using Euler video amplification.

5. The imaging system (200) according to claim 1, wherein: The processor is programmed to identify the pixels indicative of abnormal vascularity based on spatial phase variability by: respectively determining a phase shift between adjacent pixels of the plurality of pixels; Providing phase maps (522, 523, 524, 522') corresponding to enlarged image frames (521), each enlarged image frame including the plurality of pixels; determining a standard deviation (SD) of the phase shifts between the adjacent pixels of the phase map; and Pixels in the phase map having a standard deviation above a threshold standard deviation corresponding to normal vascularity are identified as indicating abnormal vascularity.

6. The imaging system (200) of claim 1, wherein: The processor is programmed to identify the pixels indicative of abnormal vascularity based on frequency, including by: respectively determining frequency components of the amplified signal corresponding to the plurality of pixels; and The pixels having corresponding frequency components outside of a physiologically feasible passband are identified as being indicative of abnormal vascularity.

7. The imaging system (200) according to claim 6, wherein: The physiologically feasible passband corresponds to a credible range of the pulse rate of the subject.

8. The imaging system (200) of claim 6, wherein: The processor is programmed to identify the pixels having corresponding frequency components outside the physiologically feasible passband by: performing a Fourier transform on the amplified signal to provide a corresponding frequency spectrum; identifying a maximum peak component in the frequency spectrum; and It is determined whether the maximum peak component is outside the physiologically feasible passband.

9. The imaging system (200) of claim 1, wherein: The processor is programmed to identify the pixels indicative of abnormal vascularity based on signal characteristics by: performing a Fourier transform on the amplified signal to provide a corresponding frequency spectrum; determining a maximum peak component in the frequency spectrum; Determine the area under the curve AUC or the full width at half maximum FWHM of the maximum peak component, respectively; and The pixels having a corresponding AUC or FWHM less than a threshold AUC or FWHM corresponding to normal vascularity are identified as indicating abnormal vascularity.

10. The imaging system (200) of claim 1, wherein: The interventional instrument (242) includes an endoscope.

11. The imaging system (200) of claim 2, wherein: The aggregated vessel map is created based on a combination of all of the vessel maps corresponding to each of spatial phase variability, frequency, and signal characteristics of at least some of the amplified signals.

12. The imaging system (200) according to claim 2, wherein the display is incorporated into an augmented reality head-up display (AR-HUD) device, wherein the augmented reality head-up display device is used to display at least one of the following: the vascular map or a combination of at least two of the vascular maps, and the AR-HUD device includes a head component worn by a user.

13. A method for identifying abnormal vascularity indicative of abnormal tissue within a subject using images from a camera (212) having color video acquisition capabilities and mounted to a distal end of an interventional instrument (242) insertable within the subject, the method comprising: receiving signals (301, 302) corresponding to a plurality of pixels (P1, P2) in each of a plurality of image frames (311, 312, 313) of a target portion in the subject provided by the camera for imaging a vascular system of the target portion; amplifying (315) changes in at least one of color and motion of the signal corresponding to the plurality of pixels in each of the plurality of image frames to provide amplified signals (301', 302'); respectively determining at least one of a spatial phase variability (522'), a frequency (401, 402) and a signal characteristic of at least some of the amplified signals corresponding to the plurality of pixels; identifying (415) pixels indicative of abnormal vascularization among the plurality of pixels in each of the plurality of image frames based on the at least one of spatial phase variability, frequency, and signal characteristics of the at least some of the amplified signals; using the identified pixels to create a vascularity map (330, 430, 530) corresponding to each of the at least one of spatial phase variability, frequency, and signal characteristics of at least some of the amplified signals, each vascularity map including a portion of the subject having the abnormal vascularity (333, 433, 533); and Displays the distribution map of each blood vessel.

14. The method according to claim 13, further comprising: combining at least two of the vessel maps (330, 430, 530) corresponding to each of the at least one of spatial phase variability, frequency, and signal characteristics of the at least some of the amplified signals to create an aggregated vessel map; and The aggregated vessel map is displayed.

15. A computer readable medium containing software executable by a computer processor (232) to identify abnormal vascularity indicative of abnormal tissue within a subject using images from a camera (212) having color video acquisition capabilities and mounted to a distal end of an interventional instrument (242) insertable within a subject, the computer readable medium comprising: receiving code for receiving signals (301, 302) corresponding to a plurality of pixels (P1, P2) in each of a plurality of image frames (311, 312, 313) of a target portion in the subject provided by the camera via the interventional instrument, for imaging a vasculature of the target portion; amplification code for respectively amplifying (315) changes in at least motion of the signal corresponding to the plurality of pixels in each of the plurality of image frames to provide amplified signals (301', 302', 503, 504); determining code for respectively determining at least one of a spatial phase variability (522'), a frequency (401, 402) and a signal characteristic of at least some of the amplified signals corresponding to the plurality of pixels; an identification code for identifying pixels indicative of abnormal vascularization among the plurality of pixels in each of the plurality of image frames based on the at least one of spatial phase variability, frequency, and signal characteristics of the at least some of the amplified signals; creating code for creating a vascularity map (330, 430, 530) corresponding to each of said at least one of spatial phase variability, frequency and signal characteristics of said at least some of said amplified signals using said identified pixels, each vascularity map including a portion of said subject having said abnormal vascularity (333, 433, 533); combining code for providing an aggregated vessel map by combining at least two of the vessel maps corresponding to each of the at least one of spatial phase variability, frequency, and signal characteristics of the at least some of the amplified signals (630); and Display code for causing the aggregated vascular map to be displayed on a display, the displayed aggregated vascular map indicating the portion of the object having the abnormal vascular distribution by enhanced contrast with the portion of the object having normal vascular distribution.

Images