System and method for reconstructing 3D images from ultrasound and camera images - Patents.com

Abstract

A method is provided for reconstructing a 3D image, the method including obtaining Doppler ultrasound images depicting blood vessels and measurements of blood flow within the vessels and 2D camera images captured by a camera depicting fiducial objects randomly distributed within a viscous material and spaced apart by random distances on the surface of an individual, calculating 3D coordinates in a world coordinate system for pixels of the Doppler images using an external reference of the pose of the ultrasound transducer calculated by analysis of relative changes in the positions of the fiducial objects in successive camera images, calculating respective estimated blood flow rates for pixels of the Doppler images at positions within the blood vessels, and reconstructing a 3D image from 3D voxels calculated from the 3D coordinates of the pixels of the Doppler images including respective estimated blood flow rates, the 3D image depicting the blood vessels and blood flow.

Description

[Technical field]

[0001] Related Applications This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 174,064, filed April 13, 2021, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] The present invention, in some embodiments thereof, relates to the processing of medical ultrasound images, and more particularly to the reconstruction of 3D medical images from 2D ultrasound images.

[0003] Ultrasound (US) is often considered the safest and least expensive technique for clinical analysis and medical intervention. US images are always captured as 2D images. There are also approaches to generate 3D ultrasound images. Summary of the Invention

[0004] According to a first aspect, a computer-implemented method for reconstructing a 3D image depicting blood flow within a blood vessel of an individual includes acquiring a Doppler ultrasound image depicting the blood vessel with blood flowing therethrough and depicting a measurement of blood flow within a region of the blood vessel, and a 2D camera image captured by a camera depicting a plurality of fiducial objects randomly distributed within a viscous material and spaced apart by random distances on a surface of a body segment of the individual, the Doppler ultrasound image including at least one Doppler ultrasound image and the 2D camera image including at least one 2D camera image; and acquiring two successive 2D camera images. the method further comprising: calculating 3D coordinates in a world coordinate system for each pixel of the Doppler ultrasound image using an external reference of ultrasound transducer pose calculated by analysis of relative changes in positions of a plurality of fiducial objects in the Doppler camera image; calculating a respective estimated blood flow rate for each of a plurality of pixels of the Doppler ultrasound image at a plurality of locations within the blood vessel; and reconstructing a 3D image from 3D voxels calculated from the 3D coordinates of the pixels of the Doppler ultrasound image, the respective estimated blood flow rates, wherein the 3D image depicts an anatomical image of the blood vessel and depicts the blood flow.

[0005] In a further implementation of the first aspect, the 3D coordinates are further calculated by correcting them to their actual scale relative to the world coordinate system by using the size of the fiducial object and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera.

[0006] In a further implementation of the first aspect, the 3D image further comprises voxels calculated from a 3D point cloud or mesh describing a surface of a body segment of the individual.

[0007] In a further implementation of the first aspect, calculating 3D coordinates of pixels of each Doppler ultrasound image includes calculating a camera pose in the camera coordinate system relative to a world coordinate system by analyzing relative changes in positions of multiple fiducial objects in successive 2D camera images, applying a calibration mapping to map pixels of the ultrasound image represented in the ultrasound coordinate system to the camera coordinate system, where the calibration mapping is based on a predetermined transformation between the camera pose and the ultrasound transducer pose, and mapping the pixels of the ultrasound image represented in the camera coordinate system to 3D coordinates in the world coordinate system.

[0008] In a further implementation of the first aspect, calculating each estimated blood flow includes an estimated blood flow calculated from a Doppler ultrasound image captured by an ultrasound transducer positioned at an angle relative to a vector indicating the direction of the blood vessel for which the blood flow is measured, and further includes correcting the estimated blood flow for this angle to obtain the estimated actual blood flow.

[0009] In a further implementation of the first aspect, the correcting includes identifying an elliptical boundary of the vessel, calculating a transformation from the elliptical shape to a circle, and applying the transformation to project the measured blood flow velocity vector to obtain the actual blood flow.

[0010] In a further implementation of the first aspect, the method further includes calculating an estimate of a longitudinal axis of the blood vessel based on an aggregation of a plurality of 3D points acquired from each respective acquisition plane of the plurality of Doppler ultrasound images, and calculating a current tangent to the longitudinal axis, and the correcting includes correcting the measurement of blood flow based on an angle between the tangent and a normal to the respective acquisition plane indicated by the tangent.

[0011] A further implementation of the first aspect further includes calculating an estimate of blood flow within the pre-calculated 3D arterial object, and using the computational fluid dynamics simulation and 3D points acquired from each acquisition plane of each of the multiple Doppler ultrasound images to calculate an estimate of blood flow within the 3D arterial object.

[0012] A further implementation of the first aspect further includes storing, for each 3D voxel, a plurality of initial estimated blood flow values ​​and corresponding normals to the plane of the ultrasound transducer from which each Doppler US image was captured that were used to calculate each initial estimated blood flow, and estimating actual blood flow at each voxel based on minimizing a projection error between the recorded blood flow velocity vector and the assumed flow velocity vector.

[0013] A further implementation of the first aspect further includes obtaining volumetric flow rates within a blood vessel on a sagittal view at positions specific to multiple cardiac cycles to calculate an average volumetric flow rate over the multiple cardiac cycles, calculating a gamma value representing a real number in the range of -1 to 1 to correct the average volumetric flow rate and the estimated actual blood flow rate, calculating a cosine angle by using the inverse of the gamma value, and correcting the average volumetric flow rate using the cosine angle and the dot product between the estimated blood flow rate and the actual blood flow rate.

[0014] A further implementation of the first aspect further includes alternating between (i) acquiring a Doppler ultrasound image and a 2D camera image and (ii) acquiring a second image including a B-mode ultrasound image and a 2D camera image, and calculating 3D coordinates of pixels of each B-mode ultrasound image in a world coordinate system, and reconstructing a 3D image in the world coordinate system from 3D voxels calculated by aggregating the 3D coordinates of the pixels of the B-mode ultrasound image and the Doppler ultrasound image, the 3D image including an estimated blood flow, the 3D image depicting an anatomical image of a blood vessel created from the aggregation of 3D voxels acquired from the B-mode pixels and depicting blood flow associated with the 3D voxels acquired from the B-mode pixels.

[0015] In a further implementation of the first aspect, each estimated blood flow is depicted as a color coding of 3D voxels of a 3D image corresponding to multiple locations within the blood vessel, and pixels of the Doppler ultrasound image are color coded to indicate the blood flow, further comprising segmenting the color pixels, and the 3D image is reconstructed from the 3D voxels corresponding to the segmented color pixels.

[0016] In a further implementation of the first aspect, the estimated blood flow for each 3D voxel is selected as the maximum value over an imaging time interval, and a Doppler ultrasound image is captured during this imaging time interval depicting the area within the blood vessel corresponding to the 3D voxel.

[0017] In a further implementation of the first aspect, the multiple Doppler ultrasound images used to calculate the 3D voxel are captured over an imaging time interval depicting variations in blood flow, and the reconstructed 3D image includes, for the 3D voxel, a respective indication of the variation in blood flow over the imaging time interval.

[0018] In a further implementation of the first aspect, the reconstructed 3D image is presented as a video over the imaging time interval by varying an indication of blood flow corresponding to the 3D voxels over the imaging time interval.

[0019] According to a second aspect, a computer-implemented method for segmenting a 3D image reconstructed from 2D ultrasound images depicting a body segment of an individual includes obtaining 2D ultrasound images depicting a common region of the body segments and 2D camera images captured by a camera depicting a plurality of fiducial objects randomly distributed in 3D within a viscous material and spaced apart by random distances on a surface of the body segment, and segmenting each 2D image using an external reference of ultrasound transducer pose calculated by analysis of relative changes in positions of the plurality of fiducial objects in successive 2D camera images. The method includes calculating a plurality of 3D voxels having 3D coordinates in a world coordinate system assigned to pixels of the ultrasound image, reconstructing a 3D image from the plurality of 3D voxels, storing, for each 3D voxel, a mapping of a multidimensional sparse data set between a pose of the ultrasound transducer and intensity values ​​acquired at the 3D coordinates corresponding to each 3D voxel and each pose of the ultrasound transducer, clustering the plurality of 3D voxels into a plurality of clusters according to a distribution of the multidimensional data set of the plurality of 3D voxels, and segmenting the 3D image according to the plurality of clusters.

[0020] A further implementation of the second aspect further includes correcting the 3D voxels to their actual scale relative to the world coordinate system by using the size of the fiducial object and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera.

[0021] In a further implementation of the second aspect, calculating 3D coordinates of pixels of each 2D ultrasound image includes calculating a camera pose in the camera coordinate system relative to the world coordinate system by analyzing relative changes in positions of multiple fiducial objects in successive 2D cameras and correcting to an actual scale relative to the world coordinate system by using the size of the fiducial objects and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera, applying a calibration transformation to map pixels of the 2D ultrasound image represented in the ultrasound coordinate system to the camera coordinate system, where the calibration mapping is based on a predetermined relationship between the camera pose and the pose of an ultrasound transducer that captures the 2D ultrasound image, and mapping the pixels of the 2D ultrasound image represented in the camera coordinate system to 3D coordinates in the world coordinate system.

[0022] In a further implementation of the second aspect, the clustering is performed based on manifold learning techniques to find similar voxels within a region with respect to the distribution of some computed value, such as color, intensity, etc., surrounding the texture.

[0023] In a further implementation of the second aspect, the clustering is performed by clustering a plurality of 3D voxels according to similar patterns of change in the ultrasound data captured with respect to different angles and / or distances of a position corresponding to each voxel relative to the ultrasound transducer.

[0024] In a further implementation of the second aspect, the 3D voxels in each cluster indicate respective tissue types, creating respective similar patterns of variation in the ultrasound data captured at different angles and / or distances relative to the ultrasound transducer.

[0025] According to a third aspect, a computer-implemented method for reconstructing 3D images of a surface and interior of a body segment of an individual includes obtaining 2D ultrasound images depicting tissue within the body segment of the individual and 2D camera images captured by a camera depicting a plurality of fiducial objects randomly distributed in 3D within a viscous material and spaced by random distances on a surface of the body segment and depicting the surface of the body segment; calculating 3D coordinates of pixels of each 2D camera image and 3D coordinates of pixels of each 2D ultrasound image in a common coordinate system based on an analysis of relative changes in positions of the plurality of fiducial objects in successive 2D camera images; and reconstructing 3D images from 3D voxels in the common coordinate system calculated by aggregating the 3D coordinates of the 2D camera images and the 2D ultrasound image, wherein the 3D images depict the surface of the body segment and the tissue within the body segment positioned relative to a surface of a patient, and the reconstructed 3D images include at least one reconstructed 3D image.

[0026] In a further implementation of the third aspect, at least one reconstructed 3D image with additional information layers including at least one of Doppler and B-mode depicts tissues selected in any compatible ultrasound organ scanning procedure including at least one of blood vessels, organs, joints, bones, cartilage, non-blood filled cavities, liver, gallbladder, and thyroid, and fuses these tissues into a whole organ or portion of an organ.

[0027] A further implementation of the third aspect further includes correcting the 3D coordinates to their actual scale relative to the world coordinate system by using the size of the fiducial object and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera.

[0028] In a further implementation of the third aspect, the common coordinate system is a real-world coordinate system that depicts real-world distances and positions, and the 3D image depicts the surface of the body segment and tissue within the body segment using distances and relative positions in real-world coordinates.

[0029] In a further implementation of the third aspect, calculating in the common coordinate system includes calculating a camera pose in the camera coordinate system relative to the common coordinate system by analyzing relative changes in positions of multiple fiducial objects in successive 2D camera images and correcting to actual scale relative to the world coordinate system by using a size of the fiducial objects and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera; mapping pixels of the 2D camera image depicting a surface of the body to 3D coordinates in the common coordinate system; applying a calibration transformation to map pixels of the 2D ultrasound image represented in the ultrasound coordinate system to the camera coordinate system, the calibration mapping being based on a predetermined relationship between the pose of the camera and the pose of an ultrasound transducer that captures the 2D ultrasound image; and mapping pixels of the 2D ultrasound image represented in the camera coordinate system to 3D coordinates in the common coordinate system.

[0030] A further implementation of the third aspect further includes repeating acquiring ultrasound images and camera images in a common region of the body segment and repeating calculating in a common coordinate system, and repeating reconstructing the 3D image includes iteratively updating a previous 3D image of a previous iteration with the 3D voxels calculated in the current iteration to obtain an updated 3D image having a higher resolution than the previous 3D image.

[0031] In a further implementation of the third aspect, the method further includes receiving a 3D anatomical image depicting the body segment, the 3D anatomical image being captured by another 3D imaging modality device, and registering between the 3D anatomical image and the reconstructed 3D image according to features extracted from the surface of the body segment.

[0032] According to a fourth aspect, the viscous material includes at least a portion of a fluid containing a plurality of fiducial objects randomly distributed within the viscous material and spaced apart by random distances for application to a body surface of a subject, the plurality of fiducial objects having a size and contrast relative to the viscous material for use in calculating 3D coordinates assigned to pixels of the 2D ultrasound image to obtain 3D voxels depicted by a 2D camera image and used to estimate the orientation of the ultrasound transducer.

[0033] A further implementation of the fourth aspect further enables computing a reconstruction of a 3D ultrasound image depicting either one of the interior of the body or the surface of a body region of a patient.

[0034] In a further implementation of the fourth aspect, the viscous material includes ultrasound gel, and the 3D image is reconstructed from 2D ultrasound images captured by ultrasound energy transmitted through the ultrasound gel.

[0035] In a further implementation of the fourth aspect, the multiple fiducial objects have a small size selected and / or are made from an acoustic material selected so that they are not depicted in a 2D ultrasound image transmitted through ultrasound gel.

[0036] In a further implementation of the fourth aspect, the plurality of fiducial objects are not fixed at specific locations within the viscous material, but rather flow within the viscous material in response to movement of the viscous material.

[0037] In a further implementation of the fourth aspect, each of the plurality of fiducial objects is about 0.5 to 1 millimeter in size.

[0038] In a further implementation of the fourth aspect, the plurality of fiducial objects are made of a material that exhibits visual enhancement in response to ultraviolet light.

[0039] In a further implementation of the fourth aspect, the plurality of fiducial objects are made of a fluorescent material.

[0040] In a further implementation of the fourth aspect, the plurality of fiducial objects are made spherical.

[0041] In a further implementation of the fourth aspect, the density of the plurality of fiducial objects is about 1-1000 per milliliter.

[0042] According to a fifth aspect, a method of treating a vascular condition in an individual includes applying a viscous material, at least in part, to a surface of a body segment of the individual at a location corresponding to a blood vessel of interest, the viscous material comprising at least a portion of a fluid containing a plurality of fiducial objects randomly distributed in 3D and separated by random distances, maneuvering an ultrasound transducer probe having an add-on component including at least one camera along the body surface to simultaneously capture ultrasound images of the blood vessel within the body and camera images depicting the surface of the body segment and depicting the plurality of fiducial objects, analyzing the reconstructions of the 3D images of the blood vessel and the surface of the body segment located within a common 3D coordinate system representing real world coordinates, diagnosing the vascular condition based on the reconstruction of the 3D image of the blood vessel relative to the surface of the body segment, and treating the vascular condition during an open surgical and / or catheter procedure.

[0043] A further implementation of the fifth aspect further includes re-manipulating the probe on the surface of the body to simultaneously capture ultrasound images of the blood vessel depicting the treated vascular condition, analyzing the treated vascular condition with another 3D reconstruction of the blood vessel created from the ultrasound images and the camera images captured during re-manipulation of the probe, and re-treating the treated vascular condition if the treated vascular condition is determined to require another treatment procedure based on the another 3D reconstruction.

[0044] A further implementation of the fifth aspect further includes repeatedly steering the ultrasound transducer in an axial and / or longitudinal orientation of the ultrasound transducer within a small area of ​​the surface corresponding to the vascular pathology to capture multiple ultrasound images of the vascular pathology at different angles and distances, wherein the resolution of the vascular pathology in the 3D image increases as the number of steering of the ultrasound transducer over the small area increases.

[0045] In a further implementation of the fifth aspect, the 3D image is segmented according to tissue type, further comprising analyzing the segmentation of the 3D image to determine the effectiveness of the vascular treatment.

[0046] In a further implementation of the fifth aspect, diagnosing a vascular condition based on reconstruction of a 3D image of a blood vessel relative to a surface of the body segment is further based on blood flow in the blood vessel depicted by the 3D image.

[0047] In a further implementation of the fifth aspect, the therapy is selected from the group consisting of a catheter-delivered stent, balloon dilation, ablation, drug injection, and manual removal and / or repair.

[0048] According to a sixth aspect, a calibration device for calibrating a transformation mapping applied to a camera image captured by a camera positioned at a fixed orientation relative to ultrasound transmission and an ultrasound image captured by an ultrasound transducer includes a single compartment having a circular top surface containing an ultrasound-transparent ultrasonic inspection medium and a ground truth pattern positioned outside the outer perimeter of the circular top surface of the compartment, the compartment and ground truth pattern being positioned such that when the ultrasound transducer is placed on the top of the compartment, the camera captures a camera image of the ground truth pattern and the ultrasound transducer captures an ultrasound image of the interior of the compartment from the circular top surface to the bottom surface.

[0049] According to a seventh aspect, an add-on to an ultrasound probe includes a connector component sized and shaped to connect to a probe of an ultrasound transducer, a first camera housing including a first camera, and a second camera housing including a second camera, the first camera housing and the second camera housing each oriented at a predetermined angle relative to a long axis of the connector component to fix the first camera and the second camera at a predetermined angle relative to the ultrasound probe and relative to an ultrasound image captured by the ultrasound transducer.

[0050] A further implementation of the seventh aspect further includes an ergonomic holder component that includes a grip designed to be held against the palm of a user's hand and a ring or trigger-like element designed to support the user's index finger.

[0051] In a further implementation of the seventh aspect, the first camera housing and the second camera housing are separated by approximately 90 degrees that is perpendicular to the longitudinal axis of the connector component.

[0052] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although methods and materials similar or equivalent to those described herein can be used to practice or test embodiments of the present invention, exemplary methods and / or materials are described below. In case of conflict, the patent specification, including definitions, will control. Furthermore, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.

[0053] Some embodiments of the present invention are herein described, by way of example only, with reference to the accompanying drawings, in which: Reference will now be made specifically to the drawings in detail, stressing that the particulars shown are for the purpose of illustrating and discussing embodiments of the present invention by way of example, and in this regard the description made with the drawings will make apparent to those skilled in the art how embodiments of the present invention may be practiced. [Brief description of the drawings]

[0054] [Figure 1] FIG. 1 is a block diagram of components of a system 100 for generating 3D images from ultrasound and camera images based on analysis of fiducial objects randomly distributed in a viscous material, according to some embodiments of the present invention. [Diagram 2] 1 is a flowchart of a method for generating a 3D image from ultrasound and camera images based on an analysis of fiducial objects randomly distributed in a viscous material, according to some embodiments of the present invention. [Diagram 3] 1 is a flowchart of a method for diagnosing and / or treating a vascular condition according to a 3D image generated from ultrasound images and camera images based on analysis of fiducial objects randomly distributed in a viscous material, according to some embodiments of the present invention. [Figure 4] 1 includes a schematic diagram depicting the process of applying ultrasound gel with fiducial objects randomly distributed therein to capture camera images used to reconstruct a 3D image, according to some embodiments of the present invention. [Diagram 5] FIG. 1 is a schematic diagram depicting a camera capturing camera images of fiducial objects randomly distributed within ultrasound gel through which an ultrasound transducer of an ultrasound probe captures ultrasound images, in accordance with some embodiments of the present invention. [Figure 6] FIG. 1 is a schematic diagram depicting exemplary x, y, z axes of an ultrasound transducer for calculating 3D coordinates within the 3D space Vx, Vx, Vz module of tissue depicted in an ultrasound image according to some embodiments of the present invention. [Figure 7] 1 is a schematic diagram depicting correction of blood flow in ultrasound images of a blood vessel acquired in multiple orientations by an ultrasound transducer according to some embodiments of the present invention. [Figure 8]1A-1C are schematic diagrams illustrating the reconstruction of a 3D image including an image of an artery depicting vascular pathology such as stenosis and indicating blood flow velocity through the artery, according to some embodiments of the present invention. [Figure 9] FIG. 1 is a schematic diagram depicting a process of collecting data for a multi-dimensional (optionally sparse) dataset of a set of 3D coordinates corresponding to a voxel and / or group of voxels, according to some embodiments of the present invention. [Figure 10] FIG. 1 is a schematic diagram illustrating a representation of a multi-dimensional dataset of a particular 3D coordinate of a particular voxel as a sphere, with the grey colour values ​​representing different B-mode greyscale values ​​obtained from various transducer poses, according to some embodiments of the present invention. [Figure 11] FIG. 1 is a schematic diagram illustrating the reconstruction of a 3D image based on segmentation of a cluster of an (optionally sparse) multidimensional dataset containing arteries segmented into different tissue types, according to some embodiments of the present invention, depicting blood flow values. [Figure 12] FIG. 2 is a schematic diagram depicting an exemplary 3D image of the brachial artery calculated based on a camera image depicting a pattern on the surface of a subject's arm and an ultrasound image of the brachial artery, in accordance with some embodiments of the present invention. [Figure 13] FIG. 1 is a schematic diagram depicting an environmental scene of generating a 3D image computed from camera images and ultrasound images with ultrasound gel with randomly distributed fiducial objects, according to some embodiments of the present invention. [Figure 14] FIG. 1 is a schematic diagram depicting an ultrasound probe for capturing camera images by a camera and an add-on to an ultrasound transducer for capturing ultrasound images used to reconstruct a 3D image, according to some embodiments of the present invention. [Figure 15]FIG. 1 is a schematic diagram depicting an exemplary add-on to an ultrasound probe including a connector component sized and shaped to connect to the ultrasound probe and a camera positioned at an angle relative to a long axis of the connector component that corresponds to the long axis of the probe's ultrasound transducer, in accordance with some embodiments of the present invention. [Figure 16] 1 is a schematic diagram of another exemplary implementation of an add-on to an ultrasound probe, according to some embodiments of the present invention. [Figure 17] 1 depicts an exemplary implementation of a calibration device according to some embodiments of the present invention. [Figure 18] 1 depicts another exemplary implementation of another calibration device, according to some embodiments of the present invention. [Figure 19] FIG. 1 is a schematic diagram illustrating standard vascular treatment using existing approaches, in accordance with some embodiments of the present invention; and FIG. 2 is a schematic diagram illustrating vascular treatment using a 3D artery of 3D point clouds and / or meshes created from ultrasound images and camera images. [Figure 20] FIG. 1 is a schematic diagram depicting an estimate of blood flow in a vessel obtained by an ultrasound transducer at multiple orientations and / or positions, calculated using a known 3D vascular model and computational fluid dynamics simulations, according to some embodiments of the present invention. [Figure 21A] 1 is a schematic diagram of another exemplary implementation of an add-on to an ultrasound probe, according to some embodiments of the present invention. [Figure 21B] 1 is a schematic diagram of another exemplary implementation of an add-on to an ultrasound probe, according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0055] The present invention, in some embodiments thereof, relates to the processing of medical ultrasound images, and more particularly to the reconstruction of 3D medical images from 2D ultrasound images.

[0056] The embodiments described herein may relate to 3D medical images depicting organs compatible with holistic imaging of perforated organs (e.g., liver, gallbladder, thyroid, joints, etc.) with or without blood flow, with or without Doppler (e.g., grayscale only).

[0057] The embodiments described herein may relate to 3D medical images depicting one or more blood vessels, e.g., femoral arteries, carotid arteries, aorta, and / or other body tissues, e.g., organs such as kidneys, liver, gallbladder, spleen, and thyroid. Body parts, e.g., joints, bones, cartilage, etc., may not necessarily have blood vessels large enough to be depicted in the image. Body parts may depict body parts that contain air, e.g., air-filled organs such as lungs and intestines. Body parts, if filled with fluid, may not necessarily be blood, and / or non-blood filled cavities, e.g., bladders and abscesses, may not necessarily flow at significant rates.

[0058] The embodiments described herein with respect to Doppler are not necessarily limiting examples, and other ultrasound imaging modalities may alternatively or additionally be used, such as B-mode, etc. As used herein, the term Doppler may be interchanged with the term B-mode where relevant.

[0059] As used herein, the term Doppler ultrasound image refers to at least one Doppler ultrasound image, the term 2D camera image refers to at least one 2D camera image, the term reconstructed 3D image refers to at least one reconstructed 3D image, and similarly for other images.

[0060] An aspect of some embodiments of the invention relates to a viscous material, including at least a portion of a fluid (optionally sonography gel) containing fiducial objects, the fiducial objects being randomly, i.e., in three dimensions, distributed within the viscous material (e.g., sonography gel) and spaced apart at random distances. The fiducial objects can be, for example, small spheres of different colors, with diameters, for example, in the range of 0.5 to 1 millimeter. The viscous material, such as sonography gel, is designed to be applied to a surface of a subject's body, such as the skin, to transmit ultrasound energy between an ultrasound transducer in contact with the surface and anatomical structures within the body to capture a 2D ultrasound image of the interior of the body. The fiducial objects have a size and contrast relative to the viscous material, such as sonography gel, and are depicted by a 2D camera image and used to calculate 3D coordinates assigned to the 2D ultrasound image, for example, 3D coordinates assigned to 2D pixels within the 2D ultrasound image, to generate a reconstruction of the 3D ultrasound image depicting the interior of the body.

[0061] Blood flow may refer to blood flowing within a blood vessel, such as an artery and / or vein. Blood may flow, for example, through a stenosis (e.g., as described herein), over plaque, and / or over the inner wall of the blood vessel. Blood flow may be measured using velocity, which may indicate the speed and / or direction of blood flow. Blood velocity may be measured, such as in centimeters per second. Blood volume may be measured in blood flow, such as in cubic centimeters per second. Direction may be measured in degrees relative to a baseline, such as in degrees relative to the long axis of the blood vessel. Exemplary blood vessels include the carotid artery, aorta, femoral artery, superior and / or inferior vena cava, brachial, renal artery, hepatic artery, and / or other blood vessels in other locations in the body. Other details of blood flow are described herein.

[0062] An aspect of some embodiments of the present invention relates to the reconstruction of a 3D image depicting blood flow at multiple 3D locations of a blood vessel, such as the entire imaged vessel. As referred to herein, a 3D image may be described as a 3D point cloud, a mesh 3D object, a graph 3D, or the like. A Doppler ultrasound image is acquired depicting the blood vessel with blood flowing therethrough and a measurement of blood flow within a region of the vessel. For example, a 2D ultrasound image depicts the vessel and a measurement of blood velocity at the pixels of the 2D ultrasound image. The image may be a standard B-mode image associated with a standard Doppler mode measurement. A 2D camera image is acquired depicting fiducial objects randomly distributed in 3D within a viscous material, such as an ultrasound gel, and spaced apart by random distances on the surface of a body segment, captured by a camera. The camera image may be acquired simultaneously with the ultrasound image, for example, the camera is an add-on to an ultrasound probe. For each Doppler ultrasound image pixel, a 3D coordinate in a world coordinate system depicting a real-world distance and position is calculated. The world coordinate system defines the scale of the 3D coordinates, which may be in Euclidean space and correspond to real-world distances and positions, i.e., no spatial transformation is applied. The scale of the world coordinate system may be 1:1 or another scale using real-world distances, e.g., a 5 cm distance between the skin and the artery depicted in the image represents a 5 cm distance inside the actual subject's body. The 3D coordinates are calculated using an external reference of the pose of the ultrasound transducer used to capture the Doppler images. This external reference is calculated by analysis of the relative change in the position of the fiducial object in successive 2D camera images captured when the ultrasound probe is manipulated while the Doppler images are captured. The 3D position and / or pose of the 2D camera in the external reference is calculated based on the analysis of the relative change in the position of the fiducial object. The 3D position and / or pose of the ultrasound transducer is calculated from the 3D position and / or pose of the 2D camera, e.g., based on a known calibrated fixed transformation between the position and / or orientation of the camera and the ultrasound transducer.According to the position and / or orientation of the camera and the ultrasound transducer, the position of the Doppler image in the external reference is calculated. The position of the Doppler image in the external reference is mapped to the world coordinate system by a calibration mapping function. A correction of the actual scale of the structure and / or motion of the transducer in world units may be obtained by additional information, such as estimating an a priori known size of the fiducial object, e.g., the diameter of a sphere, and / or by additional information from additional sensors, e.g., an inertial measurement unit, additional cameras, etc. A respective estimated blood flow is calculated for a number of pixels of the Doppler ultrasound image at a number of positions in the blood vessel. The measured blood flow is corrected to obtain a real blood flow value by estimating the angle between the ultrasound transducer axis transmitting the US beam and a vector indicating the direction of blood flow in the blood vessel. A 3D image is reconstructed from the 3D voxels in the world coordinate system. The 3D voxels are calculated by assigning 3D coordinates to the pixels of the Doppler ultrasound image. Each 3D voxel is associated with a respective estimated blood flow. The 3D images depict an anatomical image of a blood vessel and an indication of blood flow at multiple locations within the vessel. For example, different blood velocities can be color coded to allow for visual presentation of locations of high blood flow within the vessel and / or locations of low blood flow within the vessel. This allows for diagnosis of vascular pathologies, such as stenosis, where high blood flow velocity occurs within a narrow region of the vessel.

[0063] An aspect of some embodiments of the invention relates to the reconstruction of 3D images depicting the surface (e.g., skin) and internal structures of a body segment of an individual, e.g., the femoral artery and the skin of a leg. Presenting the surface and internal structures in the same set of coordinates in the same 3D image allows visualization of the position of the internal structures relative to the surface, e.g., the position of the femoral artery relative to the skin of the leg. Acquire 2D ultrasound images depicting the internal tissues in the body segment of the individual. Acquire 2D camera images captured by a camera depicting the surface of the body segment and depicting fiducial objects randomly distributed in 3D in a viscous material (e.g., ultrasound gel) and spaced at random distances on the surface of the body segment. The camera and ultrasound images may be acquired simultaneously and / or near simultaneously. The 3D coordinates of the pixels of the 2D camera images and the 3D coordinates of the pixels of the 2D ultrasound images are calculated in a common coordinate system and / or scale corrected by using two cameras and / or by an inertial measurement unit or by prior knowledge of the average size of the fiducial objects, such as spheres. The 3D coordinates are calculated based on an analysis of the relative change in the position of the fiducial object in successive 2D camera images. The 3D voxels are calculated by assigning 3D coordinates to pixels in the camera image and the ultrasound image. The 3D image is reconstructed by calculating the position of the 3D voxels based on the motion of the transducer in the world coordinate frame, which is determined by using the motion of the camera relative to fiducial markers embedded in the ultrasound gel. The 3D reconstruction (e.g., the 3D image) also depicts the surface of the body segment and tissues within the body segment positioned relative to the surface. The reconstructed 3D image, with additional layers of information including Doppler and / or B-mode, etc., depicts tissues selected in any compatible ultrasound organ scanning procedure, including at least one of blood vessels, organs, joints, bones, cartilage, non-blood filled cavities, live gallbladder, and thyroid, and fuses the tissues into one whole organ or part of an organ.

[0064] An aspect of some embodiments of the invention relates to segmenting a 3D image reconstructed from 2D ultrasound images. Acquire 2D ultrasound images depicting common regions of body segments. Acquire 2D camera images captured by a camera depicting fiducial objects randomly distributed in 3D in a viscous material (e.g., sonography gel) and spaced at random distances on the surface of the body segment. The camera and ultrasound images may be acquired simultaneously and / or near simultaneously. Calculate 3D voxels having 3D coordinates in a world coordinate system. The 3D voxels are calculated by assigning 3D coordinates to pixels of the 2D ultrasound images using an external reference of the pose of the ultrasound transducer capturing the ultrasound images. The pose of the ultrasound transducer is calculated by analyzing the relative changes in the positions of the fiducial objects in successive 2D camera images and / or is corrected by using two cameras and / or by an inertial measurement unit or by prior knowledge of the average size of the fiducial objects, such as spheres. Reconstruct the 3D image from the 3D voxels. For each 3D voxel, a multi-dimensional dataset is calculated and stored. The multi-dimensional dataset may (for example, typically) be a sparse dataset. When the ultrasound transducer moves and / or is reoriented, different intensity values ​​are acquired for the same 3D coordinates corresponding to the same 3D voxel for different postures of the ultrasound transducer. The multi-dimensional dataset maps between each posture of the ultrasound transducer and the intensity values ​​acquired at the 3D coordinates corresponding to each 3D voxel acquired for each posture of the ultrasound transducer. The 3D voxels are clustered into multiple clusters according to the distribution of the multi-dimensional dataset of the multiple 3D voxels. The 3D image is segmented according to a subset of 3D voxel members of each cluster. The voxels in each segmented cluster may represent similar tissue.Clustering may be performed by clustering the voxels according to some similarity measure of the distribution of grayscale and / or color (e.g., colored patterns of variation) Doppler values ​​obtained by capturing ultrasound data from different angles and / or distances of the location corresponding to each voxel relative to the ultrasound transducer, with the voxels in each cluster indicating respective tissue types that produce respective similar patterns of variation in the ultrasound data captured at different angles and / or distances relative to the ultrasound transducer.

[0065] When using ultrasound techniques to image organs and images of organs, extra noise is recorded and interferes with the actual image of the organ being imaged. The segmentation approach described herein also increases the signal-to-noise ratio (SNR) of the captured data. When imaging a specific region of the body, multiple images of the same region are acquired. For the same pixel, corresponding to a voxel and / or 3D location, when viewed from different angles and different distances from the ultrasound probe, each pixel, which is converted to a voxel, is associated with multiple grayscale values, i.e., intensity values, for each pose and distance. Each human tissue, including pathological tissue, may have the same grayscale value recorded from different angles and distances. An optional unsupervised learning approach, such as manifold learning, identifies those tissues that have the same grayscale value and similar grayscale value variations and / or distributions. The same tissue type is optionally segmented with some similarity measure that defines a cluster. Since the similarity between voxels is based on grayscale values ​​and grayscale distributions, which may include topological relationships between voxels, adjacent voxels need to be mapped and segmented together in the mapping space, and a higher accuracy and / or precision of segmentation of highly complex tissues may be obtained.

[0066] An aspect of some embodiments of the invention relates to the treatment of vascular pathologies within a blood vessel of an individual. A viscous material, which is a fluid (optionally an ultrasound gel) at least some of which is randomly distributed in 3D and includes fiducial objects spaced apart at random distances, is applied to a surface of a body segment of the individual at a location corresponding to the blood vessel of interest. An ultrasound transducer probe with add-on components including one or more cameras is manipulated along the surface of the body and may include an inertial measurement unit. An ultrasound image of the blood vessel within the body and a camera image depicting the surface of the body segment and depicting the fiducial objects are simultaneously captured. A reconstruction of a 3D image of the blood vessel and the surface of the body segment located within a common 3D coordinate system representing real-world coordinates depicting real-world distances and positions is generated. Alternatively or additionally, the 3D image depicts an indication of blood flow (e.g., velocity) at multiple locations within the blood vessel. Alternatively or additionally, the 3D image is segmented into multiple segments, each segment depicting a respective tissue type, e.g., a different layer of the blood vessel wall, a material causing stenosis, etc. Diagnosing vascular pathologies (e.g., stenosis) is based on reconstruction of a 3D image of the blood vessel relative to the surface of the body segment and / or based on blood flow patterns within the vessel and / or based on the segmented 3D image. Vascular pathologies are treated during open surgery and / or catheter procedures. The 3D reconstruction (e.g., the 3D image) can be used to guide a treatment tool (e.g., a catheter).

[0067] At least some implementations of the systems, methods, devices and / or code instructions described herein address the technical problem of improving the accuracy of the calculation of 3D coordinates assigned to 2D image elements, e.g., pixels of a 2D ultrasound image depicting an internal body structure to reconstruct a 3D image using the 3D coordinates, e.g., each 2D pixel is converted to a 3D voxel of the 3D image by calculating a corresponding 3D coordinate. At least some implementations of the systems, methods, devices and / or code instructions described herein improve the technique of reconstructing a 3D image from 2D ultrasound imaging data by improving the accuracy of the 3D coordinates assigned to image elements, e.g., pixels, of the 2D ultrasound image. At least some implementations of the systems, methods, devices and / or code instructions described herein provide a solution to the technical problem and / or an improvement to the technique is provided by fiducial objects randomly, i.e., distributed in 3D and spaced apart at random distances, within a viscous material (optionally an ultrasound gel). The fiducial objects have a size and / or contrast relative to the viscous material depicted by the 2D camera image that may be used to calculate the 3D coordinates assigned to the 2D US image and aid in the correction of the scale of the reconstructed 3D image. The 2D (e.g., monocular RGB) camera may be located as an add-on and / or integrated with the ultrasound probe, capturing images of the ultrasound gel containing the fiducial objects, which captures the 2D images by transmitting energy through the ultrasound gel containing the fiducial objects. The small size, random distribution and / or random spacing, and / or large density of the fiducial objects provide a highly textured image necessary for accurate calculation of the 3D coordinates. This improvement is over other existing approaches that use, for example, surface skin features, temporary tattoos with features, and other expensive sensors to sense the probe's pose, which are prone to errors and / or inaccurate in calculating very accurate 3D coordinates.

[0068] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of measuring blood flow at multiple locations in a blood vessel using Doppler. At least some implementations of the systems, methods, devices, and / or code instructions described herein improve the technique of analyzing Doppler data to measure blood flow and / or improve the technique of reconstructing 3D images from 2D ultrasound imaging data by generating a 3D image of the blood vessel depicting blood flow at multiple locations in the vessel using actual estimated blood flow values. At least some implementations of the systems, methods, devices, and / or code instructions described herein provide a solution to the technical problem and / or an improvement to the technique is provided by correcting an initial measurement of blood flow made using Doppler to obtain an actual blood flow value. Using a standard approach, a user places an ultrasound probe relative to the long axis of the blood vessel and changes the vector indicator to correct the actual angle as best as possible based on experience and / or "eyeballing". One or a few individual Doppler measurements are taken. The improvement provided herein is to calculate (herein referred to as "correcting") actual vessel values ​​from Doppler measured blood values ​​when setting the angle between the ultrasound transducer and blood velocity, without the need to manually set the angle of the actual blood flow velocity. Rather than requiring tedious manual setup of assumed blood flow angles, the proposed method calculates blood flow (e.g., velocity) vectors for multiple locations within the vessel, thereby enabling the generation of a visual 3D corrected image of blood flow that visually depicts blood flow along the vessel.

[0069] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of improving the visual representation of body structures depicted in a 3D image reconstructed from a 2D ultrasound image. At least some implementations of the systems, methods, devices, and / or code instructions described herein improve the technique of reconstruction of a 3D image from a 2D ultrasound image. At least some implementations of the systems, methods, devices, and / or code instructions described herein provide a solution to the technical problem and / or an improvement to the technique is provided by including imaging data captured by a 2D camera image. The imaging data captured by the 2D camera image is contained within the same coordinate system used to represent the ultrasound image in the 3D image. The resulting 3D image accurately depicts the position of the internal body structure imaged by the ultrasound image relative to the surface (e.g., skin) of the body structure imaged by the camera after correction to scale, which can be done by integrating additional information such as the average size of spheres in the ultrasound gel, or information from an additional sensor such as an inertial measurement unit or another camera. For example, while performing an invasive procedure on the femoral artery, a user may visually inspect the 3D image to determine where the thigh is located within the leg to improve guidance of a tool to the femoral artery.

[0070] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of segmenting a 3D image reconstructed from a 2D ultrasound image. At least some implementations of the systems, methods, devices, and / or code instructions described herein improve the technique of image processing by segmenting a 3D image reconstructed from a 2D ultrasound image. At least some implementations of the systems, methods, devices, and / or code instructions described herein provide a solution to the technical problem and / or an improvement to the technique is provided by a multi-dimensional data set that maps between each pose of an ultrasound transducer and intensity values ​​acquired at 3D coordinates corresponding to each 3D voxel acquired for each pose of the ultrasound transducer. The 3D voxels are clustered into a plurality of clusters according to a distribution of the multi-dimensional data set of the plurality of 3D voxels. The 3D image is segmented according to a subset of the 3D voxel members of each cluster. The voxels in each segmented cluster may represent similar tissue. Such segmentation is an improvement over standard segmentation approaches, which are based on visual features in, for example, 2D ultrasound images. The same tissue may appear differently in different ultrasound transducer poses, and therefore may be inaccurately segmented using standard approaches. Furthermore, some ultrasound transducer poses may have artifacts, which may reduce the power and / or accuracy of the segmentation. In contrast, as described herein, the same tissue type is segmented based on the pattern of distribution of intensity values ​​for multiple ultrasound transducer poses, and may also use neighboring distribution information, improving the accuracy of segmenting the same tissue type.

[0071] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the medical problem of improved diagnosis of vascular conditions and / or improved treatment of vascular conditions. At least some implementations of the systems, methods, devices, and / or code instructions described herein improve the field of medicine by improved diagnosis of vascular conditions and / or improved treatment of vascular conditions. At least some implementations of the systems, methods, devices, and / or code instructions described herein provide a solution to a medical problem and / or an improvement to medicine is provided by reconstruction of 3D images depicting the surface of blood vessels and body segments located in a common 3D coordinate system, and / or 3D images depicting indications of blood flow (e.g., velocity) at multiple locations within a blood vessel, and / or 3D images segmented into multiple segments each depicting a respective tissue type. The 3D images improve a user's ability to make a diagnosis, such as identifying a stenosis, to perform a treatment, such as guiding a catheter from the skin to the stenosis, and / or to evaluate the outcome of a treatment, such as checking blood flow patterns with a stent in place.

[0072] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of computing 3D and / or 4D images, particularly without the use of radiation such as x-rays. At least some implementations of the systems, methods, devices, and / or code instructions described herein improve the technical field of ultrasound imaging, particularly ultrasound images, by computing 3D and / or 4D ultrasound images from 2D ultrasound images, optionally using fiducial objects randomly distributed in an echography gel (or another viscous medium). The generated 3D and / or 4D images depict larger tissue regions depicted in each 2D ultrasound image by aggregating voxels created by assigning 3D coordinates to pixel intensity values ​​of pixels of the 2D ultrasound images. The generated 3D and / or 4D images may indicate blood flow and / or blood volume. The 2D ultrasound images can be acquired sequentially, optionally using a standard 2D ultrasound probe. The 2D ultrasound images can be captured as B-mode and / or color mode (Doppler) images.

[0073] At least some implementations of the systems, methods, devices, and / or code instructions described herein provide a fixed pattern to be depicted in a camera image captured by a camera associated with an ultrasound transducer, addressing the technical problem of generating a 3D image from an ultrasound image captured by the ultrasound transducer and / or improving the technical field. A transformation mapping maps the fixed pattern in the camera image captured by the camera to 3D coordinates assigned to the 2D ultrasound image captured by the ultrasound transducer. A scale correction to the actual scale of the structure and motion in world units can be recovered by adding known fiducial markers or by integrating information from an additional sensor, e.g., an inertial measurement unit or another camera. Other conventional approaches have used features of the skin itself, such as hairs, wrinkles, birthmarks, external sensors such as inertial sensors that measure the transducer's pose, and / or temporary tattoos placed on the skin. At least some implementations of the systems, methods, devices, and / or code instructions described herein relate to ultrasound gel with randomly distributed fiducial objects. The fiducial objects may be small enough and / or made of a selected acoustic material to prevent artifacts on the ultrasound image. The fiducial objects may be large enough to be depicted in the camera image. Ultrasound gel is used to capture the ultrasound image and also contains randomly distributed fiducial objects that provide a fixed pattern depicted in the camera image.

[0074] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical and / or medical problem of treating vascular pathologies in a subject. Using standard approaches, vascular pathologies are mapped by continuous 2D ultrasound imaging, optionally in Doppler mode, and / or by performing angiograms such as x-ray, CT, MRI, etc., which are invasive procedures that inject contrast agents that may cause allergic reactions and / or may be nephrotoxic into the subject's vasculature and / or expose the subject to high radiation doses. At least some implementations of the systems, methods, devices, and / or code instructions described herein improve the technical and / or medical field of treating vascular pathologies in a subject and / or address the technical and / or medical problem by generating 3D images from 2D ultrasound images, optionally including Doppler, that may be acquired quickly in a non-invasive manner without exposing the subject to radiation and / or without injecting contrast agents. The 3D images may be captured in real time to guide catheters used to treat vascular pathologies, for example by inserting stents, balloon dilatation, ablation, injecting drugs, and / or other approaches, and / or may be used to evaluate vascular pathologies before and after treatment. Real-time evaluation after treatment using 3D images may help determine if further immediate treatment is needed and / or if the treatment was successful. After treatment, follow-up 3D images can be acquired to determine if the treatment is successful over a longer period of time or if additional treatment is needed. Multiple 3D images such as these cannot be acquired using standard approaches to evaluate treatments due to the invasiveness of the standard approaches and / or the risk of large amounts of radiation and / or contrast agents.

[0075] In at least some implementations, the improvement is that further features are added to the acquired images using ultrasound gel or other viscous material in which fiducial objects are randomly distributed, allowing for a more robust and accurate estimation of the transducer pose, resulting in a more accurate and precise 3D image, the 3D image depicts the inside of the blood vessel with the skin in real-world coordinates depicting real-world relative distances and positions, the 3D image is corrected for blood flow (e.g., velocity, speed) at multiple locations within the vessel, and / or the 3D segmentation is based on a cluster of multi-dimensional data sets corresponding to different voxel locations and distributions, and / or a mapping between the ultrasound transducer pose and the acquired intensities. Each multi-dimensional data set may be of a set of 3D coordinates corresponding to one voxel and / or a group of voxels.

[0076] Traditional approaches have attempted to achieve real-time three-dimensional ultrasound visualization by either using volumetric probes or constantly moving two-dimensional probes. Medical imaging techniques provide clinical audiovisual information about organs for better clinical and decision-making. Thus, medical imaging of the interior of the body is often considered the most prominent technique for clinical analysis and medical intervention. Medical imaging includes a wide range of techniques, but the most common tools are X-rays, computed tomography scans (CT scans), magnetic resonance imaging (MRI), and medical ultrasound (US).

[0077] One of the standard tools in medical imaging is the CT scan. Computed tomography, also called a CT scan, uses a specialized X-ray machine to produce cross-sectional 2D images from a series of 1D X-ray images taken around a single axis of rotation centered on the object. As the bed passes through the rotating "doughnut" shaped X-ray machine, a series of 2D axial images are acquired. These images can be displayed individually or superimposed to form a 3D model. The disadvantage of a CT scan is the high degree of radiation exposure and the need for injections of nephrotoxic iodine as a contrast agent in some cases. In contrast to a CT scan, there is virtually no exposure to ionizing radiation during an MRI procedure. MRI technology uses the magnetic properties of the body to produce detailed and excellent 3D images. However, MRI is very expensive, requires a specific room, cannot be used with patients who have metal pieces embedded in them, are claustrophobic, and is highly affected by patient movement. In a standard ultrasound system, a probe transmits high voltage frequency waves into the patient's body. If these waves encounter the interior of organs with different densities, they are reflected back and produce echoes that are measured by the probe. The returning echoes are converted into electrical signals from which algorithms generate a 2D image. The advantage is the absence of radiation and injection of contrast agents that are harmful to the kidneys. It is also the most cost-effective imaging technique. However, it lacks a 3D image of the entire organ, is highly user-dependent and is therefore considered an "operator-dependent" device, resulting in the poorest imaging quality.

[0078] The use of US technology in the embodiments described herein avoids the radiation and contrast agents currently used in CT and angiography examinations, thereby eliminating the harm to the patient and caregiver from radiation and the potential for future deterioration of the patient's renal function from contrast agents.

[0079] Implementation of at least some of the systems, methods, devices, and / or code instructions described herein addresses the above technical problems and / or improves upon the above technical field by generating 3D ultrasound images, without ionizing radiation and without contrast agents, that can be rapidly captured using a standard US probe used on patients with implanted metallic parts and / or that can depict vascular physiological data such as blood velocity (e.g., centimeters per second) and / or blood volume (e.g., cubic centimeters per second).

[0080] At least some implementations of the systems, methods, devices, and / or code instructions described herein generate 3D US images of blood vessels without x-ray exposure and / or administration of contrast agents, in contrast to standard approaches for capturing angiographic images of blood vessels that require administration of contrast agents to the vasculature and / or capture of x-ray images.

[0081] At least some implementations of the systems, methods, devices, and / or code instructions described herein generate higher quality 3D US images compared to conventional approaches. At least some implementations of the systems, methods, devices, and / or code instructions described herein generate 3D US images with even more accurate segmentation and / or higher resolution by eliminating different noise sources that are drawbacks of standard US systems.

[0082] At least some implementations of the systems, methods, devices, and / or code instructions described herein add an additional layer of information regarding the reconstructed 3D US image, which can depict one or more of the velocity of blood flowing through the imaged vessel and the volume of blood flowing through the imaged vessel. In contrast, CT or MRI are primarily used for vascular imaging due to their capabilities, but cannot display important physiological information, such as can be measured using only the Doppler modality of the US system.

[0083] Implementations of at least some of the systems, methods, devices and / or code instructions described herein may be added to and / or connected to and / or integrated with existing 2D US acquisition systems.

[0084] Implementation of at least some of the systems, methods, devices, and / or code instructions described herein can reduce operator dependency, for example to a minimum, by enabling inexperienced users to operate a US imaging system and perform high quality examinations of patients.

[0085] At least some implementations of the systems, methods, devices, and / or code instructions described herein reconstruct 3D US images that depict a complete image of a tissue and / or organ compared to 2D US images and / or other approaches. The reconstructed 3D US images can be registered with other 3D images acquired by other 3D imaging modalities, such as CT angiography (CTA), MRI angiography (MRA), etc.

[0086] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of increasing the accuracy and / or computational efficiency of calculating the pose (e.g., position and / or orientation) of an ultrasound transducer during capture of an ultrasound image and / or improve the technical field of calculating the pose of an ultrasound transducer during capture of an ultrasound image. The calculated pose of the ultrasound transducer for each ultrasound image can be used to reconstruct a 3D image as described herein. Calculating the pose of the ultrasound transducer by mapping a camera image of a fiducial object in an ultrasound gel or other viscous medium with a calibrated transformation mapping is more accurate and / or less cost-effective, less complex, and / or more computationally efficient (e.g., requires less processing resources and / or less memory resources) compared to other approaches using, for example, cameras capturing natural skin features such as hair, birthmarks, optical tracking systems, mechanical systems, and optical fibers, magnetic sensors, etc., using only inertial measurement units.

[0087] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of reducing operator dependency and / or improving the user experience when performing an ultrasound examination and / or improve the technical field of ultrasound systems designed for use by inexperienced users. Using camera images of different orientations of the ultrasound transducer, the orientation and / or 3D coordinates of the ultrasound transducer relative to the ultrasound image are calculated, allowing the inexperienced user to obtain relevant information that can be further analyzed by an expert. Using the data collected by the inexperienced user, a 3D image is reconstructed.

[0088] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of creating improved ultrasound images of organs with relatively reduced noise and / or artifacts and / or improve the technical field of ultrasound imaging by creating ultrasound images of organs with relatively reduced noise and / or artifacts. Using standard approaches, capturing each 2D ultrasound image independently and reducing noise and / or artifacts relies on the ability of the user to capture the image in an optimal pose. In contrast, the 3D images created herein by aggregating 3D voxel data calculated from multiple 2D images of the ultrasound transducer from different poses reduces noise and / or artifacts in the reconstructed 3D image.

[0089] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of creating a 3D anatomical image of a blood vessel that also depicts an indication of blood flow (e.g., velocity and / or volume) through the vessel and / or improve the medical imaging field by creating a 3D anatomical image of a blood vessel that also depicts an indication of blood flow through the vessel. Standard approaches are designed for one data type, such as CT and MRI, to generate a 3D anatomical image. CT and MRI are primarily used for vascular imaging because of their capabilities, but are unable to display important physiological information obtained by using the Doppler modality of US systems. In contrast, Doppler US is designed to measure blood flow but does not provide a good anatomical image of the blood vessel.

[0090] Before describing at least one embodiment of the invention in detail, it is to be understood that the invention is not necessarily limited in its application to the details of construction and the arrangement of components and / or methods set forth in the following description and / or illustrated in the drawings and / or examples. The invention is capable of other embodiments or of being practiced or carried out in various ways.

[0091] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0092] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exclusive list of more specific examples of computer-readable storage media includes the following: portable computer diskette, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, and any suitable combination of the foregoing. A computer-readable storage medium as used herein should not be interpreted as a transitory signal in itself, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a wave guide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0093] The computer readable program instructions described herein can be downloaded from the computer readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface of each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0094] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, or the like, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may run entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to implement aspects of the invention.

[0095] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0096] These computer readable program instructions may be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing apparatus to manufacture a machine, such that the instructions executing via the processor of the computer or other programmable data processing apparatus create means for performing the functions / acts specified in the flowcharts and / or block diagrams of the block or blocks. These computer readable program instructions may also be stored on a computer readable storage medium capable of instructing a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that a computer readable storage medium having instructions stored thereon includes an article of manufacture including instructions for performing aspects of the functions / acts specified in the flowcharts and / or block diagrams of the block or blocks.

[0097] Furthermore, computer-readable program instructions can be loaded into a computer, other programmable data processing apparatus, or other device to execute a series of operational steps on the computer, other programmable data processing apparatus, or other device to generate a computer-implemented process such that the instructions executing on the computer, other programmable data processing apparatus, or other device perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0098] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur in a different order than the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified functions or acts, or a combination of special-purpose hardware and computer instructions.

[0099] Reference is now made to FIG. 1, which is a block diagram of components of a system 100 for generating a 3D image from ultrasound and camera images based on an analysis of relative changes of fiducial objects randomly distributed in a viscous material, according to some embodiments of the present invention. Reference is also made to FIG. 2, which is a flowchart of a method for generating a 3D image from ultrasound and camera images based on an analysis of relative changes of fiducial objects randomly distributed in a viscous material, according to some embodiments of the present invention. Reference is also made to FIG. 3, which is a flowchart of a method for treating a vascular condition according to a 3D image generated from ultrasound and camera images based on an analysis of relative changes of fiducial objects randomly distributed in a viscous material, according to some embodiments of the present invention. Reference is also made to FIG. 4, which includes a schematic diagram depicting a process of applying ultrasound gel with randomly distributed fiducial objects to capture a camera image used to reconstruct a 3D image, according to some embodiments of the present invention. Also referring to FIG. 5, FIG. 5 is a schematic diagram depicting a camera capturing camera images of fiducial objects randomly distributed in ultrasound gel, and an ultrasound transducer of an ultrasound probe capturing an ultrasound image through the gel, according to some embodiments of the present invention. Also referring to FIG. 6, FIG. 6 is a schematic diagram depicting an exemplary x, y, z axis of an ultrasound transducer for calculating 3D coordinates in a 3D space Vx, Vx, Vz module of tissue depicted in an ultrasound image, according to some embodiments of the present invention. Also referring to FIG. 7, FIG. 7 is a schematic diagram depicting a scheme for estimating actual blood flow in an ultrasound image of a blood vessel acquired in multiple orientations by an ultrasound transducer, according to some embodiments of the present invention. Also referring to FIG. 8, FIG. 8 is a schematic diagram depicting a reconstruction of a 3D image of an artery depicting vascular pathology and indicating blood flow velocity through the artery, according to some embodiments of the present invention. Also referring to FIG. 9, FIG. 9 is a schematic diagram depicting a process of collecting data for a multi-dimensional dataset of a set of 3D coordinates corresponding to a voxel, according to some embodiments of the present invention.Also referring to Fig. 10, Fig. 10 is a schematic diagram depicting a representation of a multidimensional sparse data set of a particular voxel according to some embodiments of the present invention, where the data set is represented as a colored sphere, and the gray values ​​are the acquired grayscale B-mode values ​​of this voxel sparsely sampled from various transducer poses, and another diagram depicts the grayscale sparsely distributed values ​​of several voxels represented as spheres, each voxel located on an XYZ grid point. Also referring to Fig. 11, Fig. 11 is a schematic diagram depicting a reconstruction of a 3D image based on a segmentation of a cluster of a multidimensional data set including an artery segmented into different tissue types having a vascular pathology according to some embodiments of the present invention. Also referring to Fig. 12, Fig. 12 is a schematic diagram depicting an exemplary 3D image of a brachial artery calculated based on a camera image depicting a pattern on the surface of a subject's arm and an ultrasound image of the brachial artery according to some embodiments of the present invention, including the surface of a patient's body. Also referring to FIG. 13, FIG. 13 is a schematic diagram depicting an environmental scene of the generation of a 3D image calculated from a camera image and an ultrasound image by ultrasound gel with randomly distributed fiducial objects according to some embodiments of the present invention. Also referring to FIG. 14, FIG. 14 is a schematic diagram depicting an add-on to an ultrasound probe for capturing a camera image by a camera and an ultrasound transducer for capturing an ultrasound image used to reconstruct a 3D image according to some embodiments of the present invention. Also referring to FIG. 15, FIG. 15 is a schematic diagram depicting an exemplary add-on to an ultrasound probe including a connector component sized and shaped to connect to the ultrasound probe and a camera placed at an angle relative to the long axis of the connector component corresponding to the long axis of the ultrasound transducer of the probe according to some embodiments of the present invention. Also referring to FIG. 16, FIG. 16 is a schematic diagram of another exemplary implementation of an add-on to an ultrasound probe according to some embodiments of the present invention. Also referring to FIG. 17, FIG. 17 is a schematic diagram depicting an exemplary implementation of a calibration device according to some embodiments of the present invention.Also referring to FIG. 18, FIG. 18 is a schematic diagram depicting another exemplary implementation of another calibration device according to some embodiments of the present invention. Also referring to FIG. 19, FIG. 19 is a schematic diagram depicting a standard vascular treatment using an existing approach, and a schematic diagram depicting a vascular treatment using a 3D image created from an ultrasound image and a camera image, according to some embodiments of the present invention. Also referring to FIG. 20, FIG. 20 is a schematic diagram depicting an estimate of blood flow in a vessel acquired by an ultrasound transducer at multiple orientations and / or positions, calculated using a known 3D vascular model and computational fluid dynamics simulation, according to some embodiments of the present invention. Also referring to FIG. 21A-B, FIG. 21A-B are schematic diagrams of another exemplary implementation of an add-on to an ultrasound probe according to some embodiments of the present invention.

[0100] The system 100 may implement the operations of the methods described with reference to Figures 2 to 21 by a hardware processor 102 of a computing device 104 executing code instructions 106A optionally stored in a memory 106.

[0101] Computing device 104 may be implemented as, for example, a client terminal, a server, a virtual server, a radiology workstation, an ultrasound workstation, a PACS server, a virtual machine, a computing cloud, a mobile device, a desktop computer, a thin client, a smart phone, a tablet computer, a laptop computer, a wearable computer, a glasses computer, a watch computer, and a ring computer. Computing 104 may include advanced visualization workstations, which may be provided as add-ons to ultrasound workstations and / or other devices for presenting 3D images computed from ultrasound and camera images, as described herein.

[0102] The computing device 104 may include locally stored software to perform one or more of the operations described with reference to Figures 2-21 and / or may function as one or more servers, such as a network server, web server, computing cloud, virtual server, etc., that provide services, such as one or more of the operations described with reference to Figures 2-21, to one or more client terminals 108, such as ultrasound probes, remotely located ultrasound workstations, remote Picture Archiving and Communication System (PACS) servers, remote Electronic Medical Record (EMR) servers, computing devices that receive ultrasound images and camera images over the network 110, for example, provide Software as a Service (SaaS) to the client terminals 108, provide applications to the client terminals 108 for local download as add-ons to a web browser and / or medical imaging viewer application, and / or provide functionality to the client terminals 108 using a remote access session, such as via a web browser.

[0103] Different architectures based on the system 100 may be implemented. In one example, the computing device 104 provides centralized services to each of a plurality of ultrasound workstations associated with the ultrasound transducers 112 of the respective ultrasound probes. Each ultrasound transducer 112 is associated with a respective camera 114, which is in a fixed orientation relative to the ultrasound transducer, as described herein. The camera 114 may be located in an add-on component 254 that is connected to the probe of the ultrasound transducer 112, as described herein. Pairs of ultrasound images captured by the respective ultrasound transducers 112 and camera images captured by the respective cameras 114 are provided to the computing device 104, for example, via the network 110, via an API, a local application, and / or transmitted using an appropriate transmission protocol via a server 118 of a data repository, e.g., PACS, EMR, etc., and / or via the client terminal 108. The computing device 104 analyzes the pairs of ultrasound images and camera images and calculates one or more 3D images, as described herein. The 3D images may be provided to the client terminal 108 and / or the server 118 for presentation on a display, storage, and / or for further processing. In another example, the computing device 104 provides dedicated services to one ultrasound transducer 112 and corresponding camera 114. For example, the computing device 104 is integrated with an ultrasound workstation connected to the ultrasound transducer 112, e.g., the code 106A is installed on the ultrasound workstation that displays the ultrasound images captured by the ultrasound transducer, and / or the computing device 104 is connected to the ultrasound transducer 112 and camera 114, e.g., a smartphone running code 106A is connected to the ultrasound transducer 112 and camera 114 via a short-range wireless connection, a USB cable, and / or other implementations.The ultrasound images captured by the ultrasound transducer 112 and the camera images captured by the camera 114 are processed by locally installed code 106A and a 3D image is provided for presentation on a display of an ultrasound workstation connected to the ultrasound transducer 112 and / or on a display of a locally connected computing device 104, such as a smartphone. As the ultrasound images and camera images are captured, or immediately after the ultrasound images are captured, the code 106A can provide an additional feature set to the ultrasound workstation connected to the ultrasound transducer 112 by dynamically computing the 3D image in real time or near real time.

[0104] In yet another example, the ultrasound images captured by the ultrasound transducer 112 and the camera images captured by the camera 114 may be stored in a data repository 122A that receives the captured ultrasound images and / or camera images, such as a memory and / or storage device of the computing device 104, a memory and / or storage device on an ultrasound workstation (e.g., 108) connected to the ultrasound transducer 112, an external hard drive connected to a client terminal 108 connected to the ultrasound transducer 112 and the camera 114, a PACS server, and / or an EMR server, and a cloud storage server.

[0105] The computing device 104 can receive ultrasound images and camera images from the ultrasound transducer 112 and / or the camera 114 and / or the data repository 122A using one or more data interfaces 120, such as a wired connection (e.g., a physical port), a wireless connection (e.g., an antenna), a local bus, a port for connecting a data storage device, a network interface card, other physical interface implementations, and / or a virtual interface (e.g., a software interface, a virtual private network (VPN) connection, an application programming interface (API), a software development kit (SDK)).

[0106] The camera 114 may be, for example, a still camera, a video camera, a CMOS, a visible light sensor, a red-green-blue (RGB) sensor such as a CCD and / or CMOS sensor, an ultraviolet camera, an infrared camera, a depth RGBD camera, and the like.

[0107] The hardware processor 102 may be implemented as, for example, a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a digital signal processor (DSP), and an application specific integrated circuit (ASIC). The processor 102 may include one or more processors (homogeneous or heterogeneous), which may be arranged for parallel processing as a cluster and / or as one or more multi-core processing units.

[0108] It is noted that at least some implementations of the systems, devices, methods, and / or code instructions described herein are designed to generate 3D images on a processor with relatively few computational resources, e.g., a CPU on a smartphone, compared to generating 3D images on a high-performance processor such as a GPU. This allows processing of images on easily available computing devices, such as laptops and / or smartphones, without the need to install a high-performance processor. Alternatively, at least some implementations of the systems, devices, methods, and / or code instructions described herein are designed to generate 3D images on a high-performance processor.

[0109] Memory 106, also referred to herein as a program store and / or data storage device, stores code instructions for execution by hardware processor 102, e.g., random access memory (RAM), read-only memory (ROM), and / or storage devices, e.g., non-volatile memory, magnetic media, semiconductor memory devices, hard drives, removable storage, and optical media (e.g., DVD, CD-ROM). For example, memory 106 can store code 106A that implements one or more operations and / or features of the methods described with reference to FIGS. 2-21.

[0110] The computing device 104 may include a data storage device 122 for storing data, e.g., one or more data repositories 122A documenting data such as received ultrasound images and camera images and / or documenting generated 3D images, which may be repeatedly updated in response to additional images, multidimensional data sets used in computing the segmentation, transformation mappings, and / or other data described herein. The data storage device 122A may be implemented as, for example, memory, a local hard drive, a removable storage device, an optical disk, a storage device, and / or as a remote server and / or a computing cloud, such as accessed via the network 110.

[0111] The system 100 may include an inertial measurement unit, IMU (not shown), which may be installed as an add-on to the ultrasonic probe 154 to provide measurements indicative of the attitude of the ultrasonic transducer 112, as described herein.

[0112] The computing device 104 may include a network interface 124 for connecting to the network 110, such as one or more of a network interface card, a wireless interface for connecting to a wireless network, a physical interface for connecting to a cable for a network connection, a virtual interface implemented in software, network communications software providing an upper layer of the network connection, and / or other implementations.

[0113] It is noted that the data interface 120 and the network interface 124 may exist as two independent interfaces, e.g., two network ports, as two virtual interfaces on a common physical interface, e.g., a virtual network on a common network port, and / or may be integrated into a single interface, e.g., a network interface. The computing device 104 may communicate with one or more of the client terminals 108 using the network 110 or another communication channel, e.g., by a direct link, e.g., a cable, wireless and / or indirect link, through an intermediate computing device, e.g., a server, and / or through a storage device. For example, when the computing device 104 functions as a central server, the central server provides a centralized 3D image generation service, e.g., SaaS, to the remote ultrasound workstations 108, which function as client terminals connected to the ultrasound transducers 112 and cameras 114, respectively, to generate 3D images from the remotely acquired ultrasound images and camera images. The server 118 may be implemented, for example, in association with a PACS, and may store ultrasound images and camera images acquired by the ultrasound transducer 112 and camera 114, and / or may store updated versions of the code 106A for upgrades thereof. 3D images may be reconstructed offline from image pairs stored on the server 118.

[0114] The computing device 104 and / or the ultrasound transducer 112 and / or the camera 114 and / or the client terminal 108 and / or the server 118 include and / or communicate with a user interface 126 having mechanisms designed for a user to input data, such as patient data, perform the calibrations described herein, and / or view data, such as reconstructed 3D images. Exemplary user interfaces 126 include, for example, one or more of a touch screen, a display, a 3D headset, a keyboard, a mouse, and voice-activated software using speakers and microphones, a VR / AR headset.

[0115] Optionally, the system 100 includes a viscous medium in which fiducial objects 150 are randomly distributed, e.g., ultrasound gel in which objects (e.g., spheres) are suspended, which are placed within the ultrasound gel and depicted in images captured by the camera 114.

[0116] Optionally, system 100 includes an add-on 154 to the ultrasound probe, as described herein, that places camera 114 in a fixed orientation relative to ultrasound transducer 112. Add-on 154 may include an illumination source that generates, for example, visible light, light of a particular color, ultraviolet light, and / or light for fluorescence activation. The light is selected according to the fixed orientation, characteristics of the fiducial object, e.g., gel, to provide enhanced visual effects, e.g., fluorescence of the fiducial object, to depict the gel fiducial object in the camera image.

[0117] Optionally, the system 100 includes a calibration device 152 for calibrating the transformation mapping 122B, as described herein.

[0118] At 202, a viscous material is provided that is at least partially fluid and includes a plurality of fiducial objects randomly distributed therein. The viscous material can be an ultrasound gel that is designed to be applied to a surface of a body, such as the skin of an individual, with which an ultrasound transducer is placed in contact to capture 2D ultrasound images, such as Doppler, B-mode, etc., that are used to reconstruct a 3D image, as described herein.

[0119] The fiducial objects are randomly, i.e., distributed in 3D within the material and spaced apart by random distances. The fiducial objects have a size and / or contrast relative to the viscous material as depicted by the 2D camera images. The fiducial objects depicted in the camera images are used to calculate 3D coordinates that are assigned to pixels of the 2D ultrasound images to generate reconstructions of 3D ultrasound images depicting the interior of the body and / or the exterior of the scanned body segment.

[0120] The fiducial object may be selected so as not to interfere with the ultrasound energy transmitted through the ultrasound gel to capture the ultrasound image and / or to avoid artifacts caused by the presence of the fiducial object, for example, the fiducial object may be made small in size so as not to interfere with the transmitted ultrasound energy and / or may be made from an acoustic material that transmits ultrasound energy.

[0121] Because the fiducial objects cannot be fixed at a specific location within the ultrasound gel, the fiducial objects flow and / or otherwise change position within the material in response to movement (e.g., shaking) of the material.

[0122] Exemplary sizes of the fiducial objects include about 0.5 to 1 millimeters (mm), or about 0.3 to 1.5 mm, or about 0.5 to 1.5, or about 0.3 to 0.7 mm, or other ranges.

[0123] Exemplary densities of fiducial objects include about 10-1000 per milliliter (mL), or about 5-50, or about 25-75 per mL, or other ranges.

[0124] Exemplary shapes for fiducial objects include spheres, squares, stars, snowflakes, cubes, and any other shape.

[0125] The fiducial object may be colored to contrast with the individual's skin so that the fiducial object is easily identifiable by code analyzing the camera image captured by the camera. For example, the fiducial object may be made of a fluorescent material, a material that enhances in response to ultraviolet light (UV), and / or has a bright color such as yellow, green-black, red, blue, or may be made in a variety of colors, etc.

[0126] Referring back now to FIG. 4, FIG. 4 includes a schematic diagram of ultrasound gel 404 with fiducial objects 406 randomly distributed for capturing an image camera used to reconstruct a 3D image. The ultrasound gel 404 may be applied, for example, from a container, such as by squeezing a plastic bottle. The ultrasound gel 404 may be spread and applied in a standard manner onto the subject's skin above the tissue to be imaged and its surroundings. Optionally, the ultrasound gel 404 is spread over a larger area of ​​the body to generate a 3D image reconstruction of a tissue area larger than the area depicted by an individual 2D ultrasound image, for example, organs such as the liver, kidneys, and / or vascular trees and / or trees. Optionally, the ultrasound gel 404 is spread over an area large enough to be captured in the camera image. An ultrasound transducer moves over the ultrasound gel 404 to capture ultrasound images, and a camera, e.g., in an add-on connected to the ultrasound transducer, captures camera images of fiducial objects 406 randomly distributed within the gel 404 as described herein, and uses these camera images to reconstruct a 3D image.

[0127] Randomly distributed fiducial objects 406 provide multiple features across the camera image to calculate 3D coordinates that are assigned to 2D pixels and obtain 3D voxels to reconstruct the 3D image. Having multiple features spread across the camera image can improve the accuracy of the 3D voxel location.

[0128] The fiducial object 406 has a size and / or color characteristic selected to be depicted in a camera image captured by a camera when the ultrasound gel 406 spreads on the surface of the subject's skin. The fiducial object 406 may have a small size selected and / or is made from a selected acoustic material so that it is not depicted in an ultrasound image captured by an ultrasound transducer through the ultrasound gel 404. The size of the fiducial object 406 may be, for example, about 0.5-3 millimeters (mm), or about 0.5-1 mm, or about 1-2 mm, or other ranges. Optionally, the gel 404 is transparent, but the color characteristic is non-transparent, and optionally of one or more different colors, for example, yellow, red, green, black, and blue. Optionally, the fiducial object 406 is made of a material that provides a color characteristic that is enhanced in response to ultraviolet light, and / or is made of a fluorescent material. The fiducial objects 406 may be of different shapes, such as spherical, square, triangular, diamond, or other shapes. The density of the fiducial objects 406 in the ultrasound gel 404 may be about 10-1000 per milliliter (mL), or about 5-50, or about 25-75, or about 10-100, or about 100-500, or about 300-700, or other ranges.

[0129] 2, at 204, an ultrasound image is obtained depicting body tissue captured by an ultrasound transducer, the ultrasound image depicting common areas of body segments, such as blood vessels, e.g., femoral arteries, carotid arteries, aorta, or other body tissue, e.g., kidney, liver, gallbladder, spleen, thyroid gland.

[0130] The ultrasound images may include Doppler 2D mode images and / or B-mode images or other 2D / 3D modes.

[0131] Body tissues (e.g., blood vessels) may include moving fluids. In the case of blood vessels, the ultrasound image may further include Doppler data, also referred to herein as a Doppler image. The ultrasound image includes a region of the blood vessel and depicts blood flowing therein. The Doppler ultrasound image further includes a measurement of blood flow within the region of the vessel. The blood flow measurement may be an instantaneous blood flow measurement, such as a maximum velocity, and / or a pattern of change in blood flow, such as velocity, over one or more cardiac cycles.

[0132] Optionally, for example when blood vessels are being imaged, Doppler and B-mode ultrasound images are taken alternately.

[0133] At 206, a 2D camera image captured by a camera is obtained. The camera image is optionally a visible light image captured by a standard video camera and / or a still camera. The camera image depicts a pattern of features, optionally depicting fiducial objects within an ultrasound gel or other viscous material. It is noted that other patterns of features may be used, for example, temporary tattoos placed on the skin and / or features of the skin itself, such as hair, birthmarks, wrinkles, etc.

[0134] The camera is placed at a known fixed orientation relative to the ultrasound transducer. Optionally, the camera is installed as an add-on to the ultrasound transducer and is designed to be placed at a fixed orientation.

[0135] Optionally, the camera image and the ultrasound image are captured simultaneously or nearly simultaneously.

[0136] Optionally, the camera images and the ultrasound images are captured as a pair, i.e., with synchronous and / or matching video frame rates. Alternatively, the camera images and the ultrasound images are captured independently, for example, with different video frame rates.

[0137] Referring back now to FIG. 5, FIG. 5 is a schematic diagram depicting a camera 502 capturing camera images depicted in a field of view 504 of fiducial objects 506 randomly distributed within an ultrasound gel 512 or other viscous material, and an ultrasound transducer 508 of an ultrasound probe 510 capturing ultrasound images through the gel. The camera images depicting the fiducial objects 506 randomly distributed within the ultrasound gel and the ultrasound images are used to calculate 3D voxels to reconstruct a 3D image, as described herein. The camera 502 may be located in an add-on 514 connected to the ultrasound probe 510, and as described herein, the add-on 512 is positioned to fix the angle of the camera 502 relative to the ultrasound transducer 508, e.g., the view for capturing the ultrasound images, e.g., the angle of the view for capturing the camera images.

[0138] Referring back now to FIG. 2, at 208, 3D voxels are calculated having 3D coordinates in a world coordinate system. The world coordinate system represents the body segments of the individual being imaged and defines real-world distances and / or real-world positions, i.e., there is no distortion. The world coordinate system may be a 3D Euclidean space in which the body segments are located and is isotopically scaled to real-world distances (e.g., 1:1 or other linear scale) and / or is in a non-deformed space. The 3D voxels are calculated by assigning 3D coordinates to pixels of the 2D ultrasound images and / or Doppler images using an external reference of the pose of the ultrasound transducer that captured the ultrasound images. The pose of the ultrasound transducer is calculated by analyzing the relative change in the position of the fiducial object in successive 2D camera images.

[0139] By calculating the camera pose in the camera coordinate system relative to the world coordinate system, 3D coordinates can be calculated for the pixels of the ultrasound and / or Doppler images. The actual scale of the structure and motion in world units may be calculated by additional information such as the size of the object in the 2D camera image, such as the diameter of a sphere embedded in ultrasound gel, or by adding information from another sensor such as an inertial measurement unit or by another camera. The camera pose may be calculated by analyzing the relative change in the position of the fiducial object in successive 2D camera images, for example using optical flow, structure from motion (SfM), visual odometry, or by visual simultaneous localization and mapping (Visual SLAM), etc. A calibration mapping, e.g., a transformation matrix, may be used to map the pixels of the ultrasound and / or Doppler images, expressed in the ultrasound coordinate system of the ultrasound transducer, to the camera coordinate system. The mapping may be based on a pre-calibrated relationship between the pose of the camera and the pose of the ultrasound transducer. Pixels of the ultrasound and / or Doppler images, expressed in the camera coordinate system, are mapped to 3D coordinates in the world coordinate system, and voxels are created by assigning pixel intensity values ​​of pixels of the ultrasound and / or Doppler images to corresponding 3D coordinates in the world coordinate system.

[0140] 6, which is a schematic diagram depicting exemplary x, y, z axes 606 of an ultrasound transducer 602 for calculating 3D coordinates in a 3D space Vx,Vx,Vz module 608 of tissue 604 depicted in an ultrasound image, i.e., a volume of interest. The orientation of the ultrasound transducer 602 defined by the axes 606 and / or calculation of 3D voxels for tissue 604 in the 3D space 608 is calculated based on a camera image captured by a camera 610 connected to an ultrasound probe 612 of the ultrasound transducer 602, e.g., housed in an add-on component 614, as described herein.

[0141] Referring now back to FIG. 2, three types of 3D images can be computed: 3D images that can be contained within a single 3D image, and / or that can be presented as different 3D images, and / or that can be presented as different overlays of the same 3D image.

[0142] Features 210-214 are for computing a 3D image depicting blood flow within a blood vessel.

[0143] The features 216-222 are for segmenting the 3D image.

[0144] Features 224-226 are for computing a 3D image depicting the body surface.

[0145] At 210, an estimated (e.g., initial) blood flow is calculated and / or obtained for multiple locations within the vessel. Optionally, an estimated (e.g., initial) blood flow is calculated and / or obtained throughout the vessel, e.g., along the length and / or diameter of the vessel.

[0146] Many or all of the (e.g., initial) estimated blood flow values ​​for different locations within the vessel are measured at non-90 degree angles between the long axis of the ultrasound transducer (e.g., with the ultrasound probe) and a vector indicating the direction of blood flow within the vessel. Measurements made at non-90 degree angles will be erroneous because Doppler blood flow measurements are designed to be made at non-90 degree angles.

[0147] At 212, the blood flow values ​​measured at non-90 degree angles are corrected to account for the actual blood flow velocity vector along the artery.

[0148] Various approaches may be used to correct the measured blood flow to obtain the actual blood flow. The first, second and third approaches described below are for calculating the flow velocity, e.g., meters / sec. The fourth approach described below is for volumetric flow (e.g., ml / sec). The fifth approach described below is an estimation of the total blood flow velocity vector field:

[0149] In a first approach, an elliptical boundary of the blood vessel is identified. The elliptical shape can be identified, for example, by a neural network trained on a training dataset of ultrasound images labeled with elliptical boundaries, by matching to a template of elliptical boundaries, by applying a filter to the image to contrast edges, using an edge shape process to find the boundary, calculating the shape of the boundary, and / or by other image processing approaches. A transformation from elliptical to circular is calculated. This transformation is applied to the blood flow measurement to obtain an estimated actual blood flow.

[0150] In a second approach, an estimate of the vessel's longitudinal axis is calculated based on an aggregation of multiple 3D points acquired from each acquisition plane of each of multiple Doppler ultrasound images. A current tangent to the vessel's longitudinal axis is calculated for each current Doppler image by performing a local principal component analysis (PCA). The measured blood flow is corrected based on a projection of the angle between the measured blood flow and the normal to each acquisition plane that lies along the tangent.

[0151] In the third approach, for each 3D voxel, multiple initial blood flow estimates and the normal corresponding to the plane of the ultrasound transducer capturing each Doppler US image used to calculate each initial blood flow estimate are recorded. The actual blood flow in each voxel is estimated by minimizing the projection error between the recorded blood flow velocity vector and the assumed flow velocity vector.

[0152] In the fourth approach, initial volumetric flow rates in a vessel in a sagittal view are acquired at multiple cardiac cycle specific locations. The initial volumetric flow rates may be calculated using standard approaches, such as Doppler, but at angles other than 90 degrees. An initial average volumetric flow rate is calculated by averaging the initial volumetric flow rates over the cardiac cycle. A gamma value, which indicates a real number in the range of -1 to 1, is calculated to correct the initial average volumetric flow rate to obtain the actual volumetric flow rate. An estimated actual blood flow rate can be obtained by correcting the initial measured blood flow rate, for example, based on the first, second, and / or third approaches described above. A cosine angle can be calculated from the inverse of the gamma value. The initial average volumetric flow rate is corrected and the actual volumetric flow rate is obtained using the cosine angle and dot product between the estimated blood flow rate and the actual blood flow rate.

[0153] In a fifth approach, also depicted with respect to FIG. 20, a 3D model of the actual blood vessel is calculated based on B-mode segmentation and a computational fluid dynamics model is used along with an aggregation of multiple 3D points acquired from each acquisition plane of each of multiple Doppler ultrasound images as boundary conditions to generate a total blood flow velocity vector field at each virtual point within the artery.

[0154] 7, a schematic diagram is provided depicting compensation for flow, e.g., velocity, speed, and blood 702, within an ultrasound image of a blood vessel 704 acquired at multiple orientations 708 by an ultrasound transducer 706. A graph 710 depicts measurements of blood flow 704 acquired at different orientations 708 of the transducer 706, compensated as described herein.

[0155] 2, at 214, a 3D image is reconstructed in a world coordinate system by aggregating a plurality of 3D voxels, each of which includes an estimated or corrected blood flow rate. The 3D image depicts an anatomical image of a blood vessel and depicts blood flow rate at a plurality of locations within the vessel.

[0156] Optionally, a 3D image is reconstructed from the 3D voxels calculated from the B-mode ultrasound image, and an indication of estimated blood flow associated with each 3D voxel is calculated from the Doppler data.

[0157] Optionally, the 3D voxels of the reconstructed 3D image are visually coded, e.g., color coded, according to the corresponding blood flow rate. For example, blood flow rates within a range indicating slow flow are colored one color, such as blue, blood flow rates within another range indicating fast flow are colored another color, such as red, and so on. Alternatively, or in addition, pixels of the Doppler ultrasound image indicative of blood flow rate are visually (e.g., color) coded. The color pixels may be segmented. A 3D image may be reconstructed from the 3D voxels corresponding to the segmented color pixels.

[0158] Optionally, the estimated blood flow rate of each of the 3D voxels is selected as a maximum value over an imaging time interval during which a Doppler ultrasound image depicting an area within the blood vessel corresponding to the 3D voxel is captured. Alternatively, or in addition, multiple Doppler ultrasound images used to calculate the 3D voxels depicting the variation in blood flow rate over the imaging time interval can be captured and plotted as a link graph, with the reconstructed 3D image including, for the 3D voxels, respective indications of the variation in blood flow rate over the imaging time interval. The reconstructed 3D image may be presented as a video over the imaging time interval by varying the blood flow rate values ​​corresponding to the 3D voxels over the imaging time interval according to the captured correlations.

[0159] Below is example pseudocode for generating a 3D image depicting blood flow velocity at multiple locations in a blood vessel: 3D reconstruction of the patient's body: USInit+transucerCamInit Record synchronized videos For every frame in transducer camera if frame is good then add to database For every frame in database featureExtraction ImageMatching FeatureMatching StructureFromMotion PrepareDenseScence ComputeDepthMapFilter Meshing MeshingFiltering Texturing end From StructureFromMotion extract Camera Poses For each camera pose in poses extractSegmentedPixels based on color and maximum value computeHomogenousTransform from the transducer corrdinate system to world system end For each colored point in PointCloud CorrectVelocityMeasurement End Display 4D data

[0160] Referring back now to FIG. 8 , a schematic diagram is provided depicting a reconstruction of a 3D image 802 of an artery 804 depicting a vascular pathology 806 (e.g., a stenosis) and showing the velocities of blood flow 812 and 814 through the artery 804. An ultrasound probe 808, including an ultrasound transducer and a camera as described herein, is manipulated over the surface of the leg above the artery 804, including the lesion 806, using, for example, a scanning motion in different orientations. The blood flow measured at different locations in different orientations is corrected as described herein. A number of standard ultrasound images 810, including optional Doppler data, are captured along with a camera image of a fiducial object in a pattern, for example, ultrasound gel, that has been applied over the surface of the leg. A 3D reconstruction 802 of the artery 804 is calculated from the ultrasound images, the camera images, and the optional Doppler data. The 3D reconstruction 802 depicts portions 812 that correspond to vascular pathology 806, identified by abnormally high blood flow velocities and / or abnormal indications, e.g., stenosis, calcification, high blood velocity, stents, and depicts other portions 814 that are normal portions of the artery, identified by normal blood velocities and / or normal geometry.

[0161] Referring now back to FIG. 2, a 3D image is reconstructed from the 3D voxels.

[0162] At 218, for each 3D voxel of the 3D image, a multi-dimensional data set is calculated. Each multi-dimensional data set includes a mapping between respective poses of the ultrasound transducer during capture of each ultrasound image depicting 3D coordinates within the body corresponding to the 3D voxel, and intensity values ​​acquired at the 3D coordinates for each ultrasound image acquired from the various transducer poses. The poses of the ultrasound transducer can be represented, for example, by six values ​​indicating six degrees of freedom. For example, for a 3D voxel with coordinates (x,y,z): (a1,b1,c1,d1,e1,f1) → intensity Q1, (a2,b2,c2,d2,e2,f2) → intensity Q2, and (a3,b3,c3,d3,e3,f3) → intensity Q3.

[0163] 9, a schematic diagram depicts the process of collecting data for a particular multi-dimensional data set for a particular set of 3D coordinates corresponding to a particular voxel. Collecting different pixel intensity values ​​for the same tissue region 902 by varying the orientation of the ultrasound transducer, visually depicted as multiple ultrasound imaging planes 904, is provided.

[0164] Referring back now to FIG. 10, a schematic diagram is provided depicting a representation of a multi-dimensional (optionally sparse) data set 1002 of a particular voxel in which the distribution of acquired grayscale values ​​represents a sphere. Each location within the sphere indicates a 3D pose and / or orientation of the ultrasound transducer during capture of a respective 3D voxel corresponding to the multi-dimensional (optionally sparse) data set. Each intensity value at each location of the sphere indicates a pixel intensity value of a 2D pixel of the 2D ultrasound image corresponding to the 3D coordinate of the 3D voxel calculated from the 2D pixel at the 3D pose and / or orientation of the ultrasound transducer corresponding to each location, and / or estimation by interpolating between values ​​can be done by interpolation or by a voting scheme that draws additional information from K-nearest neighbors. For example, the intensity at location 1004 differs from the intensity at location 1006, e.g., depicted visually darker, indicating a difference in the distribution of 3D pose and / or orientation during capture of a particular 3D voxel, with the same tissue but a different grayscale "fingerprint" identification. Diagram 1008 represents multiple multi-dimensional sparse data sets in a real-world coordinate system, one marked 1010 for clarity, with each multi-dimensional data set corresponding to one voxel used to create a 3D image. The multi-dimensional data sets may be clustered or segmented as described herein.

[0165] Referring now back to FIG. 2, at 220, the 3D voxels are clustered into clusters according to the distribution of the multi-dimensional sparse data set of 3D voxels.

[0166] Clustering may be performed based on manifold learning to find subspaces for better clustering the data.

[0167] Clustering may be performed by clustering voxels according to similar patterns of change in the captured ultrasound data for different angles and / or distances of a location corresponding to each voxel relative to the ultrasound transducer, with the voxels in each cluster indicating respective tissue types that produce respective similar patterns of change in the captured ultrasound data for different angles and / or distances relative to the ultrasound transducer.

[0168] The clustering can be based on an unsupervised learning approach, for example, by minimizing the statistical distance between voxels within each cluster and maximizing the statistical distance between clusters. The statistical distance can be based on an encoding that indicates a pattern of variation in the captured ultrasound data with respect to different angles and / or distances of the location corresponding to each voxel relative to the ultrasound transducer.

[0169] 11, a schematic diagram is provided depicting the reconstruction of a 3D image 1102 based on segmentation of a cluster of multi-dimensional data sets including an artery 1104 segmented into different tissue types having vascular pathology 1106 (e.g., stenosis, calcification, stent). An ultrasound probe 1108, including an ultrasound transducer and a camera as described herein, is repeatedly steered at multiple different orientations to the skin above the lesion 1106. Data collected for the region including the lesion 1106, including standard ultrasound images, optional Doppler data, and camera images of a pattern (e.g., a fiducial object in ultrasound gel spread across the surface of a leg), as described herein, is aggregated to generate a higher resolution 3D reconstruction 1112 of the reconstructed image 1102 and / or a higher resolution ultrasound image 1110. The ultrasound transducer pose and corresponding pixel intensities are included in each multi-dimensional data set, each for a 3D coordinate corresponding to a single voxel used to generate the 3D image. The higher resolution ultrasound image 1110 may be segmented by clustering voxels according to a similar multidimensional sparse data set, i.e., according to a similar distribution of pixel intensity values ​​of the ultrasound image between different orientations of the ultrasound transducer. Each segmented region may correspond to a respective tissue type. The 3D reconstruction 1102 may depict a higher resolution portion 1112, segmented to include vascular pathology 1106, depicting abnormal blood velocities and / or abnormal shapes (e.g., stenosis, calcification, high blood velocity, stents).

[0170] 2, at 222, the 3D image is segmented according to a subset of 3D voxel members of each cluster, with each segmentation including 3D voxels that are members of a respective cluster, and each segmentation may represent a particular tissue type.

[0171] The resulting clusters are segmented regions. Manifolds and clustering can be designed to find clusters of voxels, optionally in an unsupervised manner, taking into account the spatial and / or temporal distribution of each voxel, assigning each group as a respective tissue type. This process can improve the resolution of the ultrasound machine and / or extract the geometry and / or features of the relevant problem.

[0172] Below is an example pseudocode for computing the segmentation: For each pixel in the image Compute Pixel Location In Point Cloud Store Gray Scale Value for each voxel and colored Doppler with the transducer orientation and position as a 3D vector end Perform Voxelization

[0173] For voxel grouping based on manifold learning and / or other clustering where voxels with similar greyscale values ​​belong to the same group rather than to other groups, this involves computing group assignments based on distances between members, which may be calculated based on some computed manifold or latent representation, density thresholds, or setting up an a priori number of intended clusters, and / or may be formulated as a multi-objective optimization problem. For clustering, various approaches may be employed, such as connectivity-based clustering, centroid-based clustering, distribution-based clustering, density-based clustering, grid-based clustering, etc.

[0174] At 224, 3D coordinates are calculated for pixels of the 2D camera images that depict a surface (optionally skin) of the body segment. The entire 2D camera image may depict a surface. A surface may be segmented if some of the camera images depict the surface and some do not, e.g., depicting the background around the body part. The 3D coordinates are calculated based on an analysis of relative changes in the position of the fiducial object in successive 2D camera images, an analysis that is optionally performed to calculate the 3D coordinates of the ultrasound image.

[0175] The 3D coordinates assigned to the pixels of the 2D camera image and the pixels of the 2D ultrasound image are in a common coordinate system. Voxels are defined in the 3D coordinate system by assigning 3D coordinates to the pixels of the 2D camera image and the 2D ultrasound image.

[0176] Optionally, the 2D camera images are processed to remove fiducial objects located on the surface of the body segments depicted therein that were used to calculate the 3D coordinates. Removal of the fiducial objects can be performed after the calculation of the 3D coordinates, before and / or after assigning the 3D coordinates to pixels and calculating voxels.

[0177] At 226, a 3D image is reconstructed by aggregating voxels generated from pixels of the camera image and voxels generated from pixels of the ultrasound image in a common coordinate system. The 3D image depicts a surface of the body segment and tissue within the body segment disposed relative to the surface.

[0178] The common coordinate system may be a real-world coordinate system that describes real-world distances and positions. The 3D image describes the surface of the body segment and tissue within the body segment using real-world coordinates, distances, and relative positions. For example, a user may use the reconstructed 3D image to measure the depth of a blood vessel below a skin area. In another example, when using the 3D image to guide a catheter into deep tissue, a user may use the reconstructed 3D image to navigate by moving the catheter in a direction and / or distance to reach the target tissue as indicated in the real-world coordinate system of the reconstructed 3D image.

[0179] At 228, one or more of the blood flow and / or segmented 3D reconstructed images depicting the body surface are provided, e.g., presented on a display, stored in a storage device, and / or transferred to another remote device.

[0180] A single 3D image may be presented that represents the blood flow, the body surface, and is segmented. Alternatively, or in addition, a baseline 3D image may be presented, and the user may select one or more of the blood flow, the surface, and the segmentation to be presented as additional data, e.g., as an overlay, and / or integrated within the baseline 3D image. Alternatively, or in addition, multiple images may be presented, e.g., side-by-side on a display, each depicting different data. The multiple images may be selected from a representation of blood flow, a representation of the surface, and a representation of the segmentation.

[0181] Referring now back to FIG. 12, FIG. 12 is a schematic diagram depicting an exemplary 3D image 1202 of the brachial artery calculated based on a camera image depicting a pattern on the surface of a subject's arm and an ultrasound image of the brachial artery, and an exemplary 3D image 1204 depicting the surface of the patient's body.

[0182] 13, a schematic diagram is provided depicting an environmental scene of the generation of a 3D image 1302 computed from camera and ultrasound images, with fiducial objects 1304 randomly distributed and ultrasound gel located on the skin over an artery 1306 of a subject's leg 1308 for which the 3D image 1302 is being generated. An ultrasound probe 1310 includes an ultrasound transducer 1312 that captures ultrasound and / or Doppler images 1318 of the artery 1306, e.g., in B-mode, Doppler mode, and a camera 1314 connected to the probe 1310, optionally via an add-on component 1316 as described herein, that captures camera images of the fiducial objects 1304 randomly distributed within the gel. The ultrasound and / or Doppler images 1318 may be captured by a standard probe 1310 connected to a standard ultrasound workstation 1320, operated using standard ultrasound approaches as described herein and / or operated by a non-skilled user. A surface image 1322 of the surface of the leg 1308 may be extracted from the camera image as described herein. From data 1324 including 3D coordinates calculated for the ultrasound image using the camera image, optional Doppler data assigned to the 3D coordinates, and optional surface image 1322, a 3D image 1302 is calculated as described herein. The 3D image 1302 depicts a portion of the artery 1306 larger than any single 2D artery, such as a stretch of the artery, the entire artery, etc., with optional indications of blood flow within multiple locations of the artery 1306. The 3D image 1302 may include a 3D reconstruction of the surface image 1322 for the artery 1306 to enable visualization of the location of the artery 1306 below the skin surface within the leg 1308.

[0183] Referring now back to FIG. 2, at 230, the reconstructed 3D image may be used for other post-processing.

[0184] Optionally, the reconstructed 3D image is registered with another 3D anatomical image depicting the body segment. The 3D anatomical image is captured by a 3D imaging modality device, such as CT and MRI. The registration may be performed at least according to features depicted in the reconstructed 3D image and extracted from the surface of the body segment depicted in the 3D anatomical image. For example, registration may be performed between skin surfaces and also between internal blood vessels and / or other tissues.

[0185] Alternatively, or in addition, the reconstructed image is fed as input to another processing application, such as a surgical planning application, a neural network for diagnosing vascular pathologies.

[0186] At 232, one or more of the features described with reference to 204-230 may be repeated. The repetition may be performed by capturing additional ultrasound and camera images, collecting additional pixel and 3D coordinate data, and updating the reconstructed 3D image with the additional data to increase the resolution of the updated reconstructed 3D image. The image resolution may increase as the user moves the ultrasound probe to capture additional images.

[0187] Optionally, at 234, an add-on device is provided that is used to capture ultrasound images and camera images as described herein.

[0188] 14, a schematic diagram is provided depicting an ultrasound probe 1404 for capturing camera images by a camera 1406, and an add-on 1402 to the ultrasound transducer 1406 for capturing ultrasound images used to reconstruct a 3D image. The add-on 1402 is positioned to provide a fixed angle 1410, for example 7.2 degrees, or other value, between an axis 1412 of the ultrasound transducer 1408 for capturing ultrasound images and / or corresponding to the axis of the probe 1404, and an axis 1414 of the camera 1406 for capturing camera images. Based on the camera images captured by the camera 1406 according to the known angle 1410, a 3D pose of the ultrasound transducer 1406 may be calculated. 3D voxels corresponding to pixels of the ultrasound images used to reconstruct the 3D image may be calculated based on the calculated pose of the ultrasound transducer 1406 based on the known angle 1410, as described herein.

[0189] The add-on 1402 includes a connector component 1416 sized and shaped to fixedly connect to the probe 1404, e.g., secured by friction, e.g., the connector component 1416 includes an aperture sized to secure relative to the outer circumference of the probe 1404 so that when the aperture slides over the probe 1404, the probe 1404 is secured within the aperture by screws, adhesives, formed as an integral part of the probe 1404 by injection molding, etc., and / or by clips.

[0190] A camera housing 1418 is connected to the connector 1416. The camera housing 1418 is oriented with an axis 1412 corresponding to the axis of the probe 1404, at a predetermined angle 1410 relative to the connector component 1412. A camera 1406 is disposed within the camera housing 1418. The camera 1406 captures a camera image at a predetermined angle 1410 relative to an ultrasound image captured by the ultrasound transducer 1408.

[0191] Optionally, the add-on 1402 includes an aperture sized and shaped for the cable 1420 of the camera 1406 and / or the cable 1422 of the probe 1404. The cable can be connected to a computer for transmitting the camera images and / or ultrasound images it captures. The cable may also be for connecting to a power source. Alternatively, or in addition, an aperture is not necessarily required if the camera 1406 and / or the probe 1404 transmit images using a wireless interface and / or use a battery.

[0192] Optionally, the add-on 1402 includes a light source, for example located within the camera housing 1418 and / or within the camera 1406 and / or as a separate component. The light source may be configured to emit light at a selected frequency in response to a targeted fiducial object suspended within the ultrasound gel, for example, ultraviolet light and / or a selected frequency to enhance a fiducial object made of a fluorescent material and / or a material that enhances in response to UV light, as described herein. The light source may be placed at a preselected angle 1410 corresponding to the angle 1410 of the camera 1406 to transmit light to a portion of the ultrasound gel at the surface of the body depicted in a camera image captured by the camera 1406.

[0193] Referring now back to Figure 15, Figure 15 is a schematic diagram depicting an exemplary add-on 1502 to an ultrasound probe including a connector component 1516 sized and shaped to connect to an ultrasound probe and a camera housing 1518 oriented at an angle relative to the long axis of the connector component 1516 which corresponds to the long axis of the probe's ultrasound transducer. Schematic diagram 1550 is a left front view of the add-on 1502. Schematic diagram 1552 is a right rear view of the add-on 1502.

[0194] The connector component 1516 includes a channel 1554 designed to slide over the handle of a probe. A number of screw apertures 1556 are designed to receive screws to secure the add-on 1502 in place relative to the probe. It is noted that other fastening mechanisms may be used, as described herein.

[0195] The camera housing 1518 includes a view aperture 1558 through which a lens of a camera located within the camera housing 1518 captures camera images. The camera housing 1518 may include a camera cable aperture 1560 sized and shaped to position a cable connected to the camera. The cable may be connected to a computer to receive the camera images. Alternatively, the cable may be for connecting to a power source. Note that if a wireless interface is used to transmit images and / or if a battery is used, the cable and / or aperture 1560 are not necessary.

[0196] Referring back now to FIG. 16, FIG. 16 is a schematic diagram of another exemplary implementation of an add-on 1602 to an ultrasound probe 1604. The add-on 1602 includes two camera housings 1618A-B, each including a camera 1606A-B, i.e., camera 1606A in housing 1618A and camera 1606B in housing 1618B. As described herein, the camera housings 1618A-B are connected to a connector component 1616 that fixedly connects the add-on 1602 to the ultrasound probe 1604. When the camera housings 1618A-B are each positioned at a predetermined angle relative to a long axis of the connector component, they secure the cameras 1606A-B at a predetermined angle relative to the ultrasound probe 1604 and / or to an ultrasound image captured by an ultrasound transducer 1608 of the probe 1604, as described herein. The predetermined angle of both cameras 1606A-B may be the same, or each camera 1606A-B may be placed at a different predetermined angle.

[0197] Optionally, the camera housings 1618A-B and corresponding cameras 1606A-B are separated by about 90 degrees perpendicular to the long axis of the connector component 1616 or other value. That is, the radial distance relative to the long axis of the connector component 1616 is about 90 degrees. The 90 degree angle may be selected to capture camera images of the same area of ​​the subject's body surface during sagittal or longitudinal orientation of the ultrasound transducer 1608. Other angles may be used, in which case the known angle is used to calculate the relationship between the two images. For example, in a longitudinal examination, the long axis of the ultrasound transducer 1608 is placed parallel to the long axis of the tissue, for example along the length of a blood vessel, and the camera 1606B is positioned to capture camera images of fiducial objects randomly distributed in a viscous material (e.g., ultrasound gel) located on the body surface above the blood vessel. In a sagittal scan, the long axis of the ultrasound transducer 1608 is placed at an angle to the long axis of the tissue, for example along the diameter of a blood vessel, and the camera 1606A is positioned to capture camera images of the same fiducial objects randomly distributed in a viscous material (e.g., ultrasound gel) present at the same location on the body surface as captured by the camera 1606B during a longitudinal scan. Capturing fiducial objects randomly distributed in a viscous material during different orientations of the ultrasound transducer 1608 allows the reconstruction of a 3D image of the tissue (e.g., blood vessels). These fiducial objects may add additional information to calculate the actual scale of the transducer's structure and motion in world units. Optionally, add-on 1602 includes an inertial measurement unit (IMU) which can be used as an alternative or in addition to analyzing the relative change in position of the fiducial object in subsequent camera images to calculate the pose of the ultrasound transducer and / or the pose of the camera, adding additional information and allowing the actual scale of the transducer structure and motion in world units to be calculated.

[0198] Referring back now to FIGS. 21A-B, schematic diagrams of another exemplary implementation of an add-on 2102 to an ultrasound probe 2104 are depicted. FIG. 21A depicts a front view of the add-on 2102. FIG. 21B depicts a rear view of the add-on 2102. The add-on 2102 includes two camera housings 2118A-B, which include cameras 2106A-B, respectively, i.e., camera 2106A in housing 2118A and camera 2106B in housing 2118B. The camera housings 2118A-B are connected to an ergonomic holder component 2116 that fixedly connects the add-on 2102 to the ultrasound probe 2104, as described herein. For example, the ultrasound probe 2104 slides within a compartment formed by the add-on 2102, which has an internal size and / or shape that corresponds to the external size and / or shape of the ultrasound probe 2104. The ultrasound probe 2104 can be held in place, for example, by clicking into place with a clip that applies pressure, by friction from the interior walls of the compartment, by a strap, etc. The ergonomic holder 2116 can include a grip 2150 designed to be held against the palm of a user's hand, and a ring or trigger-like element 2152 designed to support the user's index finger. The ergonomic holder 2116 is designed to provide the user with an improved grip, allowing for greater control and / or finer movements during an ultrasound scan.

[0199] The camera housings 2118A-B, when each positioned at a predetermined angle relative to the long axis of the connector component as described herein, secure the cameras 2106A-B at a predetermined angle relative to the ultrasound probe 2104 and / or relative to the ultrasound image captured by the ultrasound transducer 2108 of the probe 2104. The predetermined angle of both cameras 2106A-B may be the same, or each camera 2106A-B may be positioned at a different predetermined angle. The camera 2106B includes a detachable lens 2120 and can be separated from the camera 2106B itself, making it easy for a user to use different lenses for particular types of body scans.

[0200] Optionally, the camera housings 2118A-B and corresponding cameras 2106A-B are separated by about 90 degrees or other value perpendicular to the long axis of the connector component 2116. That is, the radial distance relative to the long axis of the connector component 2116 is about 90 degrees. The 90 degree angle may be selected to capture camera images of the same area of ​​the subject's body surface during sagittal or longitudinal orientation of the ultrasound transducer 2108. Other angles (e.g., about 70 degrees or 80 degrees or 100 degrees or 110 degrees or other values) may be used, in which case the known angle is used to calculate the relationship between the two images. For example, in a longitudinal examination, the long axis of the ultrasound transducer 2108 is placed parallel to the long axis of the tissue, e.g., along the length of a blood vessel, and the camera 2106B is positioned to capture camera images of fiducial objects randomly distributed in a viscous material (e.g., ultrasound gel) located on the body surface above the blood vessel. In a sagittal scan, the long axis of the ultrasound transducer 2108 is placed at an angle to the long axis of the tissue, for example along the diameter of a blood vessel, and the camera 2106A is positioned to capture camera images of the same fiducial objects randomly distributed in a viscous material (e.g., ultrasound gel) present at the same location on the body surface as captured by the camera 2106B during a longitudinal scan. Capturing the fiducial objects randomly distributed in the viscous material during different orientations of the ultrasound transducer 2108 allows the reconstruction of a 3D image of the tissue (e.g., blood vessels). These fiducial objects may add additional information to calculate the actual scale of the transducer's structure and motion in world units. Optionally, the add-on 2102 includes an inertial measurement unit (IMU) (not shown as it may be an internal component) which can be used as an alternative or in addition to analyzing the relative change in position of the fiducial object in subsequent camera images to calculate the pose of the ultrasound transducer and / or the pose of the camera, adding additional information so that the actual scale of the transducer structure and motion in world units can be calculated.

[0201] Optionally, at 236, the add-on device is calibrated. Calibration may be performed to calculate one or more calibration transformations that map pixels of the ultrasound coordinate system to / from the camera coordinate system. The calibration mapping may map pixels of the ultrasound image and / or Doppler data and / or B-mode data expressed in the ultrasound coordinate system to the camera coordinate system, or vice versa. The mapping is based on a calibration relationship between the pose of the camera and the pose of the ultrasound transducer. The calibration mapping is used to map pixels of the Doppler and / or B-mode ultrasound image expressed in the camera coordinate system to 3D coordinates in the camera coordinate system.

[0202] Calibration may be performed using a specially designed device that includes a box with ultrasound gel, randomly spaced strings, and a nearby checkerboard pattern, or other patterns may be used. When an ultrasound transducer scans a silicon box, two images appear simultaneously: an ultrasound image of the cross section of the string inside the box, and a camera image of the checkerboard. The two images are related to each other, and the camera pose from the checkerboard corresponds to a different pose of the transducer from a known string immersed in silicone. The relationship between the two images is determined by a calibration process that collects several images from different checkerboard positions, and the corresponding US images are analyzed and compared to the ground truth, i.e., angles and distances enforced by 3D computer modeling. The outcome of the process provides a homogeneous calibration transformation to convert the camera image pose to the transducer pose.

[0203] Calibration may be used based on random versions of strings with no defined geometry, and on the use of checkerboards. In another example, another calibration device is based on a simple box with ultrasound gel and holes and April tags. The camera pose is based on an estimation protocol that determines the camera pose for each detected April tag that can be detected in the captured image. At the same time, the transducer captures both the top and bottom of the box, which can be used to determine the homogeneous transformation between the camera and the transducer.

[0204] 17, a schematic diagram is provided depicting an example implementation of a calibration device 1752, e.g., corresponding to the calibration device 152 described with reference to Figure 1. The calibration device 1752 includes an ultrasound section 1760 having a predetermined first ground truth pattern 1762, and a camera section 164 having a predetermined second ground truth pattern 1766.

[0205] The ultrasonic section 1760 can be implemented as a box with an open top. The ground truth pattern 1762 can be a 3D pattern with predefined 3D coordinates immersed in an ultrasonically transparent medium such as transparent silicon. The ground truth pattern 1762 can be, for example, poles arranged at various angles, and / or different objects with different shapes, slopes with matching protrusions, and / or angles arranged at different heights and / or orientations, and / or in a random pattern, e.g., boxes, spheres, pyramids, rectangles. Alternatively or additionally, the ground truth pattern 1762 can be a 2D pattern, e.g., a checkerboard pattern and / or a wire pattern. The ground truth pattern 1766 can be 2D and / or 3D, e.g., a checkerboard pattern. The shapes of the patterns 1762 and 1766 are predefined, e.g., the pattern 1762 is printed using a 3D printer and the checkerboard pattern 1766 is printed using a 2D printer.

[0206] The camera section 1762 may be positioned proximate to the ultrasound section 1760 and may optionally be separated by a transmission positioning region 1768. The arrangement of the camera section 1762, ultrasound section 1760, and transducer positioning region 1768 is selected to simultaneously, or nearly simultaneously, capture an ultrasound image depicting a predetermined first 3D ground truth pattern 1762 by an ultrasound transducer located in the transducer positioning region 1768 and a camera image depicting a predetermined second ground truth pattern 1766 by a camera located in an add-on to the ultrasound probe located on the probe connected to the ultrasound transducer located in the transducer positioning region 1768.

[0207] The following is an example pseudocode for calibration using the calibration device 1752: 1.Init Both ultrasound transducer and camera 2.Collect several checkerboard images and B-Mode images from different orientations and positions 3.Locate strings position in transducer image and estimate camera pose in world coordinate system relative the checkerboard pattern 4.Find homogenous transformation between the transducer coordinate system and the camera by minimizing the relative distance between the actual position of the recorded strings positions and the 3D known positions. 5.Return computed transformation

[0208] 18, a schematic diagram is provided depicting another exemplary implementation of another calibration device 1852, corresponding, for example, to the calibration device 152 described with reference to FIG. 1. The calibration device 1852 includes a single compartment 1854, which includes a predetermined common 2D marker and / or 3D truth pattern 1856 disposed on top of the single compartment 1854. The compartment 1854 may be shaped, for example, with a circular surface. The compartment 1854 may be shaped, for example, as a hemisphere, a cylinder, or other shape. The pattern 1856 may be disposed outside and / or around (e.g., near) the spherical boundary (e.g., perimeter) of the compartment 1854, for example, the indicators of the pattern 1856 are evenly spaced along the outer perimeter of the compartment 1854. The compartment 1854 includes (e.g., is filled with) an ultrasound-transparent echography medium, for example, a clear silicone or an echography gel. When an ultrasound transducer is placed on top of the compartment 1854 (optionally approximately in the center), the camera captures a camera image of the pattern 1856, and the ultrasound transducer captures an ultrasound image (e.g., image 1890 is an example) of the interior of the compartment 1854 from the top surface 1860 to the bottom surface 1862. Ultrasound may pass through the top surface 1860 without significant attenuation and may also pass through the hole 1864. The hole 1864 may be located, for example, on the top, on the bottom, and / or within (e.g., in the center) of the compartment 1854. The pattern 1856 may be made of a material with selected properties, such as enhanced under ultraviolet (UV) light and / or fluorescence, and may be made of a calibration pattern printed with high contrast.

[0209] The following is an example pseudocode for calibration using the calibration device 1852: 1.Init Both transducer camera and the camera 2.Collect simultaneously several camera images of the markers and the transducer images from different orientations and positions 3.For each transducer image locate floor (UV pixels) position in US image, and for each corresponding camera image locate each marker position in world units 4.For each detected tag and for each UV floor pixel compute the estimated floor depth based on homogeneous transformation from UV image to camera pose in the tag coordinates system 5.Find a homogeneous transformation that minimizes the relative distance between the actual known depth of the floor from the top surface and the estimated depth based on the above transformation (US image to world coordinates). 6.Return computed transform

[0210] Optionally, the IMU is calibrated using the IMU measured and camera recorded orientations simultaneously, with gravity being used as the Z component of the IMU coordinate system.

[0211] Referring now back to FIG. 3, at 302, a viscous material, which is a fluid (optionally an ultrasound gel), at least a portion of which is randomly distributed in its volume and contains fiducial objects spaced at random distances, is applied to a surface of an individual's body segment in locations corresponding to blood vessels of interest.

[0212] If the material is an ultrasound gel, the gel may be applied to a larger area of ​​skin and used in standard methods for obtaining ultrasound images.

[0213] At 304, an ultrasound transducer probe with add-on components including one or more cameras is manipulated along the body surface, optionally simultaneously capturing ultrasound images of blood vessels within the body and camera images depicting the surface of the body segment and depicting fiducial objects.

[0214] At 306, one or more 3D images are reconstructed from the ultrasound images and an analysis of the camera images is presented on a display.

[0215] Optionally, the 3D image depicts a visual indication of blood flow at multiple regions within the blood vessel. Alternatively, or in addition, the 3D image is segmented according to tissue type. Alternatively, or in addition, the 3D image depicts the skin and blood vessels of the body segment in a common real-world coordinate system representative of real-world coordinates, and diagnosing the vascular condition based on a reconstruction of the 3D image of the blood vessel relative to the surface of the body segment is further based on blood flow in the blood vessel depicted by the 3D image.

[0216] At 308, the 3D images are analyzed. This analysis may be performed manually by visual inspection and / or measurement, and / or may be performed automatically by feeding the 3D images to a machine learning process (e.g., a neural network) that generates an outcome that is indicative of a diagnosis.

[0217] At 310, a vascular condition is diagnosed based on analysis of the 3D images, e.g., stenosis of a blood vessel.

[0218] At 312, the vascular condition is treated during open surgery and / or catheter procedures. Introduction, navigation and / or treatment may be guided by the 3D image, for example, according to distances and / or relative positions described by a real-world coordinate system in which the skin and blood vessels of the 3D image are presented. Exemplary treatments include catheter-delivered stents, balloon dilation, ablation, drug injection, and procedural resection and / or repair.

[0219] At 314, the effectiveness of the vascular treatment may be evaluated based on the 3D images acquired after the treatment, for example, by analyzing a segmentation of the 3D images to determine whether plaque has been completely removed from the artery and / or to determine whether blood flow patterns have returned to normal.

[0220] At 316, one or more of the features described with reference to 302-316 are repeated.

[0221] Optionally, the repetition is performed, for example, by re-manipulating the probe on the surface of the body to simultaneously capture ultrasound images of the blood vessel depicting the treated vascular condition, analyzing the treated vascular condition with another 3D reconstruction of the blood vessel created from the ultrasound images and camera images captured during re-manipulation of the probe, and re-treating the treated vascular condition if the treated vascular condition is determined to require another treatment procedure based on the another 3D reconstruction.

[0222] Alternatively, or in addition, the repetition is performed by repeatedly steering the ultrasound transducer within a sub-region of a surface corresponding to the vascular pathology in an axial and / or longitudinal orientation of the ultrasound transducer to capture multiple ultrasound images of the vascular pathology at different angles and distances, where the resolution of the 3D image, optionally the resolution of the 3D image of the vascular pathology, increases with an increasing number of steering of the ultrasound transducer over the sub-region.

[0223] Alternatively, or in addition, the repetitions are based on the code analyzing the data to detect diseased segments (e.g., slow blood velocity, stents, high calcium content). In response to this detection, an indication (e.g., a beep, an audio message, a text message presented on a display, and / or a video) may be provided to the user to repeatedly scan the segment to increase the data and obtain a higher resolution image of the diseased segment. Optionally, the code may monitor the amount of data acquired to detect when enough data has been collected to generate a targeted high resolution 3D image, and provide another indication to the user to stop the ultrasound scan and / or move to the next body area to be scanned.

[0224] Referring back now to FIG. 19, there is provided a schematic diagram 1902 depicting a standard vascular treatment using existing approaches, as well as a schematic diagram 1904 depicting a vascular treatment using 3D images created from ultrasound and camera images. As depicted in schematic diagram 1902, the standard process is for treating a vascular condition in the leg, for example, where blood flow in the leg may be insufficient, and may include a standard Doppler on the leg. A CTA may be performed, which may require a 2-day hospital stay. An angiography procedure may be performed 1-2 weeks later to treat the vascular condition, which may require a 3-day hospital stay. The results of the procedure may be obtained with a 1-2 month follow-up. In contrast, schematic diagram 1904 depicts how real-time 3D images of the blood vessels in the leg may be created before, during, and after treatment, for example, to guide a catheter into the vessel, and for example, to evaluate the effectiveness of the vascular treatment, which may allow for a same-day and / or 2-day hospital stay.

[0225] FIG. 20 depicts a process for calculating an estimated blood flow velocity field 2004 by using a computational fluid dynamics simulation 2010 with an a priori 3D arterial model 2006 integrated as boundary conditions 2006 for blood flow measurements, which can be measured by the Doppler mode of an ultrasound system from various positions and, in the same coordinate system and same volume 2002, by corresponding positions in a corresponding known 3D arterial model 2006.

[0226] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0227] It is anticipated that many related viscous materials and fiducial objects will be developed during the life of the patent which matures from this application, and the scope of the terms viscous materials and fiducial objects is intended to include all such new technologies a priori.

[0228] As used herein, the term "about" means ±10%.

[0229] The terms "comprises," "comprising," "includes," "including," "having" and their cognates mean "including but not limited to." This term encompasses the terms "consisting of" and "consisting essentially of."

[0230] The phrase "consisting essentially of" means that the composition or method may include additional components and / or steps, but only if the additional components and / or steps do not materially alter the basic and novel characteristics of the claimed composition or method.

[0231] As used herein, the singular forms "a," "an," and "the" include plural referents unless the context clearly indicates otherwise. For example, the term "a compound" or "at least one compound" can include a plurality of compounds, including mixtures thereof.

[0232] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments and / or to exclude the incorporation of features from other embodiments.

[0233] As used herein, the word "optionally" is used to mean "provided in some embodiments and not provided in other embodiments." Any particular embodiment of the invention may include multiple "optional" features, unless such features are inconsistent.

[0234] Throughout this application, various embodiments of the invention may be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Thus, the description of a range should be considered to have specifically disclosed all possible subranges as well as individual numerical values ​​within that range. For example, description of a range such as "1-6" should be considered to have specifically disclosed subranges such as "1-3", "1-4", "1-5", "2-4", "2-6", "3-6", etc., and individual numerical values ​​within that range, e.g., 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

[0235] Whenever a numerical range is given herein, it is meant to include any recited numbers (fractional or integer) within the given range. The phrases "ranging / ranges between" a first designation number and a second designation number, and the phrases "ranging / ranges from" a first designation number "to" a second designation number, are used interchangeably herein and are meant to include the first designation number and the second designation number, and all fractional and integer numbers therebetween.

[0236] It is understood that certain features of the invention that are described for clarity in the context of separate embodiments can also be provided in combination in a single embodiment. Conversely, various features of the invention that are described for brevity in the context of a single embodiment can also be provided separately or in any suitable subcombination or as preferred in any other described embodiment of the invention. Particular features described in the context of various embodiments should not be considered essential features of those embodiments, unless the embodiment is inoperable without those elements.

[0237] While the present invention has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims.

[0238] It is the intention of the applicants (applicants) that all publications, patents, and patent applications mentioned herein are incorporated herein by reference in their entirety, as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated herein by reference. Furthermore, citation or identification of any reference in this application should not be construed as an admission that such reference is available as prior art to the present invention. Section headings, if used, should not be construed as necessarily limiting. Additionally, any priority documents of this application are hereby incorporated herein by reference in their entirety.

Claims

1. 1. A computer-implemented method for reconstructing a 3D image depicting blood flow within a blood vessel of an individual, comprising: acquiring a Doppler ultrasound image depicting a blood vessel with blood flowing therethrough and a measurement of blood flow within a region of the blood vessel, and a 2D camera image captured by a camera depicting a plurality of fiducial objects randomly distributed within a viscous material and spaced apart by random distances on a surface of the individual's body segment, the Doppler ultrasound image comprising at least one Doppler ultrasound image and the 2D camera image comprising at least one 2D camera image; calculating 3D coordinates in a world coordinate system for each pixel of the Doppler ultrasound image using an external reference of ultrasound transducer pose calculated by analysis of relative changes in positions of the fiducial objects in successive 2D camera images; calculating a respective estimated blood flow for each of a plurality of pixels of the Doppler ultrasound image at a plurality of locations within the blood vessel; reconstructing a 3D image including respective estimated blood flow volumes from 3D voxels calculated from the 3D coordinates of the pixels of the Doppler ultrasound images, the 3D image depicting an anatomical image of the blood vessel and depicting blood flow; The computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, wherein the 3D coordinates are further calculated by correcting them to a real scale with respect to the world coordinate system by using a size of the fiducial object and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera.

3. The computer-implemented method of claim 1 , wherein the 3D image further comprises voxels computed from a 3D point cloud or mesh depicting the surface of the body segment of the individual.

4. Calculating the 3D coordinates of each of the pixels of the Doppler ultrasound image comprises: calculating a pose of the camera in a camera coordinate system relative to the world coordinate system by analyzing relative changes in positions of the plurality of fiducial objects in the successive 2D camera images; applying a calibration mapping to map pixels of the ultrasound image represented in an ultrasound coordinate system to the camera coordinate system, the calibration mapping being based on a predetermined transformation between the pose of the camera and a pose of an ultrasound transducer; mapping the pixels of the ultrasound image expressed in the camera coordinate system to the 3D coordinates in the world coordinate system; The computer-implemented method of claim 1 , comprising:

5. 2. The computer-implemented method of claim 1, wherein calculating each estimated blood flow includes an estimated blood flow calculated from the Doppler ultrasound image captured by the ultrasound transducer positioned at an angle relative to a vector indicating a direction of the blood vessel for which the blood flow is measured, and further includes correcting the estimated blood flow for the angle to obtain an estimated actual blood flow.

6. 6. The computer-implemented method of claim 5, wherein the correcting comprises identifying an elliptical boundary of the blood vessel, calculating a transformation from the elliptical shape to a circle, and applying the transformation to project the measured blood flow velocity vector to obtain the actual blood flow.

7. 6. The computer-implemented method of claim 5, further comprising: calculating an estimate of a longitudinal axis of the blood vessel based on an aggregation of a plurality of 3D points acquired from respective acquisition planes of a plurality of Doppler ultrasound images; and calculating a current tangent to the longitudinal axis; and wherein the correcting comprises correcting the blood flow measurement based on an angle between the tangent and a normal to the respective acquisition plane indicated by the tangent.

8. 6. The computer-implemented method of claim 5, further comprising: calculating an estimate of blood flow within a pre-calculated 3D arterial object; and using the computational fluid dynamics simulation and 3D points acquired from each acquisition plane of the plurality of Doppler ultrasound images to calculate an estimate of blood flow within the 3D arterial object.

9. storing, for each 3D voxel, a number of initial estimated blood flow values ​​and corresponding normals to a plane of the ultrasound transducer at which each Doppler US image used to calculate the respective initial estimated blood flow; estimating actual blood flow at each voxel based on minimizing a projection error between the recorded blood flow velocity vector and the assumed flow velocity vector; The computer-implemented method of claim 5 , further comprising:

10. acquiring a volumetric flow rate in the blood vessel on a sagittal view at specific locations for a plurality of cardiac cycles and calculating an average volumetric flow rate over the plurality of cardiac cycles; calculating a gamma value indicating a real number within a range of −1 to 1 to correct the average volumetric flow rate and the estimated actual blood flow rate; calculating a cosine angle by using the inverse of the gamma value; correcting the mean volumetric flow using the cosine angle and a dot product between the estimated blood flow and the actual blood flow; The computer-implemented method of claim 9 , further comprising:

11. (i) alternating between acquiring the Doppler ultrasound image and the 2D camera image, and (ii) acquiring a second image comprising a B-mode ultrasound image and a 2D camera image; calculating 3D coordinates of each pixel of the B-mode ultrasound image in the world coordinate system; reconstructing the 3D image includes reconstructing the 3D image from 3D voxels in the world coordinate system calculated by aggregating the 3D coordinates of the pixels of the B-mode ultrasound image and the Doppler ultrasound image, each including an estimated blood flow; 2. The computer-implemented method of claim 1, wherein the 3D image depicts the anatomical image of the blood vessel created from the aggregation of the 3D voxels obtained from the pixels in the B-mode and depicts the blood flow associated with the 3D voxels obtained from the pixels in the B-mode.

12. 2. The computer-implemented method of claim 1, wherein each of the estimated blood flow rates is depicted as a color coding of the 3D voxels of the 3D image corresponding to the multiple locations within the blood vessel, and pixels of the Doppler ultrasound image are color coded to indicate blood flow rate, further comprising segmenting the color pixels, and the 3D image is reconstructed from 3D voxels corresponding to the segmented color pixels.

13. 2. The computer-implemented method of claim 1, wherein the estimated blood flow for each of the 3D voxels is selected as a maximum value over an imaging time interval during which the Doppler ultrasound image depicting a region within the blood vessel corresponding to the 3D voxel is captured.

14. 2. The computer-implemented method of claim 1, wherein the multiple Doppler ultrasound images used to calculate the 3D voxel are captured over an imaging time interval depicting variations in blood flow, and the reconstructed 3D image includes, for the 3D voxel, a respective indication of the variation in blood flow over the imaging time interval.

15. 13. The computer-implemented method of claim 12, wherein the reconstructed 3D image is presented as a video over an imaging time interval by varying the indication of blood flow corresponding to the 3D voxels over the imaging time interval.

16. 1. A computer-implemented method for reconstructing a surface and / or interior 3D image of a body segment of an individual, comprising: obtaining 2D ultrasound images depicting tissue within the body segment of the individual, and 2D camera images captured by a camera depicting a plurality of fiducial objects randomly distributed in 3D within a viscous material and spaced by random distances on a surface of the body segment and depicting the surface of the body segment; calculating, in a common coordinate system, 3D coordinates of each pixel of the 2D camera image and 3D coordinates of each pixel of the 2D ultrasound image based on an analysis of the relative changes in positions of the fiducial objects in the successive 2D camera images; reconstructing a 3D image from 3D voxels in the common coordinate system calculated by aggregating the 3D coordinates of the 2D camera image and the 2D ultrasound image; Including, the 3D image depicts the surface of the body segment and tissue within the body segment that is positioned relative to the surface of the patient; The computer-implemented method, wherein the reconstructed 3D images include at least one reconstructed 3D image.

17. 17. The computer-implemented method of claim 16, wherein the at least one reconstructed 3D image with additional information layers including at least one of Doppler and B-mode depicts tissues selected in any compatible ultrasound organ scanning procedure including at least one of blood vessels, organs, joints, bones, cartilage, non-blood filled cavities, liver, gallbladder, thyroid, and fuses the tissues into an entire organ or part of an organ.

18. 17. The computer-implemented method of claim 16, further comprising correcting the 3D coordinates to a real scale relative to the world coordinate system by using a size of the fiducial object and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera.

19. 17. The computer-implemented method of claim 16, wherein the common coordinate system is a real-world coordinate system that depicts real-world distances and positions, and the 3D images depict the surfaces of the body segments and tissue within the body segments using distances and relative positions in real-world coordinates.

20. Calculating in the common coordinate system calculating the pose of the camera in a camera coordinate system relative to the common coordinate system by analyzing the relative changes in the positions of the fiducial objects in the successive 2D camera images, and correcting to actual scale relative to the world coordinate system by using the size of the fiducial objects and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera; mapping pixels of the 2D camera images depicting the surface of the body to 3D coordinates within the common coordinate system; applying a calibration transformation to map pixels of the 2D ultrasound image represented in an ultrasound coordinate system to the camera coordinate system, the calibration mapping being based on a predetermined relationship between the pose of the camera and a pose of an ultrasound transducer that captures the 2D ultrasound image; mapping the pixels of the 2D ultrasound image expressed in the camera coordinate system to 3D coordinates in the common coordinate system; 20. The computer-implemented method of claim 16, comprising:

Images