System and method for reconstructing 3D images from ultrasound and camera images

Abstract

A method is provided for reconstructing a 3D image, the method including obtaining Doppler ultrasound images depicting blood vessels and measurements of blood flow within the vessels and 2D camera images captured by a camera depicting fiducial objects randomly distributed within a viscous material and spaced apart by random distances on the surface of an individual, calculating 3D coordinates in a world coordinate system for pixels of the Doppler images using an external reference of the pose of the ultrasound transducer calculated by analysis of relative changes in the positions of the fiducial objects in successive camera images, calculating respective estimated blood flow rates for pixels of the Doppler images at positions within the blood vessels, and reconstructing a 3D image from 3D voxels calculated from the 3D coordinates of the pixels of the Doppler images including respective estimated blood flow rates, the 3D image depicting the blood vessels and blood flow.

Description

[Technical Field]

[0001] Related Applications This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 174,064, filed April 13, 2021, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] The present invention, in some embodiments thereof, relates to the processing of medical ultrasound images, and more particularly to the reconstruction of 3D medical images from 2D ultrasound images.

[0003] Ultrasound (US) is often considered the safest and least expensive technique for clinical analysis and medical intervention. US images are always captured as 2D images. There are also approaches to generate 3D ultrasound images. Summary of the Invention

[0004] According to a first aspect, a computer-implemented method for reconstructing a 3D image depicting blood flow within a blood vessel of an individual includes acquiring Doppler ultrasound images depicting the blood vessel with blood flowing therethrough and depicting a measurement of blood flow within a region of the blood vessel, and 2D camera images captured by a camera depicting a plurality of fiducial objects randomly distributed within a viscous material and spaced apart by random distances on a surface of a body segment of the individual, the Doppler ultrasound images including at least one Doppler ultrasound image and the 2D camera images including at least one 2D camera image; and The method includes calculating 3D coordinates in a world coordinate system for each pixel of the Doppler ultrasound image using an external reference of ultrasound transducer pose calculated by analysis of relative changes in positions of a plurality of fiducial objects in the Doppler camera image; calculating a respective estimated blood flow rate for each of a plurality of pixels of the Doppler ultrasound image at a plurality of locations within the blood vessel; and reconstructing a 3D image from 3D voxels calculated from the 3D coordinates of the pixels of the Doppler ultrasound image, the 3D image including the respective estimated blood flow rates, wherein the 3D image depicts an anatomical image of the blood vessel and depicts the blood flow.

[0005] In a further implementation of the first aspect, the 3D coordinates are further calculated by correcting them to their actual scale relative to the world coordinate system by using the size of the fiducial object and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera.

[0006] In a further implementation of the first aspect, the 3D image further comprises voxels calculated from a 3D point cloud or mesh that describes the surface of a body segment of the individual.

[0007] In a further implementation of the first aspect, calculating the 3D coordinates of pixels of each Doppler ultrasound image includes calculating a camera pose in the camera coordinate system relative to a world coordinate system by analyzing relative changes in positions of multiple fiducial objects in successive 2D camera images, applying a calibration mapping to map pixels of the ultrasound image represented in the ultrasound coordinate system to the camera coordinate system, where the calibration mapping is based on a predetermined transformation between the camera pose and the ultrasound transducer pose, and mapping the pixels of the ultrasound image represented in the camera coordinate system to 3D coordinates in the world coordinate system.

[0008] In a further implementation of the first aspect, calculating each estimated blood flow includes calculating an estimated blood flow from a Doppler ultrasound image captured by an ultrasound transducer positioned at an angle relative to a vector indicating the direction of the blood vessel for which the blood flow is measured, and further correcting the estimated blood flow at this angle to obtain the estimated actual blood flow.

[0009] In a further implementation of the first aspect, the correcting includes identifying an elliptical boundary of the blood vessel, calculating a transformation from ellipse to circle, and applying this transformation to project the measured blood flow velocity vector to obtain the actual blood flow rate.

[0010] In a further implementation of the first aspect, the method further includes calculating an estimate of the longitudinal axis of the blood vessel based on an aggregation of multiple 3D points acquired from each acquisition plane of each of the multiple Doppler ultrasound images, and calculating a current tangent to the longitudinal axis, and the correcting includes correcting the blood flow measurement based on the angle between the tangent and a normal to each acquisition plane indicated by the tangent.

[0011] A further implementation of the first aspect further includes calculating an estimate of blood flow within the pre-calculated 3D arterial object, and using the computational fluid dynamics simulation and 3D points acquired from each acquisition plane of each of the plurality of Doppler ultrasound images to calculate the estimate of blood flow within the 3D arterial object.

[0012] A further implementation of the first aspect further includes storing, for each 3D voxel, multiple initial estimated blood flow values ​​and corresponding normals to the plane of the ultrasound transducer from which each Doppler US image was captured that were used to calculate each initial estimated blood flow, and estimating the actual blood flow at each voxel based on minimizing the projection error between the recorded blood flow velocity vector and the assumed flow velocity vector.

[0013] A further implementation of the first aspect further includes obtaining volumetric flow rates within a blood vessel on a sagittal view at positions specific to multiple cardiac cycles to calculate an average volumetric flow rate over the multiple cardiac cycles, calculating a gamma value representing a real number in the range of -1 to 1 to correct the average volumetric flow rate and the estimated actual blood flow rate, calculating a cosine angle by using the inverse of the gamma value, and correcting the average volumetric flow rate using the cosine angle and the dot product between the estimated blood flow rate and the actual blood flow rate.

[0014] A further implementation of the first aspect further includes alternating between (i) acquiring a Doppler ultrasound image and a 2D camera image and (ii) acquiring a second image including a B-mode ultrasound image and a 2D camera image, and calculating 3D coordinates of pixels in each B-mode ultrasound image in a world coordinate system, and reconstructing a 3D image in the world coordinate system from 3D voxels calculated by aggregating the 3D coordinates of the pixels of the B-mode ultrasound image and the Doppler ultrasound image, the 3D image including an estimated blood flow rate, wherein the 3D image depicts an anatomical image of blood vessels created from the aggregation of 3D voxels acquired from the B-mode pixels and depicts blood flow associated with the 3D voxels acquired from the B-mode pixels.

[0015] In a further implementation of the first aspect, each estimated blood flow is depicted as a color coding of 3D voxels of a 3D image corresponding to multiple locations within the blood vessel, and pixels of the Doppler ultrasound image are color coded to indicate the blood flow, further comprising segmenting the color pixels, and the 3D image is reconstructed from the 3D voxels corresponding to the segmented color pixels.

[0016] In a further implementation of the first aspect, the estimated blood flow for each 3D voxel is selected as the maximum value over an imaging time interval, and a Doppler ultrasound image depicting the region within the blood vessel corresponding to the 3D voxel is captured during this imaging time interval.

[0017] In a further implementation of the first aspect, the multiple Doppler ultrasound images used to calculate the 3D voxels are captured over an imaging time interval depicting variations in blood flow, and the reconstructed 3D image includes, for the 3D voxels, respective indications of the variations in blood flow over the imaging time interval.

[0018] In a further implementation of the first aspect, the reconstructed 3D image is presented as a video over the imaging time interval by varying an indication of blood flow corresponding to the 3D voxels over the imaging time interval.

[0019] According to a second aspect, a computer-implemented method for segmenting a 3D image reconstructed from 2D ultrasound images depicting a body segment of an individual includes obtaining 2D ultrasound images depicting a common region of the body segment and 2D camera images captured by a camera depicting a plurality of fiducial objects randomly distributed in 3D within a viscous material and spaced apart by random distances on the surface of the body segment, and segmenting each 2D image using an external reference for ultrasound transducer pose calculated by analysis of relative changes in the positions of the plurality of fiducial objects in successive 2D camera images. The method includes calculating a plurality of 3D voxels having 3D coordinates in a world coordinate system assigned to pixels of the ultrasound image; reconstructing a 3D image from the plurality of 3D voxels; storing, for each 3D voxel, a mapping of a multidimensional sparse data set between a posture of the ultrasound transducer and intensity values ​​acquired at the 3D coordinates corresponding to each 3D voxel and each posture of the ultrasound transducer; clustering the plurality of 3D voxels into a plurality of clusters according to a distribution of the multidimensional data set of the plurality of 3D voxels; and segmenting the 3D image according to the plurality of clusters.

[0020] A further implementation of the second aspect further includes correcting the 3D voxels to their actual scale relative to the world coordinate system by using the size of the fiducial object and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera.

[0021] In a further implementation of the second aspect, calculating the 3D coordinates of the pixels of each 2D ultrasound image includes calculating the pose of the camera in the camera coordinate system relative to the world coordinate system by analyzing relative changes in the positions of multiple fiducial objects in consecutive 2D cameras and correcting to actual scale relative to the world coordinate system by using the size of the fiducial objects and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera; applying a calibration transformation to map the pixels of the 2D ultrasound image represented in the ultrasound coordinate system to the camera coordinate system, where the calibration mapping is based on a predetermined relationship between the pose of the camera and the pose of an ultrasound transducer capturing the 2D ultrasound image; and mapping the pixels of the 2D ultrasound image represented in the camera coordinate system to 3D coordinates in the world coordinate system.

[0022] In a further implementation of the second aspect, the clustering is performed based on a manifold learning technique to find similar voxels within a region with respect to the distribution of some calculated value, such as color, intensity, etc., surrounding the texture.

[0023] In a further implementation of the second aspect, the clustering is performed by clustering a plurality of 3D voxels according to similar patterns of variation in the ultrasound data captured with respect to different angles and / or distances of the position corresponding to each voxel relative to the ultrasound transducer.

[0024] In a further implementation of the second aspect, the 3D voxels in each cluster indicate respective tissue types, creating similar patterns of variation in the ultrasound data captured at different angles and / or distances relative to the ultrasound transducer.

[0025] According to a third aspect, a computer-implemented method for reconstructing 3D images of the surface and interior of a body segment of an individual includes obtaining 2D ultrasound images depicting tissue within the body segment of the individual and 2D camera images captured by a camera depicting a plurality of fiducial objects randomly distributed in 3D within a viscous material and spaced by random distances on the surface of the body segment and depicting the surface of the body segment; calculating 3D coordinates of pixels of each 2D camera image and 3D coordinates of pixels of each 2D ultrasound image in a common coordinate system based on an analysis of relative changes in positions of the plurality of fiducial objects in successive 2D camera images; and reconstructing 3D images from 3D voxels in the common coordinate system calculated by aggregating the 3D coordinates of the 2D camera images and the 2D ultrasound images, wherein the 3D images depict the surface of the body segment and tissue within the body segment positioned relative to the surface of the patient, and the reconstructed 3D images include at least one reconstructed 3D image.

[0026] In a further implementation of the third aspect, at least one reconstructed 3D image with additional information layers including at least one of Doppler and B-mode depicts tissues selected in any compatible ultrasound organ scanning procedure, including at least one of blood vessels, organs, joints, bones, cartilage, non-blood-filled cavities, liver, gallbladder, and thyroid, and fuses these tissues into a single whole organ or portion of an organ.

[0027] A further implementation of the third aspect further includes correcting the 3D coordinates to their actual scale relative to the world coordinate system by using the size of the fiducial object and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera.

[0028] In a further implementation of the third aspect, the common coordinate system is a real-world coordinate system that depicts real-world distances and positions, and the 3D image depicts the surface of the body segment and tissue within the body segment using distances and relative positions in real-world coordinates.

[0029] In a further implementation of the third aspect, calculating in the common coordinate system includes calculating the pose of the camera in the camera coordinate system relative to the common coordinate system by analyzing relative changes in the positions of multiple fiducial objects in successive 2D camera images and correcting to actual scale relative to the world coordinate system by using the size of the fiducial objects and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera; mapping pixels of the 2D camera images depicting the surface of the body to 3D coordinates in the common coordinate system; and applying a calibration transformation to map pixels of the 2D ultrasound image represented in the ultrasound coordinate system to the camera coordinate system, wherein the calibration mapping is based on a predetermined relationship between the pose of the camera and the pose of an ultrasound transducer that captures the 2D ultrasound image; and mapping pixels of the 2D ultrasound image represented in the camera coordinate system to 3D coordinates in the common coordinate system.

[0030] A further implementation of the third aspect further includes repeating acquiring ultrasound images and camera images in a common region of the body segment and repeating calculating in a common coordinate system, and repeating reconstructing the 3D image includes repeating updating a previous 3D image of a previous iteration with the 3D voxels calculated in the current iteration to obtain an updated 3D image having a higher resolution than the previous 3D image.

[0031] A further implementation of the third aspect further includes receiving a 3D anatomical image depicting the body segment, where the 3D anatomical image is captured by another 3D imaging modality device, and registering between the 3D anatomical image and the reconstructed 3D image according to features extracted from the surface of the body segment.

[0032] According to a fourth aspect, the viscous material comprises at least a portion of a fluid containing a plurality of fiducial objects randomly distributed within the viscous material and spaced apart by random distances for application to a body surface of a subject, the plurality of fiducial objects having a size and contrast relative to the viscous material for use in calculating 3D coordinates to be assigned to pixels of the 2D ultrasound image to obtain 3D voxels depicted by a 2D camera image and used to estimate the orientation of the ultrasound transducer.

[0033] A further implementation of the fourth aspect further allows for computing a reconstruction of a 3D ultrasound image depicting either the interior of the body or the surface of a body region of a patient.

[0034] In a further implementation of the fourth aspect, the viscous material includes ultrasound gel, and the 3D image is reconstructed from 2D ultrasound images captured by ultrasound energy transmitted through the ultrasound gel.

[0035] In a further implementation of the fourth aspect, the plurality of fiducial objects have a small size selected to prevent them from appearing in 2D ultrasound images transmitted through ultrasound gel and / or are made from an acoustic material selected to do so.

[0036] In a further implementation of the fourth aspect, the plurality of fiducial objects are not fixed at specific locations within the viscous material, but rather flow within the viscous material in response to movement of the viscous material.

[0037] In a further implementation of the fourth aspect, each of the plurality of fiducial objects is about 0.5 to 1 millimeter in size.

[0038] In a further implementation of the fourth aspect, the plurality of fiducial objects are made of a material that is visually enhanced in response to ultraviolet light.

[0039] In a further implementation of the fourth aspect, the plurality of fiducial objects are made of a fluorescent material.

[0040] In a further implementation of the fourth aspect, the plurality of fiducial objects are made spherical.

[0041] In a further implementation of the fourth aspect, the density of the plurality of fiducial objects is between about 1 and 1000 per milliliter.

[0042] According to a fifth aspect, a method for treating a vascular condition in an individual includes applying a viscous material, the viscous material comprising at least a portion of a fluid containing a plurality of fiducial objects randomly distributed in 3D and separated by random distances, to a surface of a body segment of the individual at a location corresponding to a blood vessel of interest; manipulating an ultrasound transducer probe having an add-on component including at least one camera along the body surface to simultaneously capture ultrasound images of the blood vessel within the body and camera images depicting the surface of the body segment and depicting the plurality of fiducial objects; analyzing the reconstructions of the 3D images of the blood vessel and the surface of the body segment positioned within a common 3D coordinate system representing real-world coordinates; diagnosing the vascular condition based on the reconstruction of the 3D image of the blood vessel relative to the surface of the body segment; and treating the vascular condition during an open surgical and / or catheter procedure.

[0043] A further implementation of the fifth aspect further includes re-manipulating the probe on the surface of the body to simultaneously capture ultrasound images of the blood vessel depicting the treated vascular condition, analyzing the treated vascular condition in another 3D reconstruction of the blood vessel created from the ultrasound images and the camera images captured during the re-manipulation of the probe, and re-treating the treated vascular condition if the treated vascular condition is determined to require another treatment procedure based on the another 3D reconstruction.

[0044] A further implementation of the fifth aspect further includes repeatedly steering the ultrasound transducer in an axial and / or longitudinal orientation of the ultrasound transducer within a small area of ​​the surface corresponding to the vascular pathology to capture multiple ultrasound images of the vascular pathology at different angles and distances, wherein the resolution of the vascular pathology in the 3D image increases as the number of times the ultrasound transducer is steered over the small area increases.

[0045] In a further implementation of the fifth aspect, the 3D image is segmented according to tissue type, and further comprising analyzing the segmentation of the 3D image to determine the effectiveness of the vascular treatment.

[0046] In a further implementation of the fifth aspect, diagnosing a vascular condition based on reconstruction of a 3D image of a blood vessel relative to a surface of a body segment is further based on blood flow in the blood vessel depicted by the 3D image.

[0047] In a further implementation of the fifth aspect, the treatment is selected from the group consisting of a catheter-delivered stent, balloon dilation, ablation, drug injection, and manual removal and / or repair.

[0048] According to a sixth aspect, a calibration device for calibrating a camera image captured by a camera positioned at a fixed orientation relative to ultrasound transmission and a transformation mapping applied to an ultrasound image captured by an ultrasound transducer includes a single compartment having a circular top surface containing an ultrasound-transparent ultrasound inspection medium, and a ground truth pattern positioned outside the outer perimeter of the circular top surface of the compartment, the compartment and ground truth pattern being positioned such that when the ultrasound transducer is placed on top of the compartment, the camera captures a camera image of the ground truth pattern and the ultrasound transducer captures an ultrasound image of the interior of the compartment from the circular top surface to the bottom surface.

[0049] According to a seventh aspect, an add-on to an ultrasound probe includes a connector component sized and shaped to connect to an ultrasound transducer probe, a first camera housing including a first camera, and a second camera housing including a second camera, the first camera housing and the second camera housing each oriented at a predetermined angle relative to a longitudinal axis of the connector component so as to fix the first camera and the second camera at a predetermined angle relative to the ultrasound probe and relative to an ultrasound image captured by the ultrasound transducer.

[0050] A further implementation of the seventh aspect further includes an ergonomic holder component that includes a grip designed to be held against the palm of a user's hand and a ring or trigger-like element designed to support the user's index finger.

[0051] In a further implementation of the seventh aspect, the first camera housing and the second camera housing are separated by approximately 90 degrees that is perpendicular to the longitudinal axis of the connector component.

[0052] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present invention, exemplary methods and / or materials are described below. In case of conflict, the patent specification, including definitions, will control. Furthermore, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.

[0053] Some embodiments of the present invention are herein described, by way of example only, with reference to the accompanying drawings. While specific reference will now be made in detail to the drawings, it is emphasized that the details shown are for the purpose of illustrating and discussing embodiments of the invention by way of example. In this regard, the description taken with the drawings will make apparent to those skilled in the art how embodiments of the invention may be practiced. [Brief explanation of the drawings]

[0054] [Figure 1] 1 is a block diagram of components of a system 100 for generating 3D images from ultrasound and camera images based on analysis of fiducial objects randomly distributed in a viscous material, according to some embodiments of the present invention. [Figure 2] 1 is a flowchart of a method for generating a 3D image from ultrasound and camera images based on an analysis of fiducial objects randomly distributed in a viscous material, according to some embodiments of the present invention. [Figure 3] 1 is a flowchart of a method for diagnosing and / or treating vascular conditions according to a 3D image generated from ultrasound images and camera images based on analysis of fiducial objects randomly distributed in a viscous material, according to some embodiments of the present invention. [Figure 4] 1 includes a schematic diagram depicting the process of applying ultrasound gel with fiducial objects randomly distributed therein to capture camera images used to reconstruct 3D images, according to some embodiments of the present invention. [Figure 5] FIG. 1 is a schematic diagram depicting a camera capturing camera images of fiducial objects randomly distributed within ultrasound gel through which an ultrasound transducer of an ultrasound probe captures ultrasound images, according to some embodiments of the present invention. [Figure 6] FIG. 1 is a schematic diagram depicting exemplary x, y, and z axes of an ultrasound transducer for calculating 3D coordinates within the 3D space Vx, Vx, Vz module of tissue depicted in an ultrasound image, according to some embodiments of the present invention. [Figure 7] 1 is a schematic diagram depicting correction of blood flow in ultrasound images of a blood vessel acquired at multiple orientations by an ultrasound transducer, according to some embodiments of the present invention. [Figure 8]1A-1C are schematic diagrams depicting the reconstruction of a 3D image including an image of an artery depicting a vascular pathology such as a stenosis and indicating blood flow velocity through the artery, according to some embodiments of the present invention. [Figure 9] FIG. 1 is a schematic diagram depicting a process of collecting data for a multi-dimensional (optionally sparse) dataset of a set of 3D coordinates corresponding to a voxel and / or group of voxels, according to some embodiments of the present invention. [Figure 10] 1A-1C are schematic diagrams depicting a representation of a multidimensional dataset of a particular 3D coordinate of a particular voxel as a sphere, with gray color values ​​depicting different B-mode grayscale values ​​acquired from various transducer orientations, according to some embodiments of the present invention. [Figure 11] 1A-1C are schematic diagrams illustrating the reconstruction of a 3D image based on the segmentation of a cluster of an (optionally sparse) multidimensional dataset containing arteries segmented into different tissue types, depicting blood flow values, according to some embodiments of the present invention. [Figure 12] 1 is a schematic diagram depicting an exemplary 3D image of the brachial artery calculated based on a camera image depicting a pattern on the surface of a subject's arm and an ultrasound image of the brachial artery, in accordance with some embodiments of the present invention; FIG. [Figure 13] 1A-1C are schematic diagrams depicting an environmental scene of generating a 3D image computed from camera images and ultrasound images with ultrasound gel having fiducial objects randomly distributed therein, according to some embodiments of the present invention. [Figure 14] FIG. 1 is a schematic diagram depicting an ultrasound probe for capturing camera images by a camera and an add-on to an ultrasound transducer for capturing ultrasound images used to reconstruct a 3D image, according to some embodiments of the present invention. [Figure 15]FIG. 1 is a schematic diagram depicting an exemplary add-on to an ultrasound probe including a connector component sized and shaped to connect to the ultrasound probe and a camera positioned at an angle relative to a long axis of the connector component that corresponds to the long axis of the ultrasound transducer of the probe, in accordance with some embodiments of the present invention. [Figure 16] FIG. 10 is a schematic diagram of another exemplary implementation of an add-on to an ultrasound probe, according to some embodiments of the present invention. [Figure 17] 1 depicts an exemplary implementation of a calibration device according to some embodiments of the present invention. [Figure 18] 10 depicts another exemplary implementation of another calibration device, according to some embodiments of the present invention. [Figure 19] 1A and 1B are schematic diagrams depicting standard vascular treatment using existing approaches, and vascular treatment using 3D arteries of 3D point clouds and / or meshes created from ultrasound images and camera images, according to some embodiments of the present invention. [Figure 20] 1A-1C are schematic diagrams depicting estimates of blood flow in a blood vessel obtained by an ultrasound transducer at multiple orientations and / or positions, calculated using a known 3D vascular model and computational fluid dynamics simulations, according to some embodiments of the present invention. [Figure 21A] FIG. 10 is a schematic diagram of another exemplary implementation of an add-on to an ultrasound probe, according to some embodiments of the present invention. [Figure 21B] FIG. 10 is a schematic diagram of another exemplary implementation of an add-on to an ultrasound probe, according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0055] The present invention, in some embodiments thereof, relates to the processing of medical ultrasound images, and more particularly to the reconstruction of 3D medical images from 2D ultrasound images.

[0056] The embodiments described herein may relate to 3D medical images depicting organs (e.g., liver, gallbladder, thyroid, joints, etc.) compatible with holistic imaging of perforated organs, with or without blood flow, and with or without Doppler (e.g., grayscale only).

[0057] The embodiments described herein may relate to 3D medical images depicting one or more blood vessels, e.g., femoral arteries, carotid arteries, aorta, and / or other body tissues, e.g., organs such as kidneys, liver, gallbladder, spleen, and thyroid gland. Body parts, e.g., joints, bones, cartilage, etc., may not necessarily have blood vessels large enough to be depicted in the image. Body parts may depict air-containing body parts, e.g., air-filled organs such as lungs and intestines. If a body part is filled with fluid, the fluid may not necessarily be blood, and / or non-blood-filled cavities, e.g., bladders and abscesses, may not necessarily flow at significant rates.

[0058] The embodiments described herein with respect to Doppler are not necessarily limiting examples, and other ultrasound imaging modalities may alternatively or additionally be used, such as B-mode, etc. As used herein, the term Doppler may be interchanged with the term B-mode where relevant.

[0059] As used herein, the term Doppler ultrasound image refers to at least one Doppler ultrasound image, the term 2D camera image refers to at least one 2D camera image, the term reconstructed 3D image refers to at least one reconstructed 3D image, and similarly for other images.

[0060] An aspect of some embodiments of the present invention relates to a viscous material, including at least a portion of a fluid (optionally an ultrasound gel) containing fiducial objects, the fiducial objects being randomly, i.e., three-dimensionally, distributed within the viscous material (e.g., ultrasound gel) and spaced at random distances. The fiducial objects may be, for example, small spheres of different colors, with diameters in the range of 0.5 to 1 millimeter. The viscous material, such as ultrasound gel, is designed to be applied to a subject's body surface, such as the skin, to transmit ultrasound energy between an ultrasound transducer in contact with the surface and anatomical structures within the body to capture 2D ultrasound images of the body's interior. The fiducial objects, which have a size and contrast relative to the viscous material, such as ultrasound gel, are depicted by a 2D camera image and used to calculate 3D coordinates assigned to the 2D ultrasound image, e.g., 3D coordinates assigned to 2D pixels within the 2D ultrasound image, generating a reconstruction of the 3D ultrasound image depicting the body's interior.

[0061] Blood flow may refer to blood flowing within a blood vessel, such as an artery and / or vein. Blood may flow, for example, through a stenosis (e.g., as described herein), over plaque, and / or over the interior wall of a blood vessel. Blood flow may be measured using velocity, which may indicate the speed and / or direction of blood flow. Blood velocity may be measured in units such as centimeters per second. Blood volume may be measured in blood volume, such as in units of cubic centimeters per second. Direction may be measured in degrees relative to a baseline, such as in degrees relative to the long axis of the blood vessel. Exemplary blood vessels include the carotid artery, aorta, femoral artery, superior and / or inferior vena cava, brachial, renal artery, hepatic artery, and / or other blood vessels elsewhere in the body. Other details of blood flow are described herein.

[0062] An aspect of some embodiments of the present invention relates to the reconstruction of 3D images depicting blood flow at multiple 3D locations in a blood vessel, such as the entire imaged vessel. As referred to herein, a 3D image may be described as a 3D point cloud, a mesh 3D object, a graph 3D, or the like. A Doppler ultrasound image depicting the blood vessel with blood flowing therethrough and measurements of blood flow within the region of the vessel are acquired. For example, a 2D ultrasound image depicts the vessel and blood velocity measurements for pixels in the 2D ultrasound image. This image may be a standard B-mode image associated with standard Doppler mode measurements. A 2D camera image, captured by a camera, is acquired depicting fiducial objects randomly distributed in 3D within a viscous material, such as ultrasound gel, and separated by random distances on the surface of a body segment. The camera images may be acquired simultaneously with the ultrasound images, e.g., the camera is an add-on to an ultrasound probe. For each pixel in the Doppler ultrasound image, 3D coordinates in a world coordinate system depicting real-world distances and positions are calculated. The world coordinate system defines the scale of the 3D coordinates, which may be in Euclidean space and correspond to real-world distances and positions, i.e., no spatial transformations are applied. The scale of the world coordinate system may be 1:1 or another scale using real-world distances, e.g., a 5 cm distance between the skin and an artery depicted in an image represents a 5 cm distance inside the actual subject's body. The 3D coordinates are calculated using an external reference of the orientation of the ultrasound transducer used to capture the Doppler images. This external reference is calculated by analyzing the relative change in the position of the fiducial object in successive 2D camera images captured when the ultrasound probe is manipulated while the Doppler images are captured. The 3D position and / or orientation of the 2D camera within the external reference is calculated based on the analysis of the relative change in the position of the fiducial object. The 3D position and / or orientation of the ultrasound transducer is calculated from the 3D position and / or orientation of the 2D camera, e.g., based on a known, calibrated, fixed transformation between the position and / or orientation of the camera and the ultrasound transducer.The position of the Doppler image within the external reference is calculated according to the position and / or orientation of the camera and ultrasound transducer. The position of the Doppler image within the external reference is mapped to a world coordinate system by a calibration mapping function. Correction for the actual scale of the transducer's structure and / or motion in world units may be obtained using additional information, such as estimating an a priori known size of the fiducial object (e.g., the diameter of a sphere) and / or additional information from additional sensors, such as an inertial measurement unit or additional cameras. Estimated blood flow rates are calculated for multiple pixels of the Doppler ultrasound image at multiple positions within the blood vessel. The measured blood flow rates are corrected to obtain actual blood flow values ​​by estimating the angle between the ultrasound transducer axis transmitting the US beam and a vector indicating the direction of blood flow within the blood vessel. A 3D image is reconstructed from 3D voxels in the world coordinate system. 3D voxels are calculated by assigning 3D coordinates to pixels in the Doppler ultrasound image. Each 3D voxel is associated with an estimated blood flow rate. The 3D images depict an anatomical image of a blood vessel and an indication of blood flow at multiple locations within the vessel. For example, different blood velocities can be color-coded, allowing for a visual presentation of locations of high blood flow within the vessel and / or locations of low blood flow within the vessel. This allows for the diagnosis of vascular pathologies, such as stenosis, where high blood flow velocities occur within a narrow region of the vessel.

[0063] An aspect of some embodiments of the present invention relates to the reconstruction of 3D images depicting the surface (e.g., skin) and internal structures of an individual's body segment, e.g., the femoral artery and leg skin. Presenting the surface and internal structures within the same set of coordinates in the same 3D image allows visualization of the position of the internal structure relative to the surface, e.g., the position of the femoral artery relative to the leg skin. Acquire 2D ultrasound images depicting the internal tissue within the individual's body segment. Acquire 2D camera images captured by a camera depicting the surface of the body segment and depicting fiducial objects randomly distributed in 3D within a viscous material (e.g., ultrasound gel) and spaced at random distances on the surface of the body segment. The camera and ultrasound images may be acquired simultaneously and / or near simultaneously. The 3D coordinates of the pixels of the 2D camera image and the 3D coordinates of the pixels of the 2D ultrasound image are calculated in a common coordinate system and / or scale-corrected by using two cameras and / or by an inertial measurement unit or by prior knowledge of the average size of the fiducial objects, such as spheres. 3D coordinates are calculated based on an analysis of relative changes in the position of fiducial objects in successive 2D camera images. 3D voxels are calculated by assigning 3D coordinates to pixels in the camera image and the ultrasound image. The 3D image is reconstructed by calculating the position of the 3D voxels based on the transducer motion in the world coordinate frame, which is determined by using the camera motion relative to fiducial markers embedded in ultrasound gel. The 3D reconstruction (e.g., the 3D image) also depicts the surface of the body segment and tissues within the body segment positioned relative to the surface. The reconstructed 3D image, with additional layers of information, such as Doppler and / or B-mode, depicts tissues selected in any compatible ultrasound organ scanning procedure, including at least one of blood vessels, organs, joints, bones, cartilage, non-blood-filled cavities, the native gallbladder, and the thyroid gland, and fuses these tissues into a single whole organ or portion of an organ.

[0064] Some embodiments of the present invention relate to segmenting a 3D image reconstructed from 2D ultrasound images. Acquire 2D ultrasound images depicting a common region of a body segment. Acquire 2D camera images captured by a camera depicting fiducial objects randomly distributed in 3D within a viscous material (e.g., ultrasound gel) and spaced at random distances on the surface of the body segment. The camera and ultrasound images may be acquired simultaneously and / or nearly simultaneously. Calculate 3D voxels having 3D coordinates in a world coordinate system. The 3D voxels are calculated by assigning 3D coordinates to pixels in the 2D ultrasound image using an external reference for the pose of the ultrasound transducer capturing the ultrasound image. The pose of the ultrasound transducer is calculated by analyzing relative changes in the position of the fiducial objects in successive 2D camera images and / or is corrected for by using two cameras and / or by an inertial measurement unit or by prior knowledge of the average size of the fiducial objects, such as spheres. Reconstruct a 3D image from the 3D voxels. For each 3D voxel, a multidimensional dataset is calculated and stored. The multidimensional dataset may (e.g., typically) be a sparse dataset. When the ultrasound transducer moves and / or is reoriented, different intensity values ​​are acquired for the same 3D coordinates corresponding to the same 3D voxel for different ultrasound transducer orientations. The multidimensional dataset maps between each ultrasound transducer orientation and the intensity values ​​acquired at the 3D coordinates corresponding to each 3D voxel acquired for each ultrasound transducer orientation. The 3D voxels are clustered into multiple clusters according to the distribution of the multidimensional dataset of multiple 3D voxels. The 3D image is segmented according to the subset of 3D voxel members in each cluster. The voxels in each segmented cluster may represent similar tissues.Clustering may be performed by clustering voxels according to some similarity measure of grayscale and / or color (e.g., coloring the pattern of variation) Doppler value distributions obtained by capturing ultrasound data from different angles and / or distances of the location corresponding to each voxel relative to the ultrasound transducer, with the voxels in each cluster indicating respective tissue types that produce respective similar patterns of variation in the ultrasound data captured at different angles and / or distances relative to the ultrasound transducer.

[0065] When using ultrasound technology to image organs and images of the organs, extraneous noise is recorded and interferes with the actual image of the organ being imaged. The segmentation approach described herein also increases the signal-to-noise ratio (SNR) of the captured data. When imaging a specific region of the body, multiple images of the same region are acquired. For the same pixel, corresponding to a voxel and / or 3D location, when viewed from different angles and distances from the ultrasound probe, each pixel converted to a voxel is associated with multiple grayscale values, i.e., intensity values, for each pose and distance. Each human tissue, including pathological tissue, may have the same grayscale value recorded from different angles and distances. An optional unsupervised learning approach, such as manifold learning, identifies those tissues with the same grayscale value and similar grayscale value variations and / or distributions. Tissues of the same type are optionally segmented using several similarity measures that define clusters. Since the similarity between voxels is based on grayscale values ​​and grayscale distributions, which may include topological relationships between voxels, adjacent voxels need to be mapped and segmented together in the mapping space, which may result in higher accuracy and / or precision in segmenting highly complex tissues.

[0066] Some embodiments of the present invention relate to treating vascular conditions within an individual's blood vessels. A viscous material, a fluid (optionally an ultrasound gel) containing fiducial objects at least some of which are randomly distributed in 3D and spaced apart at random distances, is applied to the surface of a body segment of the individual at a location corresponding to the blood vessel of interest. An ultrasound transducer probe with add-on components including one or more cameras is manipulated along the body surface and may include an inertial measurement unit. Ultrasound images of the blood vessel within the body and camera images depicting the surface of the body segment and depicting the fiducial objects are simultaneously captured. A reconstruction of the 3D image of the blood vessel and the surface of the body segment is generated, positioned within a common 3D coordinate system representing real-world coordinates depicting real-world distances and positions. Alternatively, or additionally, the 3D image depicts an indication of blood flow (e.g., velocity) at multiple locations within the blood vessel. Alternatively, or additionally, the 3D image is segmented into multiple segments, each depicting a respective tissue type, e.g., a different layer of the blood vessel wall, a stenosis-causing substance, etc. Diagnosing vascular pathologies (e.g., stenosis) is based on the reconstruction of 3D images of blood vessels relative to the surface of a body segment, and / or based on blood flow patterns within the vessels, and / or based on segmented 3D images. Vascular pathologies are treated during open surgery and / or catheter procedures. The 3D reconstructions (e.g., 3D images) can be used to guide treatment tools (e.g., catheters).

[0067] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of improving the accuracy of calculating 3D coordinates assigned to 2D image elements, e.g., pixels of a 2D ultrasound image depicting internal body structures to reconstruct a 3D image using the 3D coordinates, e.g., converting each 2D pixel to a 3D voxel of the 3D image by calculating a corresponding 3D coordinate. At least some implementations of the systems, methods, devices, and / or code instructions described herein improve techniques for reconstructing 3D images from 2D ultrasound imaging data by improving the accuracy of 3D coordinates assigned to image elements, e.g., pixels, of the 2D ultrasound image. At least some implementations of the systems, methods, devices, and / or code instructions described herein provide a solution to the technical problem and / or an improvement to the technique is provided by fiducial objects randomly, i.e., distributed in 3D and spaced at random distances, within a viscous material (optionally an ultrasound gel). The fiducial objects have a size and / or contrast relative to the viscous material depicted by the 2D camera image that can be used to calculate 3D coordinates assigned to the 2D US image and assist in correcting the scale of the reconstructed 3D image. The 2D (e.g., monocular RGB) camera may be located as an add-on and / or integrated with the ultrasound probe, capturing images of ultrasound gel containing the fiducial objects, which then captures the 2D images by transmitting energy through the ultrasound gel containing the fiducial objects. The small size, random distribution and / or spacing, and / or high density of the fiducial objects provide the highly textured image necessary for accurate calculation of 3D coordinates. This improvement is superior to other existing approaches that use, for example, surface skin features, temporary tattoos bearing features, or other expensive sensors to sense probe orientation, which are prone to error and / or inaccurate in calculating highly accurate 3D coordinates.

[0068] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of measuring blood flow at multiple locations within a blood vessel using Doppler. At least some implementations of the systems, methods, devices, and / or code instructions described herein improve techniques for analyzing Doppler data to measure blood flow and / or for reconstructing 3D images from 2D ultrasound imaging data by generating a 3D image of the blood vessel depicting blood flow at multiple locations within the vessel using actual estimated blood flow values. At least some implementations of the systems, methods, devices, and / or code instructions described herein provide a solution to the technical problem and / or an improvement to the technique by correcting an initial measurement of blood flow made using Doppler to obtain an actual blood flow value. Using a standard approach, a user positions the ultrasound probe relative to the long axis of the blood vessel and changes the vector indicator to correct the actual angle as best as possible based on experience and / or "eyeballing." One or a few individual Doppler measurements are taken. The improvement provided herein is to calculate (herein referred to as "correcting") actual blood vessel values ​​from Doppler-measured blood values ​​when setting the angle between the ultrasound transducer and blood velocity, without having to manually set the angle of actual blood flow velocity. Rather than requiring tedious manual setup of assumed blood flow angles, the proposed method calculates blood flow (e.g., velocity) vectors for multiple locations within the vessel, thereby enabling the generation of a visually corrected 3D image of blood flow that visually depicts blood flow along the vessel.

[0069] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of improving the visual representation of body structures depicted in 3D images reconstructed from 2D ultrasound images. At least some implementations of the systems, methods, devices, and / or code instructions described herein improve the technique of reconstructing 3D images from 2D ultrasound images. At least some implementations of the systems, methods, devices, and / or code instructions described herein provide a solution to the technical problem and / or an improvement to the technique by including imaging data captured by a 2D camera image. The imaging data captured by the 2D camera image is contained within the same coordinate system used to represent the ultrasound image in the 3D image. The resulting 3D image, after scale correction, which can be performed by integrating additional information such as the average size of spheres in ultrasound gel or information from an additional sensor such as an inertial measurement unit or another camera, accurately depicts the position of the internal body structure imaged by the ultrasound image relative to the surface (e.g., skin) of the body structure imaged by the camera. For example, while performing an invasive procedure on the femoral artery, a user may visually inspect the 3D image to determine where the thigh is located within the leg, improving guidance of tools to the femoral artery.

[0070] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of segmenting a 3D image reconstructed from a 2D ultrasound image. At least some implementations of the systems, methods, devices, and / or code instructions described herein improve the technology of image processing by segmenting a 3D image reconstructed from a 2D ultrasound image. At least some implementations of the systems, methods, devices, and / or code instructions described herein provide a solution to the technical problem and / or an improvement to the technology provided by a multidimensional dataset that maps between each ultrasound transducer orientation and intensity values ​​acquired at 3D coordinates corresponding to each 3D voxel acquired for each ultrasound transducer orientation. The 3D voxels are clustered into multiple clusters according to the distribution of the multidimensional dataset of the multiple 3D voxels. The 3D image is segmented according to a subset of 3D voxel members in each cluster. Voxels within each segmented cluster may represent similar tissue. Such segmentation is an improvement over standard segmentation approaches, which are based on visual features in 2D ultrasound images, for example. Because the same tissue appears differently at different ultrasound transducer poses, standard approaches may inaccurately segment the same tissue. Furthermore, some ultrasound transducer poses may have artifacts, which can reduce the power and / or accuracy of the segmentation. In contrast, as described herein, the same tissue type is segmented based on the pattern of intensity value distribution for multiple ultrasound transducer poses, and neighboring distribution information may also be used, thereby improving the accuracy of segmenting the same tissue type.

[0071] Implementation of at least some of the systems, methods, devices, and / or code instructions described herein addresses the medical problem of improved diagnosis of vascular conditions and / or improved treatment of vascular conditions. Implementation of at least some of the systems, methods, devices, and / or code instructions described herein improves the field of medicine by improving diagnosis of vascular conditions and / or improving treatment of vascular conditions. Implementation of at least some of the systems, methods, devices, and / or code instructions described herein provides a solution to a medical problem and / or an improvement to medicine through the reconstruction of 3D images depicting the surfaces of blood vessels and body segments located within a common 3D coordinate system, and / or 3D images depicting indications of blood flow (e.g., velocity) at multiple locations within a blood vessel, and / or 3D images segmented into multiple segments, each depicting a respective tissue type. The 3D images improve a user's ability to make a diagnosis, such as identifying a stenosis, perform a treatment, such as guiding a catheter from the skin to the stenosis, and / or evaluate the outcome of a treatment, such as checking blood flow patterns with a stent in place.

[0072] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of computing 3D and / or 4D images, particularly without the use of radiation such as x-rays. At least some implementations of the systems, methods, devices, and / or code instructions described herein improve the field of ultrasound imaging, particularly the field of ultrasound images, by computing 3D and / or 4D ultrasound images from 2D ultrasound images, optionally using fiducial objects randomly distributed in an echography gel (or another viscous medium). The generated 3D and / or 4D images depict larger tissue regions depicted in each 2D ultrasound image by aggregating voxels created by assigning 3D coordinates to pixel intensity values ​​of pixels in the 2D ultrasound images. The generated 3D and / or 4D images may indicate blood flow and / or blood volume. The 2D ultrasound images can be acquired sequentially, optionally using a standard 2D ultrasound probe. The 2D ultrasound images can be captured as B-mode and / or color mode (Doppler) images.

[0073] At least some implementations of the systems, methods, devices, and / or code instructions described herein provide a fixed pattern to be depicted in a camera image captured by a camera associated with an ultrasound transducer, addressing the technical problem and / or improving the technical field of generating a 3D image from an ultrasound image captured by the ultrasound transducer. A transformation mapping maps the fixed pattern in the camera image captured by the camera to 3D coordinates assigned to the 2D ultrasound image captured by the ultrasound transducer. Scale correction relative to the actual scale of structures and motion in world units can be recovered by adding known fiducial markers or by integrating information from additional sensors, such as an inertial measurement unit or another camera. Other conventional approaches have used features of the skin itself, such as hair, wrinkles, and birthmarks; external sensors, such as inertial sensors, that measure the transducer's orientation; and / or temporary tattoos placed on the skin. At least some implementations of the systems, methods, devices, and / or code instructions described herein relate to ultrasound gel in which fiducial objects are randomly distributed. The fiducial objects may be small enough and / or made from a selected acoustic material to prevent artifacts on the ultrasound image. The fiducial objects may be large enough to be depicted in the camera image. Ultrasound gel is used to capture the ultrasound image and also contains randomly distributed fiducial objects that provide a fixed pattern depicted in the camera image.

[0074] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical and / or medical problem of treating vascular conditions in a subject. Using standard approaches, vascular conditions are mapped by continuous 2D ultrasound imaging, optionally in Doppler mode, and / or by performing angiography, such as x-ray, CT, MRI, etc.; however, angiography, such as x-ray angiography or CT angiography, is an invasive procedure that injects contrast agents into the subject's vasculature, which may cause allergic reactions and / or be nephrotoxic, and / or exposes the subject to high radiation doses. At least some implementations of the systems, methods, devices, and / or code instructions described herein improve the technical and / or medical field of treating vascular conditions in a subject and / or address the technical and / or medical problem by generating 3D images from 2D ultrasound images, optionally including Doppler, that can be acquired rapidly in a non-invasive manner without exposing the subject to radiation and / or injecting contrast agents. 3D images may be captured in real time to guide catheters used to treat vascular conditions, for example, by stent insertion, balloon dilation, ablation, drug injection, and / or other approaches, and / or may be used to evaluate vascular conditions before and after treatment. Real-time post-treatment evaluation using 3D images can help determine whether further immediate treatment is needed and / or whether treatment was successful. After treatment, follow-up 3D images can be acquired over a longer period to determine whether treatment is successful or whether additional treatment is needed. Multiple 3D images such as these cannot be acquired using standard approaches to evaluate treatment due to the invasiveness of standard approaches and / or the risk of large amounts of radiation and / or contrast agents.

[0075] In at least some implementations, improvements include adding additional features to the acquired images using ultrasound gel or other viscous material with randomly distributed fiducial objects to enable more robust and accurate estimation of the transducer pose, resulting in more accurate and precise 3D images, the 3D images depicting the interior of the blood vessel along with the skin in real-world coordinates depicting real-world relative distances and positions, the 3D images being corrected for blood flow (e.g., velocity, speed) at multiple locations within the blood vessel, and / or the 3D segmentation being based on a cluster of multidimensional datasets corresponding to different voxel locations and distributions, and / or mapping between ultrasound transducer pose and acquired intensities, each multidimensional dataset may be of a set of 3D coordinates corresponding to one voxel and / or a group of voxels.

[0076] Traditional approaches have attempted to achieve real-time three-dimensional ultrasound visualization by either using volumetric probes or constantly moving two-dimensional probes. Medical imaging techniques provide clinical audiovisual information about organs for better clinical and decision-making. Therefore, medical imaging of the interior of the body is often considered the most prominent technology for clinical analysis and medical intervention. Medical imaging includes a wide range of techniques, but the most common tools are radiography, computed tomography scans (CT scans), magnetic resonance imaging (MRI), and medical ultrasound (US).

[0077] One of the standard tools in medical imaging is the CT scan. Computed tomography, also known as a CT scan, uses specialized X-ray equipment to generate cross-sectional 2D images from a series of 1D X-ray images taken around a single rotational axis centered on the object. As a bed passes through the rotating "doughnut"-shaped X-ray machine, a series of 2D axial images are acquired. These images can be displayed individually or superimposed to form a 3D model. The disadvantages of CT scans are the high radiation exposure and the need for injections of nephrotoxic iodine as a contrast agent. In contrast to CT scans, exposure to ionizing radiation during MRI procedures is virtually zero. MRI technology uses the body's magnetic properties to produce highly detailed 3D images. However, MRI is very expensive, requires a specialized room, cannot be used on patients with implanted metal or claustrophobic tissue, and is sensitive to patient movement. In a standard ultrasound system, a probe transmits high-voltage, high-frequency waves into the patient's body. When these waves encounter organs with different densities, they are reflected, producing echoes that are measured by the probe. The returning echoes are converted into electrical signals from which an algorithm generates a 2D image. The advantages are the absence of radiation and the injection of contrast agents, which are harmful to the kidneys. It is also the most cost-effective imaging technique. However, it lacks a 3D image of the entire organ, is highly user-dependent, and is therefore considered an "operator-dependent" device, resulting in the poorest imaging quality.

[0078] The use of US technology in the embodiments described herein avoids the radiation and contrast agents currently used in CT and angiography examinations, thereby eliminating the harm to patients and caregivers from radiation and the potential for future deterioration of the patient's kidney function from contrast agents.

[0079] Implementation of at least some of the systems, methods, devices, and / or code instructions described herein addresses the above technical problems and / or improves the above technical field by generating 3D ultrasound images without ionizing radiation and without contrast agents that can be rapidly captured using a standard US probe used on patients with implanted metallic components and / or that can depict physiological data of blood vessels such as blood velocity (e.g., centimeters per second) and / or blood volume (e.g., cubic centimeters per second).

[0080] At least some implementations of the systems, methods, devices, and / or code instructions described herein generate 3D US images of blood vessels without x-ray exposure and / or administration of contrast agents, in contrast to standard approaches for capturing angiographic images of blood vessels, which require administration of contrast agents to the vasculature and / or capture of x-ray images.

[0081] Implementations of at least some of the systems, methods, devices, and / or code instructions described herein generate higher quality 3D US images compared to conventional approaches. Implementations of at least some of the systems, methods, devices, and / or code instructions described herein generate more accurate segmentation and / or higher resolution 3D US images by eliminating different noise sources that are drawbacks of standard US systems.

[0082] At least some implementations of the systems, methods, devices, and / or code instructions described herein add an additional layer of information to the reconstructed 3D US image, which can depict one or more of the velocity of blood flowing through the imaged vessel and the volume of blood flowing through the imaged vessel. In contrast, CT or MRI, which are primarily used for vascular imaging due to their capabilities, cannot display important physiological information that can be measured using only the Doppler modality of US systems.

[0083] Implementations of at least some of the systems, methods, devices, and / or code instructions described herein may be added to, and / or connected to, and / or integrated with, existing 2D US acquisition systems.

[0084] Implementation of at least some of the systems, methods, devices, and / or code instructions described herein can reduce operator dependency, for example to a minimum, by enabling less experienced users to operate US imaging systems and perform high-quality examinations of patients.

[0085] At least some implementations of the systems, methods, devices, and / or code instructions described herein reconstruct 3D US images that depict a complete image of a tissue and / or organ compared to 2D US images and / or other approaches. The reconstructed 3D US images can be registered with other 3D images acquired by other 3D imaging modalities, such as CT angiography (CTA), MRI angiography (MRA), etc.

[0086] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of increasing the accuracy and / or computational efficiency of calculating the pose (e.g., position and / or orientation) of an ultrasound transducer during the capture of an ultrasound image and / or improve the technical field of calculating the pose of an ultrasound transducer during the capture of an ultrasound image. The calculated pose of the ultrasound transducer for each ultrasound image can be used to reconstruct a 3D image, as described herein. Calculating the pose of the ultrasound transducer by mapping a camera image of a fiducial object in ultrasound gel or other viscous medium with a calibrated transformation mapping is more accurate and / or cost-effective, less complex, and / or more computationally efficient (e.g., requires fewer processing resources and / or fewer memory resources) compared to other approaches, e.g., using only inertial measurement units, cameras capturing natural skin features such as hair, birthmarks, optical tracking systems, mechanical systems, and optical fibers, magnetic sensors, etc.

[0087] Implementations of at least some of the systems, methods, devices, and / or code instructions described herein address the technical problem of reducing operator dependency and / or improving the user experience when performing ultrasound examinations and / or improve the technical field of ultrasound systems designed for use by inexperienced users. Camera images of different ultrasound transducer orientations are used to calculate the orientation and / or 3D coordinates of the ultrasound transducer relative to the ultrasound image, allowing an inexperienced user to obtain relevant information that can be further analyzed by an expert. The data collected by the inexperienced user is used to reconstruct a 3D image.

[0088] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of creating improved ultrasound images of organs with relatively reduced noise and / or artifacts, and / or improve the art of ultrasound imaging by creating ultrasound images of organs with relatively reduced noise and / or artifacts. Using standard approaches, capturing each 2D ultrasound image independently to reduce noise and / or artifacts relies on the user's ability to capture the image in an optimal position. In contrast, 3D images created herein by aggregating 3D voxel data calculated from multiple 2D images of the ultrasound transducer from different positions reduce noise and / or artifacts in the reconstructed 3D image.

[0089] Implementations of at least some of the systems, methods, devices, and / or code instructions described herein address the technical problem of creating 3D anatomical images of blood vessels that also depict an indication of blood flow (e.g., velocity and / or volume) through the vessels and / or improve the field of medical imaging by creating 3D anatomical images of blood vessels that also depict an indication of blood flow through the vessels. Standard approaches are designed for one data type, such as CT and MRI, to generate 3D anatomical images. CT and MRI are primarily used for vascular imaging due to their capabilities, but are unable to display important physiological information obtained by using the Doppler modality of US systems. In contrast, Doppler US is designed to measure blood flow but does not provide good anatomical images of blood vessels.

[0090] Before describing at least one embodiment of the present invention in detail, it is to be understood that the invention is not necessarily limited in its application to the details of construction and the arrangement of components and / or methods set forth in the following description and / or illustrated in the drawings and / or examples. The invention is capable of other embodiments or of being practiced or carried out in various ways.

[0091] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0092] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction-execution device. A computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exclusive list of more specific examples of computer-readable storage media includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EEPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as being a transitory signal itself, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over an electrical wire.

[0093] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface of each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within the respective computing / processing device.

[0094] The computer-readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, or the like, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to implement aspects of the present invention.

[0095] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0096] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to manufacture a machine, such that the instructions, executing via the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in the flowcharts and / or block diagrams of the block or blocks. These computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that a computer-readable storage medium having instructions stored thereon includes an article of manufacture containing instructions that implement aspects of the functions / acts specified in the flowcharts and / or block diagrams of the block or blocks.

[0097] Furthermore, computer-readable program instructions can be loaded into a computer, other programmable data processing apparatus, or other device to execute a series of operational steps on the computer, other programmable data processing apparatus, or other device to create a computer-implemented process, such that the instructions executing on the computer, other programmable data processing apparatus, or other device perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0098] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified functions or acts, or a combination of special-purpose hardware and computer instructions.

[0099] Referring now to FIG. 1, FIG. 1 is a block diagram of components of a system 100 for generating a 3D image from ultrasound images and camera images based on analysis of relative changes of fiducial objects randomly distributed in a viscous material, in accordance with some embodiments of the present invention. Also referring to FIG. 2, FIG. 2 is a flowchart of a method for generating a 3D image from ultrasound images and camera images based on analysis of relative changes of fiducial objects randomly distributed in a viscous material, in accordance with some embodiments of the present invention. Also referring to FIG. 3, FIG. 3 is a flowchart of a method for treating a vascular condition according to a 3D image generated from ultrasound images and camera images based on analysis of relative changes of fiducial objects randomly distributed in a viscous material, in accordance with some embodiments of the present invention. Also referring to FIG. 4, FIG. 4 includes a schematic diagram depicting a process of applying ultrasound gel with randomly distributed fiducial objects to capture camera images used to reconstruct a 3D image, in accordance with some embodiments of the present invention. Also, referring to FIG. 5, FIG. 5 is a schematic diagram illustrating a camera capturing camera images of fiducial objects randomly distributed within ultrasound gel, and an ultrasound transducer of an ultrasound probe capturing ultrasound images through the gel, in accordance with some embodiments of the present invention. Also, referring to FIG. 6, FIG. 6 is a schematic diagram illustrating exemplary x, y, and z axes of an ultrasound transducer for calculating 3D coordinates within a 3D space Vx, Vx, Vz module of tissue depicted in an ultrasound image, in accordance with some embodiments of the present invention. Also, referring to FIG. 7, FIG. 7 is a schematic diagram illustrating a scheme for estimating actual blood flow in ultrasound images of a blood vessel acquired at multiple orientations by an ultrasound transducer, in accordance with some embodiments of the present invention. Also, referring to FIG. 8, FIG. 8 is a schematic diagram illustrating the reconstruction of a 3D image of an artery depicting vascular pathology and indicating blood flow velocity through the artery, in accordance with some embodiments of the present invention. Also, referring to FIG. 9, FIG. 9 is a schematic diagram illustrating the process of collecting data for a multidimensional dataset of a set of 3D coordinates corresponding to one voxel, in accordance with some embodiments of the present invention.10 is a schematic diagram illustrating a representation of a multidimensional sparse dataset of a particular voxel, according to some embodiments of the present invention, where the dataset is represented as a colored sphere and the gray values ​​are the acquired grayscale B-mode values ​​of this voxel sparsely sampled from various transducer orientations. Another diagram illustrates the sparsely distributed grayscale values ​​of several voxels represented as spheres, each voxel located on an XYZ grid point. Also, referring to FIG. 11, FIG. 11 is a schematic diagram illustrating the reconstruction of a 3D image based on the segmentation of a cluster of a multidimensional dataset including an artery segmented into different tissue types having vascular pathologies, according to some embodiments of the present invention. Also, referring to FIG. 12, FIG. 12 is a schematic diagram illustrating an exemplary 3D image of a brachial artery calculated based on a camera image depicting a pattern on the surface of a subject's arm and an ultrasound image of the brachial artery, according to some embodiments of the present invention, including the surface of a patient's body. Also, referring to FIG. 13, FIG. 13 is a schematic diagram depicting an environmental scene for generating a 3D image calculated from camera images and ultrasound images using ultrasound gel with randomly distributed fiducial objects, in accordance with some embodiments of the present invention. Also, referring to FIG. 14, FIG. 14 is a schematic diagram depicting an add-on to an ultrasound probe for capturing camera images with a camera and an ultrasound transducer for capturing ultrasound images used to reconstruct a 3D image, in accordance with some embodiments of the present invention. Also, referring to FIG. 15, FIG. 15 is a schematic diagram depicting an exemplary add-on to an ultrasound probe including a connector component sized and shaped to connect to the ultrasound probe and a camera positioned at a predetermined angle relative to the long axis of the connector component corresponding to the long axis of the ultrasound transducer of the probe, in accordance with some embodiments of the present invention. Also, referring to FIG. 16, FIG. 16 is a schematic diagram of another exemplary implementation of an add-on to an ultrasound probe, in accordance with some embodiments of the present invention. Also, referring to FIG. 17, FIG. 17 is a schematic diagram depicting an exemplary implementation of a calibration device, in accordance with some embodiments of the present invention.Also, referring to FIG. 18, FIG. 18 is a schematic diagram depicting another exemplary implementation of another calibration device, according to some embodiments of the present invention. Also, referring to FIG. 19, FIG. 19 is a schematic diagram depicting standard vascular treatment using existing approaches, according to some embodiments of the present invention, and a schematic diagram depicting vascular treatment using 3D images created from ultrasound images and camera images. Also, referring to FIG. 20, FIG. 20 is a schematic diagram depicting estimates of blood flow in a blood vessel obtained by an ultrasound transducer at multiple orientations and / or positions, calculated using a known 3D vascular model and computational fluid dynamics simulations, according to some embodiments of the present invention. Also, referring to FIG. 21A-B, FIG. 21A-B are schematic diagrams of another exemplary implementation of an add-on to an ultrasound probe, according to some embodiments of the present invention.

[0100] The system 100 may implement the operations of the methods described with reference to Figures 2-21 by a hardware processor 102 of a computing device 104 executing code instructions 106A optionally stored in a memory 106.

[0101] Computing device 104 may be implemented as, for example, a client terminal, a server, a virtual server, a radiology workstation, an ultrasound workstation, a PACS server, a virtual machine, a computing cloud, a mobile device, a desktop computer, a thin client, a smartphone, a tablet computer, a laptop computer, a wearable computer, a glasses computer, a watch computer, and a ring computer. Computing 104 may include advanced visualization workstations, which may be provided as add-ons to ultrasound workstations and / or other devices, for presenting 3D images computed from ultrasound and camera images, as described herein.

[0102] The computing device 104 may include locally stored software that performs one or more of the operations described with reference to Figures 2-21 and / or may function as one or more servers, such as a network server, web server, computing cloud, virtual server, etc., that provide services, such as one or more of the operations described with reference to Figures 2-21, to one or more client terminals 108, such as ultrasound probes, remotely located ultrasound workstations, remote picture archiving and communication system (PACS) servers, remote electronic medical record (EMR) servers, computing devices that receive ultrasound images and camera images over the network 110, for example, providing software as a service (SaaS) to the client terminal 108, providing applications to the client terminal 108 for local download as add-ons to a web browser and / or medical imaging viewer application, and / or providing functionality to the client terminal 108 using a remote access session, such as via a web browser.

[0103] Different architectures based on the system 100 may be implemented. In one example, the computing device 104 provides centralized services to each of multiple ultrasound workstations associated with the ultrasound transducers 112 of each ultrasound probe. Each ultrasound transducer 112 is associated with a respective camera 114, which is at a fixed orientation relative to the ultrasound transducer, as described herein. The camera 114 may be located in an add-on component 254 connected to the probe of the ultrasound transducer 112, as described herein. Pairs of ultrasound images captured by each ultrasound transducer 112 and camera images captured by each camera 114 are provided to the computing device 104 via, for example, an API, a local application, over the network 110, and / or transmitted via a data repository, such as a PACS, EMR, server 118, and / or via the client terminal 108, using an appropriate transmission protocol. The computing device 104 analyzes the ultrasound image and camera image pairs and calculates one or more 3D images, as described herein. The 3D images may be provided to the client terminal 108 and / or the server 118 for presentation on a display, storage, and / or further processing. In another example, the computing device 104 provides dedicated services to one ultrasound transducer 112 and corresponding camera 114. For example, the computing device 104 may be integrated with an ultrasound workstation connected to the ultrasound transducer 112, e.g., code 106A may be installed on the ultrasound workstation that displays ultrasound images captured by the ultrasound transducer, and / or the computing device 104 may be connected to the ultrasound transducer 112 and camera 114, e.g., a smartphone running code 106A connected to the ultrasound transducer 112 and camera 114 via a short-range wireless connection, a USB cable, and / or other implementation.The ultrasound images captured by the ultrasound transducer 112 and the camera images captured by the camera 114 are processed by locally installed code 106A, and a 3D image is provided for presentation on a display of an ultrasound workstation connected to the ultrasound transducer 112 and / or on a display of a locally connected computing device 104, such as a smartphone. As the ultrasound images and camera images are captured, or shortly after the ultrasound images are captured, code 106A can provide an additional feature set to the ultrasound workstation connected to the ultrasound transducer 112 by dynamically computing the 3D image in real time or near real time.

[0104] In yet another example, the ultrasound images captured by the ultrasound transducer 112 and the camera images captured by the camera 114 may be stored in a data repository 122A that receives the captured ultrasound images and / or camera images, such as a memory and / or storage device of the computing device 104, a memory and / or storage device on an ultrasound workstation (e.g., 108) connected to the ultrasound transducer 112, an external hard drive connected to a client terminal 108 connected to the ultrasound transducer 112 and camera 114, a PACS server, and / or an EMR server, and a cloud storage server.

[0105] The computing device 104 can receive ultrasound images and camera images from the ultrasound transducer 112 and / or camera 114 and / or data repository 122A using one or more data interfaces 120, such as a wired connection (e.g., a physical port), a wireless connection (e.g., an antenna), a local bus, a port for connecting a data storage device, a network interface card, other physical interface implementation, and / or a virtual interface (e.g., a software interface, a virtual private network (VPN) connection, an application programming interface (API), a software development kit (SDK)).

[0106] The camera 114 may be, for example, a still camera, a video camera, a CMOS, a visible light sensor, a red-green-blue (RGB) sensor such as a CCD and / or CMOS sensor, an ultraviolet camera, an infrared camera, a depth RGBD camera, or the like.

[0107] The hardware processor 102 may be implemented as, for example, a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a digital signal processor (DSP), and an application specific integrated circuit (ASIC). The processor 102 may include one or more processors (homogeneous or heterogeneous), which may be arranged for parallel processing as a cluster and / or as one or more multi-core processing units.

[0108] It is noted that implementations of at least some of the systems, devices, methods, and / or code instructions described herein are designed to generate 3D images on a processor with relatively few computational resources, for example, a CPU in a smartphone, compared to generating 3D images on a high-performance processor such as a GPU. This allows images to be processed on readily available computing devices, such as laptops and / or smartphones, without the need to install a high-performance processor. Alternatively, implementations of at least some of the systems, devices, methods, and / or code instructions described herein are designed to generate 3D images on a high-performance processor.

[0109] Memory 106, also referred to herein as a program store and / or data storage device, stores code instructions for execution by hardware processor 102, e.g., random access memory (RAM), read-only memory (ROM), and / or storage devices, e.g., non-volatile memory, magnetic media, semiconductor memory devices, hard drives, removable storage, and optical media (e.g., DVD, CD-ROM). For example, memory 106 may store code 106A that implements one or more operations and / or features of the methods described with reference to FIGS. 2-21.

[0110] The computing device 104 may include a data storage device 122 for storing data, e.g., one or more data repositories 122A documenting data such as received ultrasound and camera images and / or documenting generated 3D images, which may be repeatedly updated in response to additional images, multidimensional datasets used in computing segmentation, transformation mappings, and / or other data described herein. The data storage device 122A may be implemented, for example, as memory, a local hard drive, a removable storage device, an optical disk, a storage device, and / or as a remote server and / or computing cloud, such as accessed via the network 110.

[0111] The system 100 may include an inertial measurement unit, IMU (not shown), which may be installed as an add-on to the ultrasonic probe 154 to provide measurements indicative of the attitude of the ultrasonic transducer 112, as described herein.

[0112] The computing device 104 may include a network interface 124 for connecting to the network 110, such as one or more of a network interface card, a wireless interface for connecting to a wireless network, a physical interface for connecting to a cable for network connection, a virtual interface implemented in software, network communications software providing an upper layer of network connectivity, and / or other implementations.

[0113] It is noted that the data interface 120 and the network interface 124 may exist as two independent interfaces (e.g., two network ports), as two virtual interfaces on a common physical interface (e.g., a virtual network on a common network port), and / or may be integrated into a single interface (e.g., a network interface). The computing device 104 may communicate with one or more of the client terminals 108 using the network 110 or another communication channel, e.g., by a direct link such as a cable, wireless and / or indirect link, through an intermediate computing device such as a server, and / or through a storage device. For example, if the computing device 104 functions as a central server, the central server may provide a centralized 3D image generation service, such as SaaS, to remote ultrasound workstations 108, which function as client terminals connected to ultrasound transducers 112 and cameras 114, respectively, to generate 3D images from remotely acquired ultrasound images and camera images. The server 118 may be implemented, for example, in association with a PACS, and may store ultrasound images and camera images acquired by the ultrasound transducer 112 and camera 114, and / or may store updated versions of the code 106A for upgrades thereof. 3D images may be reconstructed offline from image pairs stored on the server 118.

[0114] The computing device 104 and / or ultrasound transducer 112 and / or camera 114 and / or client terminal 108 and / or server 118 include and / or communicate with a user interface 126 having mechanisms designed for a user to input data such as patient data, perform the calibrations described herein, and / or view data such as reconstructed 3D images. Exemplary user interfaces 126 include, for example, one or more of a touchscreen, a display, a 3D headset, a keyboard, a mouse, and voice-activated software using speakers and a microphone, a VR / AR headset, etc.

[0115] Optionally, system 100 includes a viscous medium in which fiducial objects 150 are randomly distributed, e.g., ultrasound gel in which objects (e.g., spheres) are suspended, which are placed within the ultrasound gel and depicted in images captured by camera 114.

[0116] Optionally, system 100 includes an add-on 154 to the ultrasound probe, as described herein, that positions camera 114 in a fixed orientation relative to ultrasound transducer 112. Add-on 154 may include an illumination source that generates, for example, visible light, light of a specific color, ultraviolet light, and / or light for fluorescence activation. The light is selected according to the fixed orientation, e.g., the characteristics of the gel fiducial object, and provides an enhanced visual effect, e.g., fluorescence of the fiducial object, to depict the gel fiducial object in the camera image.

[0117] Optionally, the system 100 includes a calibration device 152 for calibrating the transformation mapping 122B, as described herein.

[0118] At 202, a viscous material is provided, at least a portion of which is fluid and includes a plurality of fiducial objects randomly distributed therein. The viscous material may be an ultrasound gel designed to be applied to the surface of a body, e.g., an individual's skin, with which an ultrasound transducer is placed in contact to capture 2D ultrasound images, such as Doppler or B-mode, that are used to reconstruct 3D images, as described herein.

[0119] The fiducial objects are randomly, i.e., distributed in 3D within the material and spaced apart by random distances. The fiducial objects have a size and / or contrast relative to the viscous material as depicted by the 2D camera image. The fiducial objects depicted in the camera image are used to calculate 3D coordinates that are assigned to pixels in the 2D ultrasound image to generate a reconstruction of a 3D ultrasound image depicting the interior of the body and / or the exterior of the scanned body segment.

[0120] The fiducial object may be selected so as not to interfere with the ultrasound energy transmitted through the ultrasound gel to capture the ultrasound image and / or to avoid artifacts caused by the presence of the fiducial object, for example, the fiducial object may be made small in size so as not to interfere with the transmitted ultrasound energy and / or may be made from an acoustic material that transmits ultrasound energy.

[0121] Because the fiducial object cannot be fixed at a specific location within the ultrasound gel, the fiducial object flows and / or otherwise changes position within the material in response to movement (e.g., shaking) of the material.

[0122] Exemplary sizes of the fiducial objects include about 0.5 to 1 millimeter (mm), or about 0.3 to 1.5 mm, or about 0.5 to 1.5, or about 0.3 to 0.7 mm, or other ranges.

[0123] Exemplary densities of fiducial objects include about 10-1000 per milliliter (mL), or about 5-50 per mL, or about 25-75 per mL, or other ranges.

[0124] Exemplary shapes of fiducial objects include spheres, squares, stars, snowflakes, cubes, and any other shape.

[0125] The fiducial object may be colored to contrast with the individual's skin so that the fiducial object is easily identifiable by code analyzing the camera image captured by the camera. For example, the fiducial object may be made of a fluorescent material, a material that enhances in response to ultraviolet light (UV), and / or has a bright color such as yellow, green-black, red, blue, or may be made in a variety of colors, etc.

[0126] Referring back to FIG. 4, FIG. 4 includes a schematic diagram of ultrasound gel 404 with fiducial objects 406 randomly distributed for capturing by an image camera for use in reconstructing a 3D image. The ultrasound gel 404 may be applied, for example, by squeezing a plastic bottle from a container. The ultrasound gel 404 may be spread onto the subject's skin and surrounding area above the tissue to be imaged in a standard manner. Optionally, the ultrasound gel 404 is spread over a larger area of ​​the body to generate a 3D image reconstruction of a tissue region larger than the area depicted by an individual 2D ultrasound image, e.g., organs such as the liver, kidneys, and / or vascular trees. Optionally, the ultrasound gel 404 is spread over an area large enough to be captured in the camera image. An ultrasound transducer moves over the ultrasound gel 404 to capture ultrasound images, and a camera, e.g., in an add-on connected to the ultrasound transducer, captures camera images of fiducial objects 406 randomly distributed within the gel 404, as described herein, and uses these camera images to reconstruct a 3D image.

[0127] Randomly distributed fiducial objects 406 provide multiple features across the camera image to calculate 3D coordinates to assign to 2D pixels, obtain 3D voxels, and reconstruct the 3D image. Having multiple features spread across the camera image can improve the accuracy of the 3D voxel locations.

[0128] The fiducial object 406 has a size and / or color characteristic selected to be depicted in a camera image captured by a camera when the ultrasound gel 406 is spread on the surface of the subject's skin. The fiducial object 406 may have a small size selected and / or be made from a selected acoustic material so as not to be depicted in an ultrasound image captured by an ultrasound transducer through the ultrasound gel 404. The size of the fiducial object 406 may be, for example, about 0.5 to 3 millimeters (mm), or about 0.5 to 1 mm, or about 1 to 2 mm, or other ranges. Optionally, the gel 404 is transparent, while the color characteristic is non-transparent, optionally of one or more different colors, e.g., yellow, red, green, black, and blue. Optionally, the fiducial object 406 is made of a material that provides an enhanced color characteristic in response to ultraviolet light and / or is made of a fluorescent material. The fiducial objects 406 may be of different shapes, such as spherical, square, triangular, diamond, or other shapes. The density of the fiducial objects 406 in the ultrasound gel 404 may be about 10-1000 per milliliter (mL), or about 5-50 per mL, or about 25-75 per mL, or about 10-100 per mL, or about 100-500 per mL, or about 300-700 per mL, or other ranges.

[0129] 2, at 204, ultrasound images depicting body tissue captured by an ultrasound transducer are obtained, the ultrasound images depicting common regions of body segments, such as blood vessels, e.g., femoral arteries, carotid arteries, aorta, or other body tissue, e.g., kidneys, liver, gallbladder, spleen, thyroid gland.

[0130] The ultrasound images may include Doppler 2D mode images and / or B-mode images or other 2D / 3D modes.

[0131] Body tissues (e.g., blood vessels) may contain moving fluids. In the case of blood vessels, the ultrasound image may further include Doppler data, also referred to herein as a Doppler image. The ultrasound image includes a region of the blood vessel and depicts blood flowing therein. The Doppler ultrasound image further includes a measurement of blood flow within the region of the vessel. The blood flow measurement may be an instantaneous blood flow measurement, such as a peak velocity, and / or a pattern of change in blood flow, such as velocity, over one or more cardiac cycles.

[0132] Optionally, for example when blood vessels are being imaged, Doppler and B-mode ultrasound images are taken alternately.

[0133] At 206, a 2D camera image is obtained, captured by a camera. The camera image is optionally a visible light image captured by a standard video camera and / or a still camera. The camera image depicts a feature pattern, optionally depicting a fiducial object within ultrasound gel or other viscous material. It is noted that other feature patterns may be used, such as temporary tattoos placed on the skin and / or features of the skin itself, such as hair, birthmarks, wrinkles, etc.

[0134] The camera is placed at a known fixed orientation relative to the ultrasound transducer. Optionally, the camera is installed as an add-on to the ultrasound transducer and is designed to be placed at a fixed orientation.

[0135] Optionally, the camera image and the ultrasound image are captured simultaneously or nearly simultaneously.

[0136] Optionally, the camera images and ultrasound images are captured as a pair, i.e., with synchronized and / or matching video frame rates. Alternatively, the camera images and ultrasound images are captured independently, for example, at different video frame rates.

[0137] Referring back to FIG. 5 , FIG. 5 is a schematic diagram depicting a camera 502 capturing camera images depicted in a field of view 504 of fiducial objects 506 randomly distributed within ultrasound gel 512 or other viscous material, and an ultrasound transducer 508 of an ultrasound probe 510 capturing ultrasound images through the gel. The camera images depicting the fiducial objects 506 randomly distributed within the ultrasound gel and the ultrasound images are used to calculate 3D voxels and reconstruct a 3D image, as described herein. The camera 502 may be located in an add-on 514 connected to the ultrasound probe 510, and as described herein, the add-on 512 is positioned to fix the angle of the camera 502 relative to the ultrasound transducer 508, e.g., relative to the view for capturing the ultrasound images, e.g., the angle of the view for capturing the camera images.

[0138] Referring back to FIG. 2 , at 208, 3D voxels are calculated having 3D coordinates in a world coordinate system. The world coordinate system represents the body segments of the imaged individual and specifies real-world distances and / or real-world locations, i.e., is distortion-free. The world coordinate system may be a 3D Euclidean space in which the body segments are located, isotopically scaled to real-world distances (e.g., 1:1 or other linear scale), and / or is in a non-deformed space. The 3D voxels are calculated by assigning 3D coordinates to pixels of the 2D ultrasound images and / or Doppler images using an external reference of the pose of the ultrasound transducer that captured the ultrasound images. The pose of the ultrasound transducer is calculated by analyzing the relative change in the position of a fiducial object in successive 2D camera images.

[0139] Calculating the camera pose in the camera coordinate system relative to the world coordinate system allows for the calculation of 3D coordinates for pixels of the ultrasound and / or Doppler images. The actual scale of structure and motion in world units may be calculated using additional information, such as the size of objects in the 2D camera images, such as the diameter of a sphere embedded in ultrasound gel, or by adding information from another sensor, such as an inertial measurement unit, or another camera. The camera pose may be calculated by analyzing the relative change in the position of a fiducial object in successive 2D camera images, for example, using optical flow, structure from motion (SfM), visual odometry, or visual simultaneous localization and mapping (Visual SLAM). A calibration mapping, e.g., a transformation matrix, may be used to map pixels of the ultrasound and / or Doppler images, expressed in the ultrasound coordinate system of the ultrasound transducer, to the camera coordinate system. The mapping may be based on a pre-calibrated relationship between the camera pose and the ultrasound transducer pose. Pixels of the ultrasound and / or Doppler images expressed in the camera coordinate system are mapped to 3D coordinates in the world coordinate system, and voxels are created by assigning pixel intensity values ​​of pixels of the ultrasound and / or Doppler images to corresponding 3D coordinates in the world coordinate system.

[0140] 6, which is a schematic diagram depicting exemplary x, y, and z axes 606 of an ultrasound transducer 602 for calculating 3D coordinates within a 3D space Vx,Vx,Vz module 608 of tissue 604 depicted in an ultrasound image, i.e., a volume of interest. The orientation of the ultrasound transducer 602 defined by the axes 606 and / or calculation of 3D voxels for the tissue 604 within the 3D space 608 is calculated based on a camera image captured by a camera 610 connected to an ultrasound probe 612 of the ultrasound transducer 602, e.g., housed in an add-on component 614, as described herein.

[0141] Referring back now to FIG. 2, three types of 3D images can be computed: 3D images that can be contained within a single 3D image, and / or that can be presented as different 3D images, and / or that can be presented as different overlays of the same 3D image.

[0142] Features 210-214 are for calculating a 3D image depicting blood flow within a blood vessel.

[0143] The features 216-222 are for segmenting the 3D image.

[0144] Features 224-226 are for calculating a 3D image depicting the body surface.

[0145] At 210, estimated (e.g., initial) blood flow rates are calculated and / or obtained for multiple locations within the vessel. Optionally, estimated (e.g., initial) blood flow rates are calculated and / or obtained throughout the vessel, e.g., along the length and / or diameter of the vessel.

[0146] Many or all of the (e.g., initial) estimated blood flow values ​​for different locations within the blood vessel are measured at non-90 degree angles between the long axis of the ultrasound transducer (e.g., with the ultrasound probe) and the vector indicating the direction of blood flow within the blood vessel. Measurements made at non-90 degree angles are erroneous because Doppler blood flow measurements are designed to be made at non-90 degree angles.

[0147] At 212, the blood flow values ​​measured at non-90 degree angles are corrected to account for the actual blood flow velocity vector along the artery.

[0148] Various approaches may be used to correct the measured blood flow to obtain the true blood flow. The first, second, and third approaches described below are for calculating flow velocity, e.g., meters / second. The fourth approach described below is for volumetric flow (e.g., ml / second). The fifth approach described below is for estimating the total blood flow velocity vector field:

[0149] The first approach involves identifying the oval boundary of the blood vessel. The oval shape can be identified, for example, by a neural network trained on a training dataset of ultrasound images labeled with oval boundaries, by matching to a template of oval boundaries, by applying a filter to the image to contrast edges, by using an edge shape process to find the boundary and calculate the shape of the boundary, and / or by other image processing approaches. A transformation from oval to circular is calculated, and this transformation is applied to the blood flow measurement to obtain an estimated actual blood flow.

[0150] In the second approach, an estimate of the vessel's longitudinal axis is calculated based on an aggregation of multiple 3D points acquired from each acquisition plane of each of multiple Doppler ultrasound images. For each current Doppler image, a local principal component analysis (PCA) is performed to calculate a current tangent to the vessel's longitudinal axis. The measured blood flow is corrected based on a projection of the angle between the measured blood flow and the normal to each acquisition plane that lies along the tangent.

[0151] In the third approach, for each 3D voxel, multiple initial blood flow estimates and the normal corresponding to the plane of the ultrasound transducer capturing each Doppler US image used to calculate each initial blood flow estimate are recorded. The actual blood flow at each voxel is estimated by minimizing the projection error between the recorded blood flow velocity vector and the assumed flow velocity vector.

[0152] In the fourth approach, initial volumetric flow rates are acquired in a vessel in a sagittal view at multiple cardiac cycle-specific locations. Standard approaches, such as Doppler, may be used to calculate the initial volumetric flow rates, but at angles other than 90 degrees. An initial average volumetric flow rate is calculated by averaging the initial volumetric flow rates across the cardiac cycle. A gamma value, representing a real number between -1 and 1, is calculated to correct the initial average volumetric flow rate and obtain the actual volumetric flow rate. An estimated actual blood flow rate can be obtained by correcting the initial measured blood flow rate, for example, based on the first, second, and / or third approaches described above. A cosine angle can be calculated from the reciprocal of the gamma value. The initial average volumetric flow rate is corrected, and the actual volumetric flow rate is obtained using the cosine angle and dot product between the estimated and actual blood flow rates.

[0153] In a fifth approach, also depicted with respect to FIG. 20, a 3D model of the actual blood vessel is calculated based on B-mode segmentation and a computational fluid dynamics model is used in conjunction with an aggregation of multiple 3D points acquired from each acquisition plane of each of multiple Doppler ultrasound images as boundary conditions to generate a total blood flow velocity vector field at each virtual point within the artery.

[0154] 7, a schematic diagram is provided depicting the flow, e.g., velocity, speed, and correction of blood 702 within an ultrasound image of a blood vessel 704 acquired at multiple orientations 708 by an ultrasound transducer 706. Graph 710 depicts measurements of blood flow 704 acquired at different orientations 708 of the transducer 706, corrected as described herein.

[0155] 2, at 214, a 3D image is reconstructed in a world coordinate system by aggregating a plurality of 3D voxels, each of which contains an estimated, i.e., corrected, blood flow. The 3D image depicts an anatomical image of the blood vessel and depicts blood flow at multiple locations within the vessel.

[0156] Optionally, a 3D image is reconstructed from the 3D voxels calculated from the B-mode ultrasound image, and an indication of estimated blood flow associated with each 3D voxel is calculated from the Doppler data.

[0157] Optionally, the 3D voxels of the reconstructed 3D image are visually coded, e.g., color-coded, according to the corresponding blood flow rate. For example, blood flow rates within a range indicating slow flow are colored one color, such as blue, blood flow rates within another range indicating fast flow are colored another color, such as red, and so on. Alternatively, or in addition, pixels of the Doppler ultrasound image indicating blood flow rate are visually coded (e.g., by color). The color pixels may be segmented. A 3D image may be reconstructed from 3D voxels corresponding to the segmented color pixels.

[0158] Optionally, the estimated blood flow rate for each 3D voxel is selected as the maximum value over the imaging time interval during which Doppler ultrasound images depicting the region within the blood vessel corresponding to the 3D voxel are captured. Alternatively, or in addition, multiple Doppler ultrasound images used to calculate the 3D voxels depicting the variation in blood flow rate over the imaging time interval can be captured and plotted as a link graph, with the reconstructed 3D image including, for each 3D voxel, an indication of the variation in blood flow rate over the imaging time interval. The reconstructed 3D image may be presented as a video over the imaging time interval by varying the blood flow rate values ​​corresponding to the 3D voxels over the imaging time interval according to the captured correlations.

[0159] Below is example pseudocode for generating a 3D image depicting blood flow velocity at multiple locations in a blood vessel: 3D reconstruction of the patient's body: USInit+transucerCamInit Record synchronized videos For every frame in transducer camera if frame is good then add to database For every frame in database featureExtraction ImageMatching FeatureMatching StructureFromMotion PrepareDenseScence ComputeDepthMapFilter Meshing MeshingFiltering Texturing end From StructureFromMotion extract Camera Poses For each camera pose in poses extractSegmentedPixels based on color and maximum value computeHomogenousTransform from the transducer corrdinate system to world system end For each colored point in PointCloud CorrectVelocityMeasurement End Display 4D data

[0160] Referring back now to FIG. 8 , a schematic diagram is provided depicting a reconstruction of a 3D image 802 of an artery 804 depicting a vascular pathology 806 (e.g., a stenosis) and showing the velocities of blood flow 812 and 814 through the artery 804. An ultrasound probe 808, including an ultrasound transducer and a camera as described herein, is manipulated over the surface of the leg above the artery 804, including the lesion 806, using, for example, a scanning motion in different orientations. The blood flow measured at different locations in different orientations is corrected as described herein. Multiple standard ultrasound images 810, including optional Doppler data, are captured along with camera images of a pattern, for example, a fiducial object in ultrasound gel applied over the entire surface of the leg. A 3D reconstruction 802 of the artery 804 is calculated from the ultrasound images, camera images, and optional Doppler data. The 3D reconstruction 802 depicts portions 812 corresponding to vascular pathology 806, identified by abnormally high blood flow velocities and / or abnormal indications, e.g., stenosis, calcification, high blood velocity, stents, and depicts other portions 814 that are normal portions of the artery, identified by normal blood velocity and / or normal shape.

[0161] Referring back now to Figure 2, a 3D image is reconstructed from the 3D voxels.

[0162] At 218, a multidimensional data set is calculated for each 3D voxel of the 3D image. Each multidimensional data set includes a mapping between each pose of the ultrasound transducer during capture of each ultrasound image depicting 3D coordinates within the body corresponding to the 3D voxel, and the intensity values ​​acquired at the 3D coordinates for each ultrasound image acquired from the various transducer poses. The poses of the ultrasound transducer can be represented, for example, by six values ​​representing six degrees of freedom. For example, for a 3D voxel with coordinates (x, y, z): (a1, b1, c1, d1, e1, f1) → intensity Q1, (a2, b2, c2, d2, e2, f2) → intensity Q2, and (a3, b3, c3, d3, e3, f3) → intensity Q3.

[0163] 9, a schematic diagram depicts the process of collecting data for a particular multi-dimensional dataset for a particular set of 3D coordinates corresponding to a particular voxel. Varying the orientation of the ultrasound transducer provides for collecting different pixel intensity values ​​for the same tissue region 902, visually depicted as multiple ultrasound imaging planes 904.

[0164] Referring back to FIG. 10 , a schematic diagram is provided depicting a representation of a multidimensional (optionally sparse) dataset 1002 of a particular voxel, where the distribution of acquired grayscale values ​​represents a sphere. Each location within the sphere indicates a 3D pose and / or orientation of the ultrasound transducer during capture of a respective 3D voxel corresponding to the multidimensional (optionally sparse) dataset. Each intensity value at each location on the sphere indicates a pixel intensity value of a 2D pixel in the 2D ultrasound image corresponding to the 3D coordinate of the 3D voxel calculated from the 2D pixel at the 3D pose and / or orientation of the ultrasound transducer corresponding to each location, and / or estimation by interpolating between values ​​can be performed by interpolation or a voting scheme that derives additional information from K-nearest neighbors. For example, the intensity at location 1004 may differ from the intensity at location 1006, e.g., be visually darker, indicating a difference in the distribution of 3D pose and / or orientation during capture of the particular 3D voxel, identifying the same tissue but with a different grayscale “fingerprint.” Diagram 1008 represents multiple multidimensional sparse data sets in a real-world coordinate system, one marked 1010 for clarity, with each multidimensional data set corresponding to one voxel used to create the 3D image. The multidimensional data sets may be clustered or segmented as described herein.

[0165] Referring now back to FIG. 2, at 220, the 3D voxels are clustered into clusters according to the distribution of the multidimensional sparse data set of 3D voxels.

[0166] Clustering may be performed based on manifold learning to find subspaces for better clustering of the data.

[0167] Clustering may be performed by clustering voxels according to similar patterns of variation in the ultrasound data captured for different angles and / or distances of the location corresponding to each voxel relative to the ultrasound transducer, with the voxels in each cluster indicating respective tissue types that produce respective similar patterns of variation in the ultrasound data captured at different angles and / or distances relative to the ultrasound transducer.

[0168] The clustering can be based on an unsupervised learning approach, for example, by minimizing the statistical distance between voxels within each cluster and maximizing the statistical distance between clusters. The statistical distance can be based on encodings that indicate patterns of variation in the captured ultrasound data with respect to different angles and / or distances of the location corresponding to each voxel relative to the ultrasound transducer.

[0169] 11 , a schematic diagram is provided depicting the reconstruction of a 3D image 1102 based on segmentation of a cluster of multidimensional datasets, including an artery 1104 segmented into different tissue types having vascular pathology 1106 (e.g., stenosis, calcification, stent). An ultrasound probe 1108, including an ultrasound transducer and a camera as described herein, is repeatedly steered at multiple different orientations over the skin above the lesion 1106. As described herein, collected data for the region containing the lesion 1106, including standard ultrasound images, optional Doppler data, and camera images of a pattern (e.g., a fiducial object in ultrasound gel spread across the surface of the leg), is aggregated to generate a higher-resolution 3D reconstruction 1112 of the reconstructed image 1102 and / or a higher-resolution ultrasound image 1110. The ultrasound transducer pose and corresponding pixel intensities are included in each multidimensional dataset, each for a 3D coordinate corresponding to a single voxel used to generate the 3D image. The higher-resolution ultrasound image 1110 may be segmented by clustering voxels according to similar multidimensional sparse data sets, i.e., according to similar distributions of pixel intensity values ​​in the ultrasound image between different orientations of the ultrasound transducer. Each segmented region may correspond to a respective tissue type. The 3D reconstruction 1102 depicts a higher-resolution portion 1112, segmented to include vascular pathology 1106, and may depict abnormal blood velocities and / or abnormal shapes (e.g., stenosis, calcification, high blood velocity, stents).

[0170] Referring back to Figure 2, at 222, the 3D image is segmented according to a subset of 3D voxel members of each cluster, with each segmentation including 3D voxels that are members of a respective cluster. Each segmentation may represent a particular tissue type.

[0171] The resulting clusters are segmented regions. Manifolds and clustering can be designed to consider the spatial and / or temporal distribution of each voxel and find clusters of voxels, optionally in an unsupervised manner, that assign each cluster to a respective tissue type. This process can improve the resolution of the ultrasound machine and / or extract the geometry and / or features of the relevant problem.

[0172] Below is an example pseudocode for computing the segmentation: For each pixel in the image Compute Pixel Location In Point Cloud Store Gray Scale Value for each voxel and colored Doppler with the transducer orientation and position as a 3D vector end Perform Voxelization

[0173] For voxel grouping based on manifold learning and / or other clustering where voxels with similar grayscale values ​​belong to the same group rather than to other groups, this involves computing group assignments based on distances between members, which may be computed based on some computed manifold or latent representation, density threshold, or setting up an a priori number of intended clusters, and / or may be formulated as a multi-objective optimization problem. For clustering, various approaches may be employed, such as connectivity-based clustering, centroid-based clustering, distribution-based clustering, density-based clustering, grid-based clustering, etc.

[0174] At 224, 3D coordinates are calculated for pixels in the 2D camera images that depict the surface (optionally the skin) of the body segment. The entire 2D camera image may depict the surface. A surface may be segmented if some of the camera images depict the surface and some do not, for example, depicting the background surrounding the body part. The 3D coordinates are calculated based on an analysis of relative changes in the position of the fiducial object in successive 2D camera images, optionally an analysis performed to calculate the 3D coordinates in the ultrasound image.

[0175] The 3D coordinates assigned to the pixels of the 2D camera image and the pixels of the 2D ultrasound image are in a common coordinate system. Voxels are defined in the 3D coordinate system by assigning 3D coordinates to the pixels of the 2D camera image and the 2D ultrasound image.

[0176] Optionally, the 2D camera images are processed to remove fiducial objects located on the surface of the body segments depicted therein that were used to calculate the 3D coordinates. Removal of the fiducial objects can occur after the calculation of the 3D coordinates, before and / or after assigning the 3D coordinates to pixels and calculating voxels.

[0177] At 226, a 3D image is reconstructed by aggregating voxels generated from pixels of the camera image and voxels generated from pixels of the ultrasound image in a common coordinate system, the 3D image depicting the surface of the body segment and tissue within the body segment positioned relative to the surface.

[0178] The common coordinate system may be a real-world coordinate system that depicts real-world distances and positions. The 3D image depicts the surface of the body segment and tissue within the body segment using real-world coordinates, distances, and relative positions. For example, a user may use the reconstructed 3D image to measure the depth of a blood vessel below the skin area. In another example, when using the 3D image to guide a catheter into deep tissue, a user may use the reconstructed 3D image to navigate by moving the catheter in a direction and / or distance to reach the target tissue as indicated in the real-world coordinate system of the reconstructed 3D image.

[0179] At 228, one or more of the blood flow and / or segmented 3D reconstructed images depicting the body surface are provided, for example, presented on a display, stored on a storage device, and / or transferred to another remote device.

[0180] A single 3D image representing blood flow, body surface, and segmentation may be presented. Alternatively, or in addition, a baseline 3D image may be presented, and the user may select one or more of the blood flow, surface, and segmentation to be presented as additional data, e.g., as an overlay, and / or integrated into the baseline 3D image. Alternatively, or in addition, multiple images may be presented, e.g., side-by-side on a display, each depicting different data. The multiple images may be selected from a representation of blood flow, a representation of the surface, and a representation of the segmentation.

[0181] Referring now back to FIG. 12, FIG. 12 is a schematic diagram illustrating an exemplary 3D image 1202 of the brachial artery calculated based on a camera image depicting a pattern on the surface of a subject's arm and an ultrasound image of the brachial artery, and an exemplary 3D image 1204 depicting the surface of the patient's body.

[0182] 13 , a schematic diagram is provided depicting an environmental scene for the generation of a 3D image 1302 computed from camera and ultrasound images, with fiducial objects 1304 randomly distributed and ultrasound gel positioned on the skin over an artery 1306 in a subject's leg 1308 for which the 3D image 1302 is being generated. An ultrasound probe 1310 includes an ultrasound transducer 1312 that captures ultrasound and / or Doppler images 1318 of the artery 1306, e.g., in B-mode or Doppler mode, and a camera 1314 connected to the probe 1310, optionally via an add-on component 1316 described herein, that captures camera images of the fiducial objects 1304 randomly distributed within the gel. The ultrasound and / or Doppler images 1318 may be captured by a standard probe 1310 connected to a standard ultrasound workstation 1320, operated using standard ultrasound approaches and / or operated by an unskilled user, as described herein. As described herein, a surface image 1322 of the surface of the leg 1308 can be extracted from the camera image. As described herein, a 3D image 1302 is calculated from data 1324 including 3D coordinates calculated for the ultrasound image using the camera image, optional Doppler data assigned to the 3D coordinates, and optional surface image 1322, as described herein. The 3D image 1302 depicts a portion of the artery 1306 that is larger than any single 2D artery, such as a stretch of the artery, the entire artery, etc., along with optional indications of blood flow within multiple locations of the artery 1306. The 3D image 1302 can include a 3D reconstruction of the surface image 1322 for the artery 1306 to enable visualization of the location of the artery 1306 below the skin surface within the leg 1308.

[0183] Referring now back to FIG. 2, at 230, the reconstructed 3D image can be used for other post-processing.

[0184] Optionally, the reconstructed 3D image is registered with another 3D anatomical image depicting the body segment. The 3D anatomical image is captured by a 3D imaging modality device, such as CT or MRI. Registration may be performed at least according to features depicted in the reconstructed 3D image and extracted from the surface of the body segment depicted in the 3D anatomical image. For example, registration may be performed between skin surfaces and also between internal blood vessels and / or other tissues.

[0185] Alternatively, or in addition, the reconstructed image is fed as input to another processing application, such as a surgical planning application, a neural network for diagnosing vascular pathologies.

[0186] At 232, one or more of the features described with reference to 204-230 may be repeated. The repetition may be performed by capturing additional ultrasound and camera images, collecting additional pixel and 3D coordinate data, and updating the reconstructed 3D image with the additional data to increase the resolution of the updated reconstructed 3D image. The image resolution may increase as the user moves the ultrasound probe to capture additional images.

[0187] Optionally, at 234, an add-on device is provided that is used to capture ultrasound images and camera images as described herein.

[0188] 14 , a schematic diagram is provided depicting an ultrasound probe 1404 for capturing camera images by a camera 1406 and an add-on 1402 to the ultrasound transducer 1406 for capturing ultrasound images used to reconstruct a 3D image. The add-on 1402 is positioned to provide a fixed angle 1410, e.g., 7.2 degrees, or some other value, between an axis 1412 of the ultrasound transducer 1408 for capturing ultrasound images and / or corresponding to the axis of the probe 1404, and an axis 1414 of the camera 1406 for capturing camera images. A 3D pose of the ultrasound transducer 1406 may be calculated based on the camera images captured by the camera 1406 according to the known angle 1410. 3D voxels corresponding to pixels of the ultrasound images used to reconstruct the 3D image may be calculated based on the calculated pose of the ultrasound transducer 1406 based on the known angle 1410, as described herein.

[0189] The add-on 1402 includes a connector component 1416 sized and shaped to fixedly connect to the probe 1404, e.g., secured by friction, e.g., the connector component 1416 includes an aperture sized to securely fit relative to the outer periphery of the probe 1404 so that when the aperture slides over the probe 1404, the probe 1404 is secured within the aperture by screws, adhesive, injection molding, etc., formed as an integral part of the probe 1404, and / or by a clip.

[0190] A camera housing 1418 is connected to the connector 1416. The camera housing 1418 is oriented on an axis 1412 corresponding to the axis of the probe 1404, at a predetermined angle 1410 relative to the connector component 1412. A camera 1406 is disposed within the camera housing 1418. The camera 1406 captures camera images at the predetermined angle 1410 relative to the ultrasound image captured by the ultrasound transducer 1408.

[0191] Optionally, the add-on 1402 includes an aperture sized and shaped for the cable 1420 of the camera 1406 and / or the cable 1422 of the probe 1404. The cable can be connected to a computer for transmitting the captured camera images and / or ultrasound images. The cable may also be for connecting to a power source. Alternatively, or additionally, if the camera 1406 and / or the probe 1404 transmit images using a wireless interface and / or use a battery, an aperture is not necessarily required.

[0192] Optionally, add-on 1402 includes a light source, located, for example, within camera housing 1418 and / or within camera 1406 and / or as a separate component. The light source may be configured to emit light at a selected frequency in response to a targeted fiducial object suspended in ultrasound gel, as described herein, for example, ultraviolet light and / or a selected frequency to enhance fiducial objects made of fluorescent material and / or a material that enhances in response to UV light. The light source may be positioned at a preselected angle 1410 corresponding to the angle 1410 of camera 1406 to transmit light through a portion of the ultrasound gel at the surface of the body depicted in a camera image captured by camera 1406.

[0193] Referring now back to Figure 15, Figure 15 is a schematic diagram depicting an exemplary add-on 1502 to an ultrasound probe including a connector component 1516 sized and shaped to connect to the ultrasound probe and a camera housing 1518 oriented at an angle relative to the long axis of the connector component 1516, which corresponds to the long axis of the probe's ultrasound transducer. Schematic diagram 1550 is a left front view of the add-on 1502. Schematic diagram 1552 is a right rear view of the add-on 1502.

[0194] The connector component 1516 includes a channel 1554 designed to slide over the handle of the probe. A plurality of screw apertures 1556 are designed to receive screws to secure the add-on 1502 in place relative to the probe. It is noted that other securing mechanisms may be used, as described herein.

[0195] The camera housing 1518 includes a view aperture 1558 through which a lens of a camera located within the camera housing 1518 captures camera images. The camera housing 1518 may include a camera cable aperture 1560 sized and shaped to position a cable connected to the camera. The cable may be connected to a computer to receive the camera images. Alternatively, the cable may be for connecting to a power source. Note that if a wireless interface and / or a battery is used to transmit images, the cable and / or aperture 1560 are not necessary.

[0196] Referring back now to FIG. 16, FIG. 16 is a schematic diagram of another exemplary implementation of an add-on 1602 to an ultrasound probe 1604. The add-on 1602 includes two camera housings 1618A-B, each containing a camera 1606A-B, i.e., camera 1606A in housing 1618A and camera 1606B in housing 1618B. As described herein, the camera housings 1618A-B are connected to a connector component 1616 that securely connects the add-on 1602 to the ultrasound probe 1604. When each camera housing 1618A-B is positioned at a predetermined angle relative to the longitudinal axis of the connector component, it secures the cameras 1606A-B at a predetermined angle relative to the ultrasound probe 1604 and / or to ultrasound images captured by the ultrasound transducer 1608 of the probe 1604, as described herein. The predetermined angle for both cameras 1606A-B may be the same, or each camera 1606A-B may be placed at a different predetermined angle.

[0197] Optionally, the camera housings 1618A-B and corresponding cameras 1606A-B are separated by approximately 90 degrees perpendicular to the long axis of the connector component 1616 or some other value. That is, the radial distance relative to the long axis of the connector component 1616 is approximately 90 degrees. The 90-degree angle may be selected to capture camera images of the same area of ​​the subject's body surface between sagittal or longitudinal orientations of the ultrasound transducer 1608. Other angles may also be used, in which case the known angle is used to calculate the relationship between the two images. For example, in a longitudinal examination, the long axis of the ultrasound transducer 1608 is positioned parallel to the long axis of the tissue, e.g., along the length of a blood vessel, and the camera 1606B is positioned to capture camera images of fiducial objects randomly distributed in a viscous material (e.g., ultrasound gel) located on the body surface above the blood vessel. In a sagittal scan, the long axis of the ultrasound transducer 1608 is positioned at an angle to the long axis of the tissue, for example along the diameter of a blood vessel, and camera 1606A is positioned to capture camera images of the same fiducial objects randomly distributed in a viscous material (e.g., ultrasound gel) present at the same location on the body surface as captured by camera 1606B during a longitudinal scan. Capturing fiducial objects randomly distributed in the viscous material during different orientations of the ultrasound transducer 1608 allows for the reconstruction of a 3D image of the tissue (e.g., blood vessels). These fiducial objects may add additional information for calculating the actual scale of the transducer's structure and motion in world units. Optionally, add-on 1602 includes an inertial measurement unit (IMU) that can be used as an alternative or in addition to analyzing relative changes in the position of the fiducial object in subsequent camera images to calculate the pose of the ultrasound transducer and / or the pose of the camera, adding additional information and allowing the actual scale of the transducer structure and motion in world units to be calculated.

[0198] Referring back now to FIGS. 21A-B, schematic diagrams of another exemplary implementation of an add-on 2102 to an ultrasound probe 2104 are depicted. FIG. 21A depicts a front view of the add-on 2102. FIG. 21B depicts a rear view of the add-on 2102. The add-on 2102 includes two camera housings 2118A-B, each containing a camera 2106A-B. That is, camera 2106A is included in housing 2118A, and camera 2106B is included in housing 2118B. The camera housings 2118A-B are connected to an ergonomic holder component 2116 that securely connects the add-on 2102 to the ultrasound probe 2104, as described herein. For example, the ultrasound probe 2104 slides within a compartment formed by the add-on 2102, which has an interior size and / or shape that corresponds to the exterior size and / or shape of the ultrasound probe 2104. The ultrasound probe 2104 can be held in place, for example, by clicking into place with a clip that applies pressure, by friction from the interior walls of the compartment, by a strap, etc. The ergonomic holder 2116 can include a grip 2150 designed to be held against the palm of a user's hand, and a ring or trigger-like element 2152 designed to support the user's index finger. The ergonomic holder 2116 is designed to provide the user with an improved grip, allowing for greater control and / or finer movements during an ultrasound scan.

[0199] The camera housings 2118A-B, when each positioned at a predetermined angle relative to the longitudinal axis of the connector component as described herein, secure the cameras 2106A-B at a predetermined angle relative to the ultrasound probe 2104 and / or relative to the ultrasound image captured by the ultrasound transducer 2108 of the probe 2104. The predetermined angle for both cameras 2106A-B may be the same, or each camera 2106A-B may be positioned at a different predetermined angle. The camera 2106B includes a detachable lens 2120 and can be separated from the camera 2106B itself, making it easy for a user to use different lenses for particular types of body scans.

[0200] Optionally, the camera housings 2118A-B and corresponding cameras 2106A-B are separated by approximately 90 degrees or other value perpendicular to the long axis of the connector component 2116. That is, the radial distance relative to the long axis of the connector component 2116 is approximately 90 degrees. The 90-degree angle may be selected to capture camera images of the same region of the subject's body surface during sagittal or longitudinal orientation of the ultrasound transducer 2108. Other angles (e.g., approximately 70 degrees, 80 degrees, 100 degrees, 110 degrees, or other values) may also be used, in which case the known angle is used to calculate the relationship between the two images. For example, in a longitudinal examination, the long axis of the ultrasound transducer 2108 is positioned parallel to the long axis of the tissue, e.g., along the length of a blood vessel, and the camera 2106B is positioned to capture camera images of fiducial objects randomly distributed in a viscous material (e.g., ultrasound gel) located on the body surface above the vessel. In a sagittal scan, the long axis of the ultrasound transducer 2108 is positioned at an angle to the long axis of the tissue, for example along the diameter of a blood vessel, and the camera 2106A is positioned to capture camera images of the same fiducial objects randomly distributed in a viscous material (e.g., ultrasound gel) present at the same location on the body surface as captured by the camera 2106B during a longitudinal scan. Capturing fiducial objects randomly distributed in the viscous material during different orientations of the ultrasound transducer 2108 allows for the reconstruction of a 3D image of the tissue (e.g., blood vessels). These fiducial objects may add additional information to calculate the actual scale of the transducer's structure and motion in world units. Optionally, add-on 2102 includes an inertial measurement unit (IMU) (not shown, as it may be an internal component) which can be used as an alternative or in addition to analyzing the relative changes in the position of the fiducial object in subsequent camera images to calculate the pose of the ultrasound transducer and / or the pose of the camera, adding additional information and allowing the actual scale of the transducer structure and movement in world units to be calculated.

[0201] Optionally, at 236, the add-on device is calibrated. Calibration may be performed to calculate one or more calibration transformations that map pixels of the ultrasound coordinate system to / from the camera coordinate system. The calibration mapping may map pixels of the ultrasound image and / or Doppler data and / or B-mode data expressed in the ultrasound coordinate system to the camera coordinate system, or vice versa. The mapping is based on a calibration relationship between the pose of the camera and the pose of the ultrasound transducer. The calibration mapping is used to map pixels of the Doppler and / or B-mode ultrasound image expressed in the camera coordinate system to 3D coordinates in the camera coordinate system.

[0202] Calibration can be performed using a specially designed device that includes a box with ultrasound gel, randomly spaced strings, and a nearby checkerboard pattern, or other patterns can be used. When an ultrasound transducer scans a silicone box, two images appear simultaneously: an ultrasound image of the cross-section of the string inside the box and a camera image of the checkerboard. The two images are related to each other, and the camera pose from the checkerboard corresponds to different transducer poses from known silicone-immersed strings. The relationship between the two images is determined by a calibration process that collects several images from different checkerboard positions, and the corresponding US images are analyzed and compared to ground truth, i.e., angles and distances imposed by 3D computer modeling. The outcome of the process provides a homogeneous calibration transformation for converting the camera image pose to the transducer pose.

[0203] Calibration may be used based on random versions of strings with no defined geometry, and the use of checkerboards. In another example, another calibration device is based on a simple box with ultrasound gel, holes, and April tags. The camera pose is based on an estimation protocol that determines the camera pose relative to each detected April tag that can be found in the captured image. At the same time, the transducer captures both the top and bottom of the box, which can be used to determine the homogeneous transformation between the camera and the transducer.

[0204] 17, a schematic diagram is provided depicting an example implementation of a calibration device 1752, corresponding to, for example, the calibration device 152 described with reference to Figure 1. The calibration device 1752 includes an ultrasound section 1760 having a predetermined first ground truth pattern 1762 and a camera section 164 having a predetermined second ground truth pattern 1766.

[0205] The ultrasonic section 1760 can be implemented as a box with an open top. The ground truth pattern 1762 can be a 3D pattern with predetermined 3D coordinates immersed in an ultrasound-transparent medium such as transparent silicone. The ground truth pattern 1762 can be, for example, poles arranged at various angles, and / or different objects with different shapes, slopes with matching protrusions, and / or angles, such as boxes, spheres, pyramids, and rectangles, at different heights and / or orientations, and / or arranged in a random pattern. Alternatively, or in addition, the ground truth pattern 1762 can be a 2D pattern, such as a checkerboard pattern and / or a wire pattern. The ground truth pattern 1766 can be 2D and / or 3D, such as a checkerboard pattern. The shapes of the patterns 1762 and 1766 are predefined; for example, the pattern 1762 is printed using a 3D printer, and the checkerboard pattern 1766 is printed using a 2D printer.

[0206] The camera section 1762 may be positioned proximate to the ultrasound section 1760, and may optionally be separated by a transmission positioning region 1768. The arrangement of the camera section 1762, ultrasound section 1760, and transducer positioning region 1768 is selected to simultaneously or nearly simultaneously capture an ultrasound image depicting a predetermined first 3D ground truth pattern 1762 by an ultrasound transducer located in the transducer positioning region 1768 and a camera image depicting a predetermined second ground truth pattern 1766 by a camera located in an add-on to the ultrasound probe located on the probe connected to the ultrasound transducer located in the transducer positioning region 1768.

[0207] The following is exemplary pseudocode for calibration using calibration device 1752: 1.Init Both ultrasound transducer and camera 2.Collect several checkerboard images and B-Mode images from different orientations and positions 3.Locate strings position in transducer image and estimate camera pose in world coordinate system relative the checkerboard pattern 4.Find homogenous transformation between the transducer coordinate system and the camera by minimizing the relative distance between the actual position of the recorded strings positions and the 3D known positions. 5.Return computed transformation

[0208] 18 , a schematic diagram is provided depicting another exemplary implementation of another calibration device 1852, corresponding to, for example, the calibration device 152 described with reference to FIG. 1 . The calibration device 1852 includes a single compartment 1854 that includes a predetermined common 2D marker and / or 3D truth pattern 1856 disposed on top of the single compartment 1854. The compartment 1854 may be shaped, for example, to have a circular surface. The compartment 1854 may be shaped, for example, as a hemisphere, a cylinder, or other shape. The pattern 1856 may be located outside and / or around (e.g., near) the spherical boundary (e.g., perimeter) of the compartment 1854, for example, the indicators of the pattern 1856 are evenly spaced along the outer perimeter of the compartment 1854. The compartment 1854 includes (e.g., is filled with) an ultrasound-transparent echography medium, for example, a clear silicone or an echography gel. When an ultrasound transducer is positioned at the top (optionally approximately in the center) of compartment 1854, the camera captures a camera image of pattern 1856, and the ultrasound transducer captures an ultrasound image (e.g., image 1890 is an example) of the interior of compartment 1854 from top surface 1860 to bottom surface 1862. Ultrasound may pass through top surface 1860 without significant attenuation and may also pass through hole 1864. Hole 1864 may be located, for example, on the top, bottom, and / or within (e.g., in the center) of compartment 1854. Pattern 1856 may be made of a material with selected properties, such as being enhanced under ultraviolet (UV) light and / or fluorescence, and may be made of a high-contrast printed calibration pattern.

[0209] The following is exemplary pseudocode for calibration using the calibration device 1852: 1.Init Both transducer camera and the camera 2.Collect simultaneously several camera images of the markers and the transducer images from different orientations and positions 3.For each transducer image locate floor (UV pixels) position in US image, and for each corresponding camera image locate each marker position in world units 4.For each detected tag and for each UV floor pixel compute the estimated floor depth based on homogeneous transformation from UV image to camera pose in the tag coordinates system 5.Find a homogeneous transformation that minimizes the relative distance between the actual known depth of the floor from the top surface and the estimated depth based on the above transformation (US image to world coordinates). 6.Return computed transform

[0210] Optionally, the IMU is calibrated using the IMU measured and camera recorded orientation simultaneously, with gravity as the Z component of the IMU coordinate system.

[0211] Referring back now to FIG. 3, at 302, a viscous material, which is a fluid (optionally an ultrasound gel) at least a portion of which is randomly distributed in its volume and contains fiducial objects spaced at random distances, is applied to the surface of an individual's body segment at locations corresponding to blood vessels of interest.

[0212] If the material is an ultrasound gel, the gel may be applied to a larger area of ​​skin and used in standard methods to obtain ultrasound images.

[0213] At 304, an ultrasound transducer probe with add-on components including one or more cameras is manipulated along the body surface, optionally simultaneously capturing ultrasound images of blood vessels within the body and camera images depicting the surface of the body segment and depicting the fiducial object.

[0214] At 306, one or more 3D images are reconstructed from the ultrasound images and an analysis of the camera images is presented on a display.

[0215] Optionally, the 3D image depicts a visual indication of blood flow at multiple regions within the blood vessel. Alternatively, or in addition, the 3D image is segmented according to tissue type. Alternatively, or in addition, the 3D image depicts the skin and blood vessels of the body segment within a common real-world coordinate system representing real-world coordinates, and diagnosing a vascular condition based on reconstruction of the 3D image of the blood vessels relative to the surface of the body segment is further based on blood flow in the blood vessels depicted by the 3D image.

[0216] At 308, the 3D images are analyzed. This analysis may be performed manually by visual inspection and / or measurement, and / or may be performed automatically by feeding the 3D images into a machine learning process (e.g., a neural network) that generates an outcome that indicates a diagnosis.

[0217] In 310, vascular pathologies are diagnosed based on analysis of 3D images, such as stenosis of blood vessels.

[0218] At 312, the vascular condition is treated during open surgery and / or catheter procedures. Introduction, navigation, and / or treatment may be guided by the 3D image, for example, according to distances and / or relative positions described by a real-world coordinate system in which the skin and blood vessels in the 3D image are presented. Exemplary treatments include catheter-delivered stents, balloon dilation, ablation, drug injection, and procedural resection and / or repair.

[0219] At 314, the effectiveness of the vascular treatment may be evaluated based on 3D images acquired after the treatment, for example, by analyzing a segmentation of the 3D image to determine whether plaque has been completely removed from the artery and / or whether blood flow patterns have returned to normal.

[0220] At 316, one or more features described with reference to 302-316 are repeated.

[0221] Optionally, the repetition is performed by, for example, re-manipulating the probe on the surface of the body to simultaneously capture ultrasound images of the blood vessel depicting the treated vascular condition, analyzing the treated vascular condition in another 3D reconstruction of the blood vessel created from the ultrasound images and camera images captured during the re-manipulation of the probe, and re-treating the treated vascular condition if the treated vascular condition is determined to require another treatment procedure based on the another 3D reconstruction.

[0222] Alternatively, or in addition, the repetition is performed by repeatedly steering the ultrasound transducer within a small area of ​​the surface corresponding to the vascular pathology in the axial and / or longitudinal orientation of the ultrasound transducer to capture multiple ultrasound images of the vascular pathology at different angles and distances, where the resolution of the 3D image, optionally the resolution of the 3D image of the vascular pathology, increases with increasing number of steering of the ultrasound transducer over the small area.

[0223] Alternatively, or in addition, the repetitions may be based on code that analyzes the data to detect diseased segments (e.g., slow blood velocity, stents, high calcium content). In response to this detection, an indication (e.g., a beep, an audio message, a text message presented on a display, and / or a video) may be provided to the user to repeatedly scan the segment to increase the data and obtain higher resolution imaging of the diseased segment. Optionally, the code may monitor the amount of data acquired to detect when enough data has been collected to generate a targeted high resolution 3D image, and may provide another indication to the user to stop the ultrasound scan and / or move on to the next body region to be scanned.

[0224] Referring back to FIG. 19 , a schematic diagram 1902 is provided depicting a standard vascular treatment using existing approaches, as well as a schematic diagram 1904 depicting a vascular treatment using 3D images created from ultrasound and camera images. As depicted in schematic diagram 1902, the standard process, for example, for treating a vascular condition in the leg that may result in insufficient blood flow to the leg, may include a standard Doppler of the leg. A CTA may be performed, which may require a two-day hospital stay. An angiography procedure may be performed one to two weeks later to treat the vascular condition, which may require a three-day hospital stay. The results of the procedure may be obtained with a one to two month follow-up. In contrast, schematic diagram 1904 depicts a method by which real-time 3D images of the leg's blood vessels may be created before, during, and after treatment, e.g., to guide a catheter into the vessel, and after treatment, e.g., to evaluate the effectiveness of the vascular treatment, potentially allowing for a same-day and / or two-day hospital stay.

[0225] FIG. 20 depicts a process for calculating an estimated blood flow velocity field 2004 by using a computational fluid dynamics simulation 2010 with an a priori 3D arterial model 2006 integrated as boundary conditions 2006 for blood flow measurements, which can be measured by the Doppler mode of an ultrasound system from various positions and, in the same coordinate system and same volume 2002, by corresponding positions in a corresponding known 3D arterial model 2006.

[0226] The description of various embodiments of the present invention has been presented for purposes of illustration and is not intended to be exhaustive or limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles of the embodiments, practical applications or technical improvements to technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0227] It is anticipated that many related viscous materials and fiducial objects will be developed during the life of the patent that matures from this application, and the scope of the terms viscous materials and fiducial objects is intended to include all such new technologies a priori.

[0228] As used herein, the term "about" means ±10%.

[0229] The terms "comprises," "comprising," "includes," "including," "having," and their cognates mean "including, but not limited to." This term encompasses the terms "consisting of" and "consisting essentially of."

[0230] The phrase "consisting essentially of" means that the composition or method may include additional components and / or steps, but only if the additional components and / or steps do not materially alter the basic and novel characteristics of the claimed composition or method.

[0231] As used herein, the singular forms "a," "an," and "the" include plural referents unless the context clearly indicates otherwise. For example, the terms "a compound" or "at least one compound" can include multiple compounds, including mixtures thereof.

[0232] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments and / or to exclude the incorporation of features from other embodiments.

[0233] As used herein, the word "optionally" is used to mean "provided in some embodiments and not provided in other embodiments." Any particular embodiment of the present invention may include multiple "optional" features unless such features are inconsistent.

[0234] Throughout this application, various embodiments of the present invention may be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the present invention. Accordingly, the description of a range should be considered to have specifically disclosed not only each individual numerical value within that range, but also all possible subranges. For example, a description of a range such as "1 to 6" should be considered to have specifically disclosed subranges such as "1 to 3," "1 to 4," "1 to 5," "2 to 4," "2 to 6," "3 to 6," etc., as well as each individual numerical value within that range, e.g., 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

[0235] Whenever a numerical range is given herein, it is meant to include any recited number (fractional or integer) within the range given. The phrases "ranging / ranges between" a first and second designator number and "ranging / ranges from" a first designator number "to" a second designator number are used interchangeably herein and are meant to include the first and second designators and all fractional and integer numbers therebetween.

[0236] It is understood that certain features of the invention that are, for clarity, described in the context of separate embodiments, can also be provided in combination in a single embodiment. Conversely, various features of the invention that are, for brevity, described in the context of a single embodiment, can also be provided separately or in any suitable subcombination, or as preferred in any other described embodiment of the invention. Particular features described in the context of various embodiments should not be considered essential features of those embodiments, except to the extent that the embodiment cannot function without those elements.

[0237] While the present invention has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications, and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications, and variations that fall within the spirit and broad scope of the appended claims.

[0238] It is the intention of the applicants (applicants) that all publications, patents, and patent applications mentioned herein be incorporated by reference in their entireties, as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference herein. Furthermore, citation or identification of any reference in this application should not be construed as an admission that such reference is available as prior art to the present invention. Section headings, if used, should not be construed as necessarily limiting. Additionally, any priority documents of this application are hereby incorporated by reference in their entireties.

Claims

1. 1. A computer-implemented method for reconstructing a 3D image depicting blood flow within a blood vessel of an individual, comprising: acquiring Doppler ultrasound images through a blood vessel with blood flowing therethrough and depicting a measurement of blood flow within a region of the blood vessel, and 2D camera images captured by a camera depicting a plurality of fiducial objects randomly distributed within a viscous material and separated by random distances on a surface of the individual's body segment, the Doppler ultrasound images comprising at least one Doppler ultrasound image and the 2D camera images comprising at least one 2D camera image; calculating 3D coordinates in a world coordinate system for each pixel of the Doppler ultrasound image using an external reference of ultrasound transducer pose calculated by analyzing relative changes in positions of the plurality of fiducial objects in successive 2D camera images; calculating a respective estimated blood flow rate for each of a plurality of pixels of the Doppler ultrasound image at a plurality of locations within the blood vessel; reconstructing a 3D image including respective estimated blood flow volumes from 3D voxels calculated from the 3D coordinates of the pixels of the Doppler ultrasound image, the 3D image depicting an anatomical image of the blood vessel and depicting blood flow; The computer-implemented method includes:

2. 2. The computer-implemented method of claim 1, wherein the 3D coordinates are further calculated by correcting to a real scale relative to the world coordinate system by using a size of the fiducial object and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera.

3. The computer-implemented method of claim 1 , wherein the 3D image further comprises voxels computed from a 3D point cloud or mesh depicting the surface of the body segment of the individual.

4. Calculating the 3D coordinates of each pixel of the Doppler ultrasound image comprises: calculating a pose of the camera in a camera coordinate system relative to the world coordinate system by analyzing relative changes in positions of the plurality of fiducial objects in the successive 2D camera images; applying a calibration mapping to map pixels of the ultrasound image, expressed in an ultrasound coordinate system, to the camera coordinate system, the calibration mapping being based on a predetermined transformation between the pose of the camera and the pose of an ultrasound transducer; mapping the pixels of the ultrasound image expressed in the camera coordinate system to the 3D coordinates in the world coordinate system; The computer-implemented method of claim 1 , comprising:

5. 2. The computer-implemented method of claim 1, wherein calculating each estimated blood flow includes calculating an estimated blood flow from the Doppler ultrasound image captured by the ultrasound transducer positioned at an angle relative to a vector indicating the direction of the blood vessel for which the blood flow is measured, and further includes correcting the estimated blood flow for the angle to obtain an estimated actual blood flow.

6. 6. The computer-implemented method of claim 5, wherein the correcting comprises identifying an elliptical boundary of the blood vessel, calculating a transformation from the elliptical shape to a circle, and applying the transformation to project the measured blood flow velocity vector to obtain the actual blood flow.

7. 6. The computer-implemented method of claim 5, further comprising: calculating an estimate of a longitudinal axis of the blood vessel based on an aggregation of multiple 3D points acquired from respective acquisition planes of multiple Doppler ultrasound images; and calculating a current tangent to the longitudinal axis; and correcting the blood flow measurement based on an angle between the tangent and a normal to the respective acquisition plane indicated by the tangent.

8. 6. The computer-implemented method of claim 5, further comprising: calculating an estimate of blood flow within a pre-calculated 3D arterial object; and using the computational fluid dynamics simulation and 3D points acquired from each acquisition plane of the plurality of Doppler ultrasound images to calculate the estimate of blood flow within the 3D arterial object.

9. storing, for each 3D voxel, a plurality of initial estimated blood flow values ​​and corresponding normals to a plane of the ultrasound transducer at which each Doppler US image used to calculate the respective initial estimated blood flow values ​​was captured; estimating actual blood flow at each voxel based on minimizing the projection error between the recorded blood flow velocity vector and the assumed flow velocity vector; The computer-implemented method of claim 5 further comprising:

10. acquiring volumetric flow in the blood vessel on sagittal views at specific locations for a plurality of cardiac cycles and calculating an average volumetric flow over the plurality of cardiac cycles; calculating a gamma value, which represents a real number within the range of −1 to 1, to correct the mean volumetric flow rate and the estimated actual blood flow rate; calculating a cosine angle by using the inverse of the gamma value; correcting the mean volumetric flow using the cosine angle and a dot product between the estimated blood flow and the actual blood flow; The computer-implemented method of claim 9 further comprising:

11. (i) alternating between acquiring the Doppler ultrasound image and the 2D camera image, and (ii) acquiring a second image comprising a B-mode ultrasound image and a 2D camera image; calculating 3D coordinates of each pixel of the B-mode ultrasound image in the world coordinate system; reconstructing the 3D image includes reconstructing the 3D image from 3D voxels in the world coordinate system calculated by aggregating the 3D coordinates of the pixels of the B-mode ultrasound image and the Doppler ultrasound image, each including an estimated blood flow rate; 2. The computer-implemented method of claim 1, wherein the 3D image depicts the anatomical image of the blood vessel created from the aggregation of the 3D voxels obtained from the pixels in the B-mode and depicts the blood flow associated with the 3D voxels obtained from the pixels in the B-mode.

12. 2. The computer-implemented method of claim 1, wherein each estimated blood flow is depicted as a color coding of the 3D voxels of the 3D image corresponding to the multiple locations within the blood vessel, and pixels of the Doppler ultrasound image are color coded to indicate blood flow, further comprising segmenting the color pixels, and the 3D image is reconstructed from 3D voxels corresponding to the segmented color pixels.

13. 2. The computer-implemented method of claim 1, wherein the estimated blood flow for each of the 3D voxels is selected as a maximum value over an imaging time interval during which the Doppler ultrasound image depicting a region within the blood vessel corresponding to the 3D voxel is captured.

14. 2. The computer-implemented method of claim 1, wherein the multiple Doppler ultrasound images used to calculate the 3D voxels are captured over an imaging time interval depicting variations in blood flow, and the reconstructed 3D images include, for the 3D voxels, respective indications of variations in blood flow over the imaging time interval.

15. 13. The computer-implemented method of claim 12, wherein the reconstructed 3D image is presented as a video over an imaging time interval by varying the indication of blood flow corresponding to the 3D voxels over the imaging time interval.

16. 1. A computer-implemented method for reconstructing a surface and / or interior 3D image of a body segment of an individual, comprising: obtaining 2D ultrasound images depicting tissue within the body segment of the individual, and 2D camera images captured by a camera depicting a plurality of fiducial objects randomly distributed in 3D within a viscous material and spaced by random distances on a surface of the body segment and depicting the surface of the body segment; calculating, in a common coordinate system, 3D coordinates of each pixel of the 2D camera images and 3D coordinates of each pixel of the 2D ultrasound images based on an analysis of the relative changes in position of the plurality of fiducial objects in the successive 2D camera images; reconstructing a 3D image from 3D voxels in the common coordinate system calculated by aggregating the 3D coordinates of the 2D camera image and the 2D ultrasound image; Including, the 3D image depicts the surface of the body segment and tissue within the body segment that is positioned relative to the surface of the patient; The computer-implemented method, wherein the reconstructed 3D images include at least one reconstructed 3D image.

17. 17. The computer-implemented method of claim 16, wherein the at least one reconstructed 3D image with additional information layers including at least one of Doppler and B-mode depicts tissues selected in any compatible ultrasound organ scanning procedure, including at least one of blood vessels, organs, joints, bones, cartilage, non-blood-filled spaces, liver, gallbladder, and thyroid, and fuses the tissues into one whole organ or part of an organ.

18. 17. The computer-implemented method of claim 16, further comprising correcting the 3D coordinates to a real scale relative to the world coordinate system by using a size of the fiducial object and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera.

19. 17. The computer-implemented method of claim 16, wherein the common coordinate system is a real-world coordinate system that depicts real-world distances and positions, and the 3D image depicts the surfaces of the body segments and tissue within the body segments using distances and relative positions in real-world coordinates.

20. Calculating in the common coordinate system calculating the pose of the camera in a camera coordinate system relative to the common coordinate system by analyzing the relative changes in position of the plurality of fiducial objects in the successive 2D camera images, and correcting for actual scale relative to the world coordinate system by using the size of the fiducial objects and / or by adding information from other sensors selected from the group consisting of an inertial measurement unit and another camera; mapping pixels of the 2D camera images depicting the surface of the body to 3D coordinates within the common coordinate system; applying a calibration transformation to map pixels of the 2D ultrasound image expressed in an ultrasound coordinate system to the camera coordinate system, the calibration mapping being based on a predetermined relationship between the pose of the camera and a pose of an ultrasound transducer capturing the 2D ultrasound image; mapping the pixels of the 2D ultrasound image expressed in the camera coordinate system to 3D coordinates in the common coordinate system; 17. The computer-implemented method of claim 16, comprising:

Images