Smart measurement assistance for ultrasound imaging and associated devices, systems, and methods

Abstract

Ultrasound image devices, systems, and methods are provided. An ultrasound imaging system includes a processor circuit in communication with an ultrasound transducer array, the processor circuit configured to receive a set of images of a three-dimensional (3D) volume of an anatomical structure of a patient including an anatomical feature from the ultrasound transducer array; obtain first measurement data of the anatomical feature in a first image of the set of images; generate second measurement data for the anatomical feature in one or more images of the set of images by propagating the first measurement data from the first image to the one or more images; and output the second measurement data for the anatomical feature to a display in communication with the processor circuit.

Description

Technical Field

[0001] This disclosure relates generally to ultrasound imaging, and more specifically to the automated measurement of anatomical features provided from ultrasound images. Background Technology

[0002] Ultrasound imaging systems are widely used in medical imaging. For example, a medical ultrasound system may include an ultrasound transducer probe coupled to a processing system and one or more display devices. The ultrasound transducer probe may include an array of ultrasound transducer elements that emit sound waves into the patient's body and record sound waves reflected from internal anatomical structures within the patient's body, which may include tissues, blood vessels, and internal organs. The emission and / or reception of reflected sound waves or echo responses may be performed by the same group of ultrasound transducer elements or different groups of ultrasound transducer elements. The processing system may apply beamforming, signal processing, and / or imaging processing to the received echo responses to create images of the patient's internal anatomy. The images may be presented to the clinician in the form of a brightness mode (B-mode) image, where each pixel of the image is represented by a brightness level or intensity level corresponding to the echo intensity.

[0003] Ultrasound imaging can be used for diagnostic examinations, interventions, and / or procedures. Furthermore, it can be used as a quantitative tool for measuring anatomical features. For example, during obstetric examinations, ultrasound imaging can be used to capture images of the fetus carried by the pregnant woman, and clinicians can assess fetal growth based on measurements of the fetal head or other parts of the fetus from the ultrasound images. Alternatively, during cardiac examinations, ultrasound imaging can be used to capture images of the patient's heart, and clinicians can perform quantitative cardiac measurements based on the ultrasound images.

[0004] To perform measurements based on ultrasound images, clinicians can place measurement points (e.g., calipers) on regions of interest (ROIs) within the image. The processing system can be equipped with software that can determine the measurement results of the ROI based on the measurement points. The measurement points can be manipulated as endpoints to be measured. Various ultrasound measurement tools are available for today's ultrasound transducers, such as the depth of a point in the image relative to the transducer surface, the distance between two points in the ROI, the diameter of a circle based on a point placed on the periphery of the ROI, and measurements of the major and / or minor axes of an ellipse based on a point placed on the periphery of the ROI. However, all these measurements are one-dimensional or two-dimensional (1D / 2D) representations of three-dimensional (3D) anatomy measured from 2D images. Thus, the measurement results themselves and the images derived from them are derived from a lower-dimensional representation of the actual anatomical structure. Sonographers are aware of this limitation and can therefore use additional features in the image to ensure that the imaging plane is a representative cross-section of the 3D anatomy to be measured. Therefore, measurements can vary depending on the sonographer, and the process of locating the optimal measurement plane can be time-consuming.

[0005] US2017 / 325783A1 discloses a method for determining the dimensions of structures in medical images.

[0006] US2008 / 221446A1 discloses a method for tracking points in ultrasound images. Summary of the Invention

[0007] This invention is defined by the claims. There remains a clinical need for improved systems and techniques for providing efficient and accurate ultrasound-based measurements of anatomical features. Embodiments of this disclosure provide techniques for automated anatomical feature measurements based on ultrasound images. In the disclosed embodiments, an ultrasound imaging system can acquire a set of image frames around an anatomical feature of interest (e.g., fetal head or heart chamber) using a tracked ultrasound probe. The ultrasound probe may be equipped with an inertial measurement tracker (e.g., including an accelerometer, gyroscope, and / or sensor) that can provide position and / or motion information of the ultrasound probe during image acquisition. Furthermore, the ultrasound probe may include markers that can be tracked by an external electromagnetic (EM) tracking and / or optical tracking system. A three-dimensional (3D) volume surrounding the anatomical feature of interest can be reconstructed based on the acquired images and the tracked position and / or motion information of the probe. The 3D volume can surround an optimal measurement plane for measuring the anatomical feature of interest. Clinicians can place measurement markers (e.g., calipers) on a first image for targeted measurements (e.g., the maximum length or diameter of the fetal head or the width of the heart chamber). The system can provide measurement assistance to clinicians by utilizing a prediction network (e.g., a deep learning network) to propagate measurement labels from a first image to other acquired images based on a reconstructed 3D volume. Furthermore, the prediction network can be trained to create a multiplanar reconstruction (MPR) based on the reconstructed 3D volume and propagate measurement labels from the first image to all MPRs. Thus, the prediction network can provide cross-planes (e.g., MPRs) for obtaining the optimal measurement results for the features of interest. Additionally, the prediction network can be trained to segment anatomical features of interest from the images and perform measurements based on the segmented features. The prediction network can output the final measurement based on the measurements obtained from all images, the statistics of the measurements, and / or the confidence level of the measurements.

[0008] In one embodiment, an ultrasound imaging system includes processor circuitry in communication with an ultrasound transducer array, the processor circuitry being configured to: receive from the ultrasound transducer array a set of images of a three-dimensional (3D) volume of a patient's anatomical structure including anatomical features; obtain first measurement data of the anatomical features in a first image of the set of images; generate second measurement data for the anatomical features in one or more images of the set of images by propagating the first measurement data from the first image to one or more images of the set of images; and output the second measurement data for the anatomical features to a display in communication with the processor circuitry.

[0009] In some aspects, the system may further include processor circuitry configured to acquire the first measurement data, which is configured to receive, from a user interface communicating with the processor circuitry, the first measurement data comprising at least two measurement markers across the anatomical feature on the first image. The set of images is associated with multiple imaging planes across a 3D volume of the anatomical structure of the patient including the anatomical feature. The processor circuitry is configured to propagate the first measurement data from the first image to the one or more images based on positional data of the ultrasound transducer array relative to the multiple imaging planes. In some aspects, the system may further include processor circuitry configured to generate the second measurement data, which is configured to determine 3D spatial data for the first image and the one or more images based on the positional data of the ultrasound transducer array; and propagate the first measurement data from the first image to the one or more images based on the 3D spatial data. In some aspects, the system may further include a probe comprising the ultrasonic transducer array and an inertial measurement tracker, wherein the processor circuitry is configured to: receive inertial measurement data associated with the ultrasonic transducer array and the plurality of imaging planes from the inertial measurement tracker, and wherein the processor circuitry configured to determine the 3D spatial data is configured to: determine the position data of the ultrasonic transducer array relative to the plurality of imaging planes based on the inertial measurement data and an inertial measurement data-to-image transformation. In some aspects, the system may further include wherein the processor circuitry is configured to: generate third measurement data for the anatomical feature based on the first measurement data and the second measurement data, wherein the third measurement data is associated with at least one plane within the 3D volume that is a first imaging plane or a second imaging plane different from the plurality of imaging planes; and output the third measurement data to the display. In some aspects, the system may further include wherein the second imaging plane intersects with the first imaging plane. In some aspects, the system may further include wherein the third measurement data comprises at least one of the following: the second measurement data, the distance between two measurement markers across the anatomical feature, a confidence measure of the first measurement data, a confidence measure of the second measurement data, the average of the first and second measurement data, the variance of the first and second measurement data, or the standard deviation of the first and second measurement data. In some aspects, the system may further include a user interface that communicates with the processor circuitry and is configured to provide selections associated with the third measurement data.In some aspects, the system may further include processor circuitry configured to generate second measurement data for the anatomical features in the one or more images, configured to propagate the first measurement data from the first image to the one or more images based on image segmentation. In some aspects, the system may further include processor circuitry configured to generate second measurement data for the anatomical features in the one or more images, configured to propagate the first measurement data from the first image to the one or more images using a prediction network trained for at least one of image segmentation or feature measurement. In some aspects, the system may further include the prediction network being trained on a set of image-measurement pairs for the feature measurement results, and wherein each image-measurement pair in the set of image-measurement pairs includes an image in a sequence of images of a 3D anatomical volume and a measurement result of a feature of the 3D anatomical volume for the image. In some aspects, the system may further include the prediction network being trained on a set of image-segmentation pairs for image segmentation, wherein each image-segmentation pair in the set comprises an image in a sequence of images of a 3D anatomical volume and a segmentation of features of the 3D anatomical volume for the image. In some aspects, the system may further include the anatomical feature comprising a fetal head, and wherein the first measurement data and the second measurement data are associated with at least one of the circumference or length of the fetal head. In some aspects, the system may further include the anatomical feature comprising a left ventricle, and wherein the first measurement data and the second measurement data are associated with at least one of the width, height, area, or volume of the left ventricle.

[0010] In one embodiment, an ultrasound imaging method includes: receiving a set of images of a three-dimensional (3D) volume of a patient's anatomical structure, including anatomical features, at a processor circuit in communication with an ultrasound transducer array; obtaining first measurement data of the anatomical features in a first image of the set of images; generating second measurement data for the anatomical features in the one or more images of the set of images at the processor circuit by propagating the first measurement data from the first image to one or more images of the set of images; and outputting the second measurement data for the anatomical features to a display in communication with the processor circuit.

[0011] In some aspects, the method may further include obtaining the first measurement data by receiving first measurement data comprising at least two measurement markers across the anatomical feature from a user interface communicating with the processor circuitry. The set of images is associated with multiple imaging planes across the 3D volume of the patient's anatomical structure including the anatomical feature. The first measurement data is propagated from the first image to the one or more images based on positional data of the ultrasound transducer array relative to the multiple imaging planes. Optionally, generating the second measurement data includes determining 3D spatial data for the first image and the one or more images based on the positional data of the ultrasound transducer array relative to the multiple imaging planes; and propagating the first measurement data from the first image to the one or more images based on the 3D spatial data. In some aspects, the method may further include receiving inertial measurement data associated with the ultrasound transducer array from an inertial measurement tracker communicating with the processor circuitry, and determining positional data of the ultrasound transducer array relative to the first image and the one or more images based on the inertial measurement data and an inertial measurement result-to-image transformation.

[0012] Additional aspects, features, and advantages of this disclosure will become apparent from the following detailed description. Attached Figure Description

[0013] Illustrative embodiments of this disclosure will be described with reference to the accompanying drawings, in which:

[0014] Figure 1 This is a schematic diagram of an ultrasound imaging system according to aspects of this disclosure.

[0015] Figure 2 This is a schematic diagram of an automated ultrasound image-based measurement scheme according to aspects of this disclosure.

[0016] Figure 3 This is a schematic diagram of an automated ultrasound image-based measurement scheme according to aspects of this disclosure.

[0017] Figure 4 This is a schematic diagram of an automated ultrasound image-based measurement scheme according to aspects of this disclosure.

[0018] Figure 5 This is a schematic diagram of an automated measurement scheme based on deep learning and ultrasound images according to aspects of this disclosure.

[0019] Figure 6 This is a schematic diagram of a deep learning network configuration for ultrasound image-based measurements according to aspects of this disclosure.

[0020] Figure 7This is a schematic diagram of a deep learning network training scheme based on ultrasound image measurement according to aspects of this disclosure.

[0021] Figure 8 This is a schematic diagram of an automated measurement scheme based on deep learning and ultrasound images according to aspects of this disclosure.

[0022] Figure 9 This is a schematic diagram of a user interface for an automated ultrasound image-based measurement system according to aspects of this disclosure.

[0023] Figure 10 This is a schematic diagram of a processor circuit according to an embodiment of the present disclosure.

[0024] Figure 11 This is a flowchart of a deep learning-based ultrasound image measurement method according to aspects of this disclosure. Detailed Implementation

[0025] For the purpose of facilitating an understanding of the principles of this disclosure, reference will now be made to embodiments illustrated in the accompanying drawings, and these will be described using specific language. However, it should be understood that this is not intended to limit the scope of this disclosure. Any changes and further modifications to the described devices, systems, and methods, as well as any further applications of the principles of this disclosure, are fully contemplated and included within this disclosure, as will commonly conceived by those skilled in the art to which this disclosure pertains. In particular, it is fully contemplated that features, components, and / or steps described with respect to one embodiment may be combined with features, components, and / or steps described with respect to other embodiments of this disclosure. However, for the sake of brevity, numerous iterations of these combinations will not be described separately.

[0026] Figure 1 This is a schematic diagram of an ultrasound imaging system 100 according to aspects of this disclosure. System 100 is used to scan areas or volumes of a patient's body. System 100 includes an ultrasound imaging probe 110 that communicates with a host computer 130 via a communication interface or link 120. Probe 110 includes a transducer array 112, a beamformer 114, processor circuitry 116, and a communication interface 118. Host computer 130 includes a display 132, processor circuitry 134, and a communication interface 136.

[0027] In an exemplary embodiment, probe 110 is an external ultrasound imaging device including a housing configured for hand-held operation by a user. Transducer array 112 may be configured to acquire ultrasound data as the user grips the housing of probe 110, such that transducer array 112 is positioned adjacent to and / or in contact with the patient's skin. Probe 110 is configured to acquire ultrasound data of anatomical structures within the patient's body when probe 110 is positioned externally to the patient's body. In some embodiments, probe 110 may be an external ultrasound probe suitable for fetal examination. In some other embodiments, probe 110 may be a transthoracic (TTE) or transesophageal (TEE) ultrasound probe suitable for cardiac examination.

[0028] Transducer array 112 emits ultrasound signals toward the anatomical object 105 of the patient and receives echo signals reflected back to transducer array 112 from the object 105. Ultrasound transducer array 112 may include any suitable number of acoustic elements, including one or more acoustic elements and / or multiple acoustic elements. In some instances, transducer array 112 includes a single acoustic element. In some instances, transducer array 112 may include an array of acoustic elements having any number of acoustic elements in any suitable configuration. For example, transducer array 112 may include values ​​between 1 and 10,000 acoustic elements, including values ​​such as 2 acoustic elements, 4 acoustic elements, 36 acoustic elements, 64 acoustic elements, 128 acoustic elements, 500 acoustic elements, 812 acoustic elements, 1,000 acoustic elements, 3,000 acoustic elements, 8,000 acoustic elements, and / or other values ​​greater than or less than both. In some instances, transducer array 112 may comprise an array of acoustic elements having any number of acoustic elements in any suitable configuration, such as linear arrays, planar arrays, curved arrays, wavy arrays, circular arrays, ring arrays, phased arrays, matrix arrays, one-dimensional (1D) arrays, 1.x-dimensional arrays (e.g., 1.5D arrays), or two-dimensional (2D) arrays. The array of acoustic elements (e.g., one or more rows, one or more columns, and / or one or more orientations) can be controlled and activated uniformly or independently. Transducer array 112 may be configured to acquire one-dimensional, two-dimensional, and / or three-dimensional images of patient anatomy. In some embodiments, transducer array 112 may comprise piezoelectric micromechanical ultrasonic transducers (PMUTs), capacitive micromechanical ultrasonic transducers (CMUTs), single crystals, lead zirconate titanate (PZT), PZT composites, other suitable transducer types, and / or combinations thereof.

[0029] Object 105 may include any anatomical structure suitable for ultrasound imaging examination, such as blood vessels, nerve fibers, airways, mitral valve leaflets, cardiac structures, abdominal tissue structures, kidneys, and / or liver of a fetus within a patient and / or pregnant mother. In some embodiments, object 105 may include at least a portion of a patient's heart, lungs, and / or skin. This disclosure can be implemented in the context of any number of anatomical locations and tissue types, including but not limited to organs, including the liver, heart, kidneys, gallbladder, pancreas, lungs; ducts; intestines; nervous system structures, including the brain, dural sac, spinal cord, and peripheral nerves; the urinary tract; and valves within blood vessels, blood, chambers or other parts of the heart, the uterus of a pregnant mother, and / or other systems of the body. In some embodiments, object 105 may include malignant tumors, such as tumors, cysts, lesions, hemorrhages, or blood pools within any part of a human anatomical structure. Anatomical structures may be blood vessels, such as arteries or veins of a patient's vascular system, including cardiac vessels, peripheral vessels, neurovascular vessels, renal vessels, and / or any other suitable lumens within the body. In addition to natural structures, this disclosure can be implemented in the context of artificial structures, such as, but not limited to, heart valves, stents, shunts, filters, implants and other devices.

[0030] Beamformer 114 is coupled to transducer array 112. For example, beamformer 114 controls transducer array 112 for transmitting ultrasound signals and receiving ultrasound echo signals. Beamformer 114 provides image signals to processor circuitry 116 based on the response of the received ultrasound echo signals. Beamformer 114 may include multiple stages of beamforming. Beamforming can reduce the number of signal lines used for coupling to processor circuitry 116. In some embodiments, transducer array 112 combined with beamformer 114 may be referred to as an ultrasound imaging component.

[0031] Processor circuitry 116 is coupled to beamformer 114. Processor circuitry 116 may include a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), controller, field-programmable gate array (FPGA) device, another hardware device, firmware device, or any combination thereof configured to perform the operations described herein. Processor circuitry 134 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors coupled with a DSP core, or any other such configuration. Processor circuitry 116 is configured to process beamformed image signals. For example, processor circuitry 116 may perform filtering and / or quadrature demodulation to modulate the image signals. Processor circuitry 116 and / or 134 may be configured to control array 112 to obtain ultrasound data associated with object 105.

[0032] Communication interface 118 is coupled to processor circuitry 116. Communication interface 118 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. Communication interface 118 may include hardware and / or software components implemented with a specific communication protocol adapted to transmit signals to host 130 via communication link 120. Communication interface 118 may be referred to as a communication device or communication interface module.

[0033] Communication link 120 can be any suitable communication link. For example, communication link 120 can be a wired link, such as a Universal Serial Bus (USB) link or an Ethernet link. Alternatively, communication link 120 can be a wireless link, such as an Ultra Wideband (UWB) link, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 WiFi link, or a Bluetooth link.

[0034] At host 130, communication interface 136 can receive image signals. Communication interface 136 can be substantially similar to communication interface 118. Host 130 can be any suitable computing and display device, such as a workstation, personal computer (PC), laptop computer, tablet computer, or mobile phone.

[0035] Processor circuitry 134 is coupled to communication interface 136. Processor circuitry 134 can be implemented as a combination of software and hardware components. Processor circuitry 134 may include a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), controller, FPGA device, another hardware device, firmware device, or any combination thereof configured to perform the operations described herein. Processor circuitry 134 can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors coupled with a DSP core, or any other such configuration. Processor circuitry 134 can be configured to generate image data based on image signals received from probe 110. Processor circuitry 134 may apply advanced signal processing and / or image processing techniques to the image signals. In some embodiments, processor circuitry 134 may form a three-dimensional (3D) volumetric image based on the image data. In some embodiments, processor circuitry 134 may perform real-time processing on the image data to provide a streaming video of an ultrasound image of object 105.

[0036] Display 132 is coupled to processor circuitry 134. Display 132 may be a monitor or any suitable display. Display 132 is configured to display ultrasound images, video images, and / or any imaging information of object 105 and / or medical device 108.

[0037] System 100 can be used to assist sonographers in performing measurements based on acquired ultrasound images. In some aspects, system 100 can capture a sequence of ultrasound images of object 105. Clinicians can be interested in measurements that determine a particular anatomical feature of object 105. In an example, a sonographer can perform a 2D transthoracic echocardiogram of object 105, including the patient's cardiac structures, and perform quantitative structural and / or functional measurements of the cardiac structures. For example, a sonographer can use techniques such as the disc-based dual-plane approach to perform linear measurements during cardiac systole and diastole and / or LV volume measurements to estimate functional parameters (e.g., ejection fraction (EF)) during the scan. Similarly, a sonographer can perform right ventricular (RV) structural measurements during the scan to assess RV function. For example, the right ventricular outflow tract (RVOT) can be measured in both proximal and distal directions. Furthermore, structural dimensional measurements, such as ventricular equivalence (involving the relative size of the left and right atria), aortic root diameter, and inferior vena cava diameter, can be made during echocardiographic scans. To obtain accurate measurements, it is important to avoid shortening. Shortening refers to the situation where the 2D ultrasound plane does not cut through the apex of the heart structure. Shortening can produce erroneous measurements. In another example, a sonographer may perform fetal imaging to obtain measurements of the fetal head circumference, a key measurement indicating fetal growth. Therefore, accurate and precise measurements of the fetal head circumference are important. To obtain accurate and precise measurements of the fetal head circumference, the measurement must be performed in an axial cross-sectional plane, which is a plane perpendicular to the infant's foot-to-head axis through the infant's head. Furthermore, the measurement must be performed at a level that maximizes the measurement result. Fetal head circumference measurements performed on arbitrary imaging planes can be misleading. To ensure that the correct or optimal measurement imaging plane is captured, the sonographer may look for the presence of skull features in the image that indicate the correct imaging plane. Although the sonographer may look for additional anatomical features to ensure that the measurement is performed at the correct measurement plane, scan times can be long, and the resulting measurements can be user-dependent.

[0038] According to embodiments of this disclosure, system 100 is also configured to provide automated ultrasound image-based measurements by using a tracked ultrasound probe to acquire ultrasound images (2D ultrasound images) at and / or around an optimal measurement plane and by using tracking information to generate a 3D volume surrounding the measurement plane. In some aspects, probe 110 may include an inertial measurement tracker 117. The inertial measurement tracker 117 may include accelerometers, gyroscopes, and / or sensors to acquire and track the motion of probe 110 while scanning object 105. System 100 may additionally include an external tracking system that may be based on electromagnetic (EM) tracking and / or optical tracking, and probe 110 may include markers that can be tracked by the external tracking system to provide position and / or information of probe 110. Processor circuitry 134 may create a 3D volume of object 105 or define a 3D spatial dataset of object 105 in 3D space based on tracking information and acquired images. The 3D spatial information may allow for more accurate measurements with the aid of artificial intelligence (AI) or depth-tilt-based agents.

[0039] In some aspects, processor circuitry 134 may implement one or more deep learning-based prediction networks trained to identify regions of interest on ultrasound images for measurement, propagate user-identified measurement locations from one image to adjacent images, create multiplanar reconstructions (MPRs) for cross-planar measurements, and / or segment anatomical structures of interest from images for automated measurement. Mechanisms for providing automated measurement results based on ultrasound images are described in more detail herein.

[0040] In some aspects, system 100 can be used to collect ultrasound images to form a training dataset for training a deep learning network. For example, host 130 may include memory 138, which can be any suitable storage device, such as cache memory (e.g., cache memory of processor circuitry 134), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory devices, hard disk drives, solid-state drives, other forms of volatile and non-volatile memory, or combinations of different types of memory. Memory 138 can be configured to store image dataset 140 to train the series of predictive or deep learning networks for providing automated ultrasound image-based measurements. The mechanisms for training predictive or deep learning networks are described in more detail herein.

[0041] Figure 2-6 The diagrams together illustrate the mechanism for automated measurements based on ultrasound images. Figure 2This is a schematic diagram of an automated ultrasound image-based measurement scheme 200 according to aspects of this disclosure. Figure 3 This is a schematic diagram of an automated ultrasound image-based measurement scheme 300 according to aspects of this disclosure. Figure 4 This is a schematic diagram of an automated ultrasound image-based measurement scheme 400 according to aspects of this disclosure. Figure 5 This is a schematic diagram of an automated deep learning-based ultrasound image measurement scheme 500 according to aspects of this disclosure. Figure 6 This is a schematic diagram of a deep learning network configuration 600 for ultrasound image-based measurements according to aspects of this disclosure. Configurations 200, 300, 400, and 500 can be implemented by system 100.

[0042] refer to Figure 2 Solution 200 includes an inertial measurement tracker 220, an image frame acquisition unit 230, a volume reconstruction unit 240, a measurement marker placement unit 250, a measurement marker propagation unit 260, and a measurement result determination unit 270. The inertial measurement tracker 220, image frame acquisition unit 230, volume reconstruction unit 240, measurement marker placement unit 250, measurement marker propagation unit 260, and measurement result determination unit 270 can be implemented by a combination of hardware (e.g., including processing circuitry, logic, and / or gates) and / or software. In some instances, the volume reconstruction unit 240, measurement marker placement unit 250, measurement marker propagation unit 260, and / or measurement result determination unit 270 can be implemented by processor circuitry 134.

[0043] At a high level, scheme 200 uses a tracking ultrasound probe 210, similar to probe 110, to acquire a set of images 202 of the patient's anatomical structure (e.g., object 105) around the measurement plane, wherein measurements of features of interest within the patient's anatomical structure can be performed. In this regard, the sonographer or user can sweep the probe 210 around the region of interest to be measured. The image frame acquisition unit 230 acquires images 202 while sweeping the probe 210 as indicated by the dashed arrow 201. Images 202 are shown as f(0), f(1), ..., f(N-2), and f(N-1). In some instances, probe 210 may be a 1D ultrasound probe configured to acquire 2D ultrasound images 202, and sweeping may include physically sweeping the probe 210 around the region of interest within the 3D volume of the patient's anatomical structure. In some other instances, probe 210 may be a 2D ultrasound probe capable of performing 3D imaging, and sweeping may include electronically manipulating the ultrasound beam to acquire 2D images 202 at various 2D imaging planes within the 3D volume. The image frame acquisition unit 230 can be configured to acquire images 202 at a specific frame rate. The image frame acquisition unit 230 can then provide this set of images 202 to the volumetric reconstruction unit 240.

[0044] Inertial measurement tracker 220 is similar to inertial measurement tracker 117 (e.g., including an accelerometer, gyroscope, and / or sensor) and may be located within probe 210. Inertial measurement tracker 220 is configured to track the motion of probe 210 while acquiring the set of images 202. Inertial measurement tracker 220 is configured to record position information 222 of probe 210 while acquiring images 202, such that the position or coordinates of the imaging plane of each image 202 are known for subsequent processing. For example, at time T1, probe 210 may acquire image 202f(0) at a first imaging plane. At the next time T2, probe 210 may acquire image 202f(1) at a second imaging plane. Position information 222 may include translations and / or rotations applied to probe 210 or beam steering between time T1 and time T2, allowing probe 210 to reach the second imaging plane. Position information 222 is provided to volume reconstruction component 240 for volume reconstruction.

[0045] In some respects, the inertial measurement tracker 220 can provide six degrees of freedom (6DOF) in a space with three acceleration axes and three rotational velocity axes. While three-axis acceleration and three-axis gyroscope information may be sufficient to determine the motion of the ultrasonic probe 210 relative to a reference coordinate system, additional information provided by other inertial measurement tracking systems, such as electromagnetic (EM) field readings and / or optical field readings, can improve overall measurement accuracy. In this regard, the ultrasonic probe 210 can be equipped with markers that can be tracked via EM-based tracking and / or optical tracking.

[0046] Inertial measurement tracker 220 can provide acceleration, translation, and / or rotation measurements in its local coordinate system (relative to the axes of inertial measurement tracker 220). However, these measurements can be noisy and may have specific measurement biases. Various techniques can be applied to calculate the attitude information of probe 210 in a global coordinate system (e.g., a specific reference frame) to provide more accurate position information. In this regard, probe 210 can be attached to sensors, and sensor information can be integrated with acceleration, translation, and / or rotation measurement data to provide more accurate position information. For example, readings from sensors can be used to calibrate the local coordinate system of inertial measurement tracker 220 relative to an image coordinate system in 3D space, as described in more detail below. In some other instances, scheme 200 can apply specific filtering operations and / or fusion algorithms to the position information to reduce noise and / or bias, thereby improving measurement accuracy. In some instances, in addition to the inertial measurement tracker 220, image-based tracking can be used to estimate the position and / or motion (e.g., translation and / or rotation) of the probe 210. For example, image-based tracking may include deep learning-based algorithms that regress the 6DOF pose based on extracted anatomical features. Alternatively, image-based tracking may include conventional image processing algorithms, such as registration-based algorithms and / or speckle tracking algorithms. Typically, scheme 200 may use position information 222 obtained from the inertial measurement tracker 220 combined with position information measured by any other suitable tracking system to determine the position of the probe 210 relative to the global coordinate system.

[0047] During acquisition, the system can continuously buffer images 202 in memory (e.g., memory 138) until the user freezes or stops acquisition. Volumetric reconstruction component 240 is configured to determine the 3D pose information of each image 202 in 3D space. In other words, volumetric reconstruction component 240 can determine the relative position between two acquired images 202. In this regard, volumetric reconstruction component 240 can determine the 3D image pose by multiplying the pose of the sensor or inertial measurement tracker 220 by a transformation matrix, as follows:

[0048] (1)

[0049] in, This represents the 3D pose of image 202. This indicates the 3D attitude of the sensor or inertial measurement tracker 220. Represents the transformation matrix. It is the unit vector of the rotation component of a transformation relative to a specific reference frame in 3D space, and This represents translation. The volume reconstruction component 240 can calculate the attitude of the sensor or inertial measurement tracker 220 based on the position information 222 (e.g., using a fusion algorithm). The transformation matrix can be obtained through a calibration process. In this context, sensor position / or orientation and image orientation are measured relative to each other using actual measurement results and / or computer-aided design (CAD) images.

[0050] In some respects, calibration can be performed on each probe 210 during the setup phase before imaging or during the manufacturing phase. Transformation matrix The coordinate system of the transducers (e.g., transducer array 112) on probe 210 is transformed to the coordinate system of image 202 (defined by inertial measurement tracker 220). For example, image orientation can be measured based on transducer orientation. Calibration can determine the transformation matrix. Translation and / or rotation of the transformation matrix The application can convert the attitude of the inertial measurement tracker 220 into an image attitude. After obtaining the 3D image attitude of each image 202, the coordinates of all points (e.g., pixels) in image 202 relative to other images 202 are known, and the 3D placement of all points in the set of images 202 is known.

[0051] As an example, each image 202 may include 256 pixels, each represented by an intensity value at a point defined by homogeneous (x, y) coordinates, which can be transformed to transducer space using a first transformation matrix. The first transformation matrix is ​​a 3×3 matrix comprising 2×2 rotations (at entries (1,1), (1,2), (2,1), (2,2)) and 2 translations (at the last column of the matrix). The 2×2 rotations correspond to the in-plane rotation components of the image (i.e., around the z-axis perpendicular to the image). Any vector on image 202 can be described using the first transformation matrix and a reference point located at the end of probe 210 at the beginning of the image. The image reference point and sensor coordinate system are also transformed by a known (or measurable) second transformation matrix (e.g., ...). The two transformations are correlated with each other. Since the image reference point and the sensor coordinate system are in three-dimensional space, the second transformation matrix is ​​a 4×4 matrix with 3×3 rotations and 3 translations. When the probe 210 moves from one imaging plane (e.g., the imaging plane of f(0) at time T1) to the next imaging plane (e.g., the imaging plane of f(1) at time T2), the motion is represented by a third transformation matrix (e.g., a 4×4 matrix) from sensor readings (e.g., position information 222), which corresponds to how much the sensor (or inertial measurement tracker 220) has moved and in which direction the sensor has moved. The third transformation matrix can be multiplied by the second transformation matrix to obtain the amount of movement by the image reference point (the end of the probe) during that motion. Subsequently, the product of the second and third transformation matrices can be multiplied by the first transformation matrix to obtain the motion experienced by a particular pixel position. Similarly, point P1 in the first image 202 (e.g., f(0)) and point P2 in the second image 202 (e.g., f(1)) can be correlated with each other using the same set of transformation matrices. Therefore, the volume reconstruction component 240 generates a 3D volume or 3D spatial dataset 242, which includes xyz coordinates in 3D space and the corresponding intensity value of each pixel in each image 202.

[0052] In some other instances, the volume reconstruction component 240 can use a deep learning network (e.g., a CNN) to construct a 3D volume (e.g., a 3D spatial dataset 242), which regresses the pose of the image relative to the local coordinate system of a specific anatomical structure on which the network is trained. In still other instances, the relative distance (e.g., z-coordinate values) between two images 202 can be determined based on speckle statistics and speckle decorrelation using the image acquisition frame rate and beam focusing / steering configuration, without requiring tracking measurements and / or transformations. This method may require some initial calibration of the image-based tracking by the transducer manufacturer, such as speckle calibration curves.

[0053] The image frame acquisition unit 230 can also provide the image 202 to the measurement mark placement unit 250. The measurement mark placement unit 250 can place measurement marks or calipers (in...) Figure 3 The markers (shown as 310a and 310b) are placed on image 202 (e.g., image f(0)) where measurements can be taken. Image 202 f(0) is shown as 202_f(0). The measurement marker placement component 250 can output first measurement data 252 including image 202_f(0) and information associated with the measurement markers. In some instances, the placement of the markers can be received via user input. In other words, the sonographer performing the scan can determine the location where the markers can be placed for measurement. In some other instances, the placement of the markers can be generated by a deep learning network trained for measurement point recognition.

[0054] As an example, image 202 is acquired during a fetal examination, wherein image 202 may include a view of the fetal head within the mother's uterus. Measurement markers may be placed on the image for fetal head measurement, as shown below. Figure 3 As discussed in the article.

[0055] refer to Figure 3 The measurement marker placement component 250 can operate on image 202f(0). Image 202 includes a view of the fetal head 320 and markers 310a and 310b placed around the periphery of the fetal head 320, such that the distance 322 between markers 310a and 310b can represent the diameter of the fetal head 320. Measurement markers 310a and 310b can be in the form of measurement points as shown. In some other instances, measurement markers 310a and 310b can be in the form of lines, crosses, and / or any other suitable form, symbol, and / or shape. As discussed above, the placement of markers 310a and 310b can be performed by an ultrasound physician, and therefore the measurement marker placement component can simply receive the positions of markers 310a and 310b from the ultrasound physician.

[0056] Return to Figure 2 After the measurement marks 310a and 310b are placed on the image 202_f(0), the measurement mark propagation component 260 is configured to propagate the measurement marks 310a and 310b from the image 202_f(0) to other adjacent images 202 stored in the buffer memory.

[0057] refer to Figure 3 The measurement mark propagation component 260 propagates the measurement marks 310a and 310b selected for image 202_f(0) to other images 202 in the group (e.g., f(1), f(2), ..., f(N-1)). In this respect, the measurement mark propagation component 260 may first register the adjacent image 202 (e.g., image f(1)) with the image 202_f(0) where the marks are placed, and then transfer the measurement marks 310a and 310b to the adjacent image 202_f(1). Image registration may refer to the spatial alignment of one image with another image, as shown below. Figure 4 and 5 The propagation of measurement markers 310a and 310b can continue, wherein the propagation measurement result of image 202_(f1) can be used as the original measurement result and propagated to the next image 202 (e.g., image f(2) in the group), and so on. Propagation can be performed sequentially in the group of images 202 until markers 310a and 310b have been transferred to all images 202 in the group. The propagation measurement markers on image 202_f(1) are shown as 312a and 312b.

[0058] In some instances, propagation may skip one or more images 202 in the group. Typically, propagation can be performed to propagate measurement markers from image 202 f(i) to adjacent images 202 f(i+L) in the group, where L can be 1, 2, 3, or 4. Although propagation can be configured to skip one or more images 202 in the group by changing L, as L increases, the two images 202_f(i) and 202 f(i+L) may become less similar, and therefore the registration may become less accurate. Therefore, it is preferable to perform registration between images 202 that are immediately adjacent to each other.

[0059] refer to Figure 4 Scheme 400 can be implemented by the measurement mark propagation component 260. Figure 4 A more detailed view of the registration process is provided. Scheme 400 includes a registration and propagation component 410, which may include hardware and / or software configured to perform image registration and measurement marker propagation. In some aspects, the registration and propagation component 410 may use model-based, mutual information-based, or similarity-based registration techniques to register images 202_f(0) and 202_f(1) relative to each other. Some example similarity measurements may include sum of squared differences (SSD) or sum of absolute differences (SAD) measurements. The registration and propagation component 410 may align image features of image 202_f(1) to image features of image 202_f(0). Although images 202_f(1) and 202_f(0) may be parallel to each other, they may not be aligned. Therefore, the propagation marker 312a on image 202_f(1) may not be at the same pixel location (e.g., pixel (x, y) coordinates) as the marker 310a on image 202_f(0). After registration, the registration and propagation unit 410 propagates the measurement markers 310a and 310b from image 202_f(0) to image 202_f(1).

[0060] In some aspects, in order to propagate measurement markers 310a and 310b from image 202_f(0) to image 202_f(1), the registration and propagation component 410 can copy measurement points (e.g., the positions of measurement markers 310a and 310b) from image 202_f(0) to image 202_f(1). The registration and propagation component 410 can also adjust the copied marker positions to optimal positions. For example, the initial placement of measurement markers 310a and 310b on image 202_f(0) can be based on locally maximizing or minimizing a specific cost function. In some instances, the cost function can be based on image brightness. Therefore, the measurement point or marker positions copied to image 202_f(1) may need to be re-evaluated near the copied measurement point or marker positions, for example, to optimize the cost function. The re-evaluation or optimization of the copied measurement point or marker positions can be processed as a learning task to determine the most probable positions of measurement points on image 202_f(1), which can be done via, for example, Figure 5 The deep learning techniques shown are used to implement this.

[0061] For the purpose of simplifying the illustration, Figure 4 The propagation of marker 310a from image 202_f(0) to image 202_f(1) is illustrated only within the dashed box (as shown by marker 312a). However, a similar propagation can be shown for marker 310b. As can be observed, if image 202_f(0) is to be directly superimposed on top of image 202_f(1) without registration, marker 310a is offset from marker 312a on image 202_f(1). Output 402 shows images 202_f(0) and 202_f(1) after registration, where measurement markers 310a and 312a are spatially aligned and are shown as 406a.

[0062] refer to Figure 5 Scheme 500 can be implemented by measurement mark propagation component 260. Scheme 500 includes a deep learning network 510 trained to perform image registration and measurement mark propagation. As shown, the deep learning network 510 can be applied to images 202_f(1) and 202_f(0) and measurement marks 310a and 310b. The deep learning network 510 generates an output 502 including an image 202_f(1) with propagated marks having propagated from marks 310a and 310b on image 202_f(0). For the purpose of simplifying the illustration, Figure 5 Only propagation marker 312a is illustrated. However, a similar propagation can be shown for marker 310b.

[0063] In some instances, the deep learning network 510 may include two CNNs: a first CNN 512 for registration and a second CNN 514 for measurement marker propagation. The first CNN is trained to regress the translation and rotation components of the registration process given a fixed image (e.g., image 202_f(0)) and a moving image (e.g., image 202_f(1)). The second CNN is trained to regress the coordinates of the measurement points given an input ultrasound image. In some other instances, the deep learning network 510 may include a measurement CNN without a registration CNN. Registration may be performed between images 202_f(0) and 202_f(1) before the deep learning network 510 is applied. In still other instances, the deep learning network 510 may include a single CNN trained to perform image registration and measurement marker propagation. These are described below respectively. Figure 6 and 7 The configuration and training of the 510 deep learning network are described in more detail.

[0064] refer to Figure 6 Configuration 600 can be implemented by a deep learning network 510. Configuration 600 includes a deep learning network 610, which comprises one or more CNNs 612. For simplicity of illustration and discussion, Figure 6 A CNN 612 is illustrated. However, the embodiment can be scaled to include any suitable number of CNNs 612 (e.g., about 2, 3 or more). Configuration 600 is described in the context of measuring label propagation. However, configuration 600 can be applied to measuring label placement and / or image registration by training a deep learning network 610 for measuring label placement and / or image registration, as described in more detail below.

[0065] CNN 612 may include a set of N convolutional layers 620 followed by a set of K fully connected layers 630, where N and K can be any positive integers. The convolutional layers 620 are shown as 620. （1） Up to 620 （N） The fully connected layer 630 is shown as 630. （1） Up to 630 （k）Each convolutional layer 620 may include a set of filters 622 configured to extract features from an input 602 including images 202_f(L) and 202_f(i). Image 202_f(L) may include views of anatomical features of interest and measurement markers. For example, image 202_f(L) may correspond to image 202_f(0) having measurement markers 310a and 310b placed by a user around the periphery of the fetal head 320 for measuring the diameter of the fetal head. Image 202_f(i) may correspond to images 202 in this set other than image 202_f(L). The values ​​N and K, and the size of the filters 622, may vary depending on the embodiment. In some instances, convolutional layer 620 （1） Up to 620 （N） and fully connected layer 630 （1） Up to 630 （K-1） Leaky rectified nonlinear (ReLU) activation functions and / or batch normalization can be used. The fully connected layer 630 can be nonlinear, and the high-dimensional output can be gradually reduced to the dimension of the prediction result 604.

[0066] Input images 202_f(L) and 202_f(i) can be passed sequentially through each layer 620, 630 for feature extraction, analysis, and / or classification. Each layer 620, 630 may include weights applied to the input images 202_f(L) and 202_f(i) or the output of a previous layer 620 or 630 (e.g., filter coefficients of filter 622 in convolutional layer 620 and nonlinear weights of fully connected layer 630). In some instances, input images 202_f(L) and 202_f(i) can be fed into the deep learning network 610 image-by-image. In some other instances, images 202_f(L) and 202_f(i) can be fed into the deep learning network 610 as a 3D volumetric dataset.

[0067] CNN 612 can output a prediction 604 based on input images 202_f(L) and 202_f(i). The prediction 604 can include various types of data, depending on the training of the deep learning network 610, as discussed in more detail below. In some examples, the prediction 604 can include images 202_f(i), each with propagated measurement labels (e.g., labels 312a and 312b) propagated from measurement labels on image 202_f(L). Additionally or alternatively, images 202_f(i) can be measured based on the corresponding propagated labels and / or based on user-placed measurement labels, and the prediction 604 can include statistical measures of the measurements, such as mean, median, variance, or standard deviation. Additionally or alternatively, the prediction 604 can include a confidence metric or confidence score calculated based on the variance of the measurements. Typically, the deep learning network 610 can be trained to output the image 202 in the prediction result 604 with any suitable combination of propagation measurement labels, statistical measures and / or confidence measures.

[0068] Return to Figure 2 After the measurement marks 310a and 310b are propagated to the remaining images 202 in the group of images 202 (e.g., f(1) to f(N-1)), the measurement result determination component 270 is configured to determine the final measurement result 272 based on the output 262 provided by the measurement mark propagation component 260 (e.g., the image 202 with the propagated measurement marks).

[0069] Return to Figure 3 The measurement result determination component 270 determines the diameter of the fetal head 320 based on the measurement result 272 of the image 202 propagated using measurement markers. Figure 3 Side views of multiple images 202 for determining measurement results are provided. Images 202 are shown as dashed lines, and measurement markers are shown as solid circles on the periphery of the fetal head 320. Initially selected measurement markers 310a and 310b are shown for image 202 f(0). Propagation measurement markers 312 and 312b are also shown for image 202 f(1). The measurement result determination component 270 can determine the optimal measurement plane for measuring the diameter or maximum length of the fetal head 320 based on the measurement markers on all images 202.

[0070] In some instances, the optimal measurement plane may lie on the imaging plane of one of the images 202 (e.g., image 202 f(1)). In some other instances, the optimal measurement plane may not lie on any of the imaging planes used to acquire image 202. For example, the optimal measurement plane may lie between two of the acquired imaging planes (e.g., between the imaging planes of image 202 f(0) and image 202 f(1)). Alternatively, the measurement result determination component 270 may determine an intersecting plane 340 for obtaining the optimal measurement result for the diameter of the fetal head 320. The intersecting plane 340 may intersect with one or more of the acquired imaging planes. As shown, the intersecting plane 340 intersects with the imaging plane of image 202 f(2). The final measurement 272 may correspond to the distance between two points on the periphery of the fetal head 320 in the intersecting plane 340.

[0071] In some aspects, deep learning networks 510 or 610 can be trained to create multi-plane reconstructions (MPRs) from a 3D volume (e.g., 3D spatial data 242) and propagate measurement labels from the initial image 202 f(0) to all MPRs, as discussed in more detail below. This enables deep learning networks 510 or 610 to make optimal measurement results (e.g., final measurement result 272) on planes that are not the image plane of image 202 (e.g., intersecting plane 340). Such measurement result 272 will have been missed by the user because it is not on the imaging plane (indicated by the dashed line). In some instances, deep learning networks 510 or 610 may include 2D convolutional layers (e.g., convolutional layer 620) and can be applied to 2D image 202, such as Figure 6 As shown. In some other instances, deep learning networks 510 or 610 may include 3D convolutional layers (e.g., convolutional layer 620) and may be applied to 3D volumes.

[0072] Figure 7This is a schematic diagram of a deep learning network training scheme 700 for ultrasound image-based measurements according to aspects of this disclosure. Scheme 700 can be implemented by system 100. To train the deep learning network 610 for ultrasound image-based measurements, a training dataset (e.g., image dataset 140) can be created. The training dataset can include image-measurement pairs. For each image-measurement pair, the training dataset can associate an ultrasound image 702 of an anatomical structure (e.g., fetal head 320) with standard data (ground truth) including measurement markers placed on the ultrasound image by an expert for a specific measurement (e.g., the diameter of the fetal head 320). The ultrasound image 702 can be an image of a phantom, a live patient, and / or a cadaver acquired by a probe (such as probe 110 or 210). The deep learning network 610 can, for example, use forward propagation applied to each image 702 in the dataset to obtain an output or score for the input image. The coefficients of filter 622 in convolutional layer 620 and the weights in fully connected layer 630 can be adjusted, for example, by using backpropagation, to minimize prediction error (e.g., the difference between standard data and prediction result 704). Prediction result 704 may include the predicted placement of measurement markers 710a and 710b on image 702. In some instances, the coefficients of filter 622 in convolutional layer 620 and the weights in fully connected layer 630 can be adjusted for each input image-measurement pair. In some other instances, a batch training process can be used to adjust the coefficients of filter 622 in convolutional layer 620 and the weights in fully connected layer 630. For example, the prediction error is accumulated for a subset of image-measurement pairs before adjusting the coefficients of filter 622 in convolutional layer 620 and the weights in fully connected layer 630.

[0073] In some respects, the training dataset can cover a large population. For example, for fetal imaging, the training dataset can include ultrasound images of fetuses of different ages, sizes, weights, and / or rare abnormalities, allowing the deep learning network 610 to learn to predict fetal head 320 measurements for various conditions.

[0074] In some aspects, regions of interest (ROIs) can be identified from image 702 based on measurement markers placed by experts, and deep learning networks 610 can be trained on image patches that include ROIs. For example, a portion 703 of image 702 that includes fetal head 320 (e.g., ROI) is used as input to deep learning network 610 for training.

[0075] In some aspects, the deep learning network 610 can provide a probability distribution map at the measurement marker locations, and the user measurement results (e.g., user-selected marker locations) are propagated as a probability distribution. The probability distribution can be a Gaussian distribution, where the peaks of the probability distribution correspond to the registered measurement points (e.g., propagated measurement markers). The predicted probability distribution of the measurement markers can be reshaped based on the probability distribution of the user-selected measurement marker locations. The peaks of the reshaped probability distribution can provide more accurate marker locations. In this respect, prediction can be formulated as a Bayesian inference problem, as follows:

[0076] p(x|y) = p(y|x)p(x) , (2)

[0077] Here, p(x|y) represents the conditional probability distribution of the measurement marker position x given a new image y, p(y|x) represents the conditional probability between image y and marker position x as predicted by the deep learning network 610 based on the observed image y, and p(x) represents the prior distribution of x (i.e., the measurement point placed by the user propagates to the new image plane with a specific variance around it due to registration accuracy and user uncertainty). If the probability distributions p(y|x) and p(x) are multiplied as shown in equation (2), the peak position of the distribution p(x|y) can provide the optimal position for the measurement marker on image y. The probability function is a 2D function (such as a Gaussian function), and therefore the marker position can be determined based on the x, y position of the maximum value of the final probability function (e.g., p(y)).

[0078] In some aspects, different initial conditions and / or different convergence conditions can be used to train the deep learning network 610. Different training instances can produce different prediction results 704. Scheme 700 can compute statistical measures (e.g., mean, median, variance, standard deviation) of the prediction results 704 from different training instances. The prediction results 704 from different training instances can have a Gaussian-like probability distribution. Scheme 700 can update the measured label location based on the peak of the probability distribution to provide the optimal label location.

[0079] In some aspects, the deep learning network 610 can be trained to provide user-selected measurement marker locations and / or predicted or propagated statistical and / or confidence metrics for the measurement markers. In this regard, the deep learning network 610 can be trained using a training dataset comprising image-metric pairs (e.g., training dataset 140), each image-metric pair including an ultrasound image (e.g., image 202) and standard data indicating measurement markers placed by an expert and their corresponding statistical or confidence metrics. The deep learning network 610 can be trained using similar per-frame update training or batch training as discussed above.

[0080] In some respects, the deep learning network 610 can be trained to provide the final measurement results (e.g., Figure 3 The measurement result 272 on the cross plane 340 is shown. In this regard, the deep learning network 610 can be trained using a training dataset (e.g., training dataset 140) that includes image-measurement pairs, each image-measurement pair including a set of ultrasound images in a 3D volume (e.g., image 202) and standard data indicating the optimal measurement plane in the 3D volume (e.g., cross plane 340) and the corresponding measurement result on the measurement plane (e.g., measurement result 272) as determined by an expert. The deep learning network 610 can be trained using similar per-frame update training or batch training as discussed above.

[0081] Although schemes 200-600 are described in the context of fetal imaging, similar mechanisms can be used for cardiac measurements in cardiac imaging. Some examples of cardiac measurements may include the length, width, size, and / or volume of cardiac chambers (e.g., left ventricle, right ventricle, left atrium, right atrium, aorta, inferior vena cava). When deep learning networks 510 or 610 are used to propagate measurement markers for cardiac measurements (e.g., markers 310a, 310b, 312a, and / or 312b), deep learning networks 510 or 610 are trained using a training dataset that includes image-measurement pairs, each image-measurement pair comprising an ultrasound image of a cardiac structure and the corresponding measurement. Typically, each deep learning network can be trained to measure specific types of anatomical structures (e.g., fetal head circumference, cardiac chamber diameter, femur length, abdominal circumference) because deep learning networks can use surrounding anatomical features extracted from images to predict significant measurement points, and the training dataset can cover large populations (e.g., different ages, weights, sizes, and / or medical conditions). Deep learning networks can be trained to provide any suitable form of measurement, such as length, width, area, volume, size, radius, diameter, perimeter, and / or any suitable type of geometric measurement.

[0082] Furthermore, although schemes 500-600 are described in the context of image-based learning and prediction for measuring anatomical structures, where the input to deep learning networks 510 and / or 610 is an image, deep learning networks 510 and / or 610 can alternatively be trained to operate on 3D spatial datasets. (Reference) Figure 6As shown in the example, the deep learning network 610 can receive an input 3D spatial dataset comprising data points in 3D space for each pixel on each image 202 f(L) and f(i). Each data point, referred to as D, is defined by its x, y, z coordinates in 3D space and its associated intensity value, which can be represented by d(x, y, z, intensity level). Furthermore, the 3D spatial dataset may include the (x, y, z) coordinates of initially selected measurement markers (e.g., markers 310a and 310b) of the image 202_f(L).

[0083] Figure 8 This is a schematic diagram of an automated deep learning-based ultrasound image measurement scheme 800 according to aspects of this disclosure. Scheme 800 can be implemented by system 100. Scheme 20 can use scheme 800 for measurement marker placement, measurement marker propagation, and / or measurement result determination in place of measurement marker placement component 250, measurement marker propagation component 260, and / or measurement result determination component 270.

[0084] Scheme 800 can apply a deep learning network 810 trained to perform image segmentation 812 and measurement 814. As shown in the figure, the deep learning network 810 can be applied to the set of images 202. The deep learning network 810 can be trained to segment specific anatomical features from a given input image and then determine the measurement results of the segmented features. Figure 8 In the example shown, the input image 202 includes a view of the fetal head 320. A deep learning network 810 is trained to segment the fetal head (shown as 830) from image 202 and determine the diameter of the segmented fetal head segment 830. As shown, the output 804 predicted by the deep learning network 810 includes the segmented fetal head 830 on image 202 (shown by dashed lines) and an intersecting plane 840 for measuring the diameter of the fetal head 830 (shown as measurement result 842). The intersecting plane 840 may intersect the imaging plane of image 202 f(2).

[0085] In some instances, the deep learning network 810 may include one CNN (e.g., CNN 612) trained for segmentation 812 and another CNN trained for measurement 814. In some instances, the deep learning network 810 may include a single CNN trained for both segmentation 812 and measurement 814. Scheme 700 can be used to train the deep learning network 810. The training dataset may include image-segmentation pairs (e.g., ultrasound images and corresponding segmentation features) and / or image-measurement pairs (e.g., ultrasound images and corresponding measurements). Similarly, each deep learning network 810 may be trained for segmentation and measurement of a certain type of anatomical structure. Although... Figure 8The illustration shows segmentation and measurement using a deep learning network, but in some other instances, measurement can be performed using image processing and / or feature detection-based algorithms 812 and measurement can be performed using a CNN 814.

[0086] Figure 9 This is a schematic diagram of a user interface 900 for an automated ultrasound image-based measurement system according to aspects of this disclosure. The user interface 900 may be implemented by system 100 and may be displayed on display 132.

[0087] In some aspects, the user interface 900 may include marker selection 905 and measurement type selection 910. Marker selection 905 may allow the user to select an image (e.g., image 202) and place measurement markers (e.g., measurement markers 310a and 310b) on the selected image, for example via a pop-up window. Measurement type selection 910 may be in the form of a drop-down menu or other user interface, whereby the user can select the type of measurement (e.g., fetal head circumference, maximum fetal head length measurement, heart chamber length, width, size, and / or volume). The underlying system 100 may implement various deep learning networks (e.g., deep learning networks 510, 610, and 810) trained for measurements of different anatomical structures, or different types of measurements for specific types of anatomical structures, and / or segmentation of different types of anatomical structures. In some instances, the deep learning network may also be trained to detect and segment relevant anatomical structures in the input image based on measurement type selection 910.

[0088] In some aspects, the user interface 900 may include a measurement plane display panel 920, wherein an imaging plane 922 selected by a deep learning network and measurements 924 performed by the deep learning network on the imaging plane 922 are displayed to the user. In some instances, the measurement plane display panel 920 may display one or more of an image with propagated measurement marks and / or an initial image in which the user places the measurement marks.

[0089] In some aspects, the user interface 900 can provide the user with various options regarding measurements performed by the deep learning network. In this regard, the user interface 900 includes a measurement acceptance option 930, a measurement correction option 940, a new measurement option 950, and / or an image plane selection 960. The user can select the measurement acceptance option 930 to accept a measurement 924 performed by the deep learning network. The user can select the measurement correction option 940 to correct the measurement 924 performed by the deep learning network. The user can select the new measurement option 950 to request the deep learning network to perform another measurement. The user can select the image plane selection 960 to select a specific image plane for the measurement. In some aspects, the user's selections can be used to enhance the training of the deep learning network.

[0090] In some aspects, the user interface 900 may display confidence metrics 970 and / or statistical metrics 980 determined by a deep learning network. As discussed above, a user may place measurement markers on the acquired image (e.g., image 202f(0)), and the deep learning network may propagate the measurement markers to neighboring images. Confidence metrics 970 and / or statistical metrics 980 may provide confidence or statistical measures, respectively, regarding the user-selected placement of the measurement markers. Statistical metric 980 may include the mean, median, variance, and standard deviation of the user-placed markers and the propagated markers. In some instances, confidence metrics 970 may be used to color-code the display of measurement 924 values. For example, measurement 924 values ​​may be displayed in red, yellow, or green to represent low confidence, medium confidence, or high confidence, respectively.

[0091] Figure 10 This is a schematic diagram of a processor circuit 1000 according to an embodiment of the present disclosure. The processor circuit 1000 can be implemented in... Figure 1 The probe 110 and / or host 130 are located within the probe 110 and / or host 130. In this example, the processor circuitry 1000 can communicate with the transducer array 112 in the probe 110. As shown, the processor circuitry 1000 may include a processor 1060, a memory 1064, and a communication module 1068. These components may communicate directly or indirectly with each other, for example, via one or more buses.

[0092] Processor 1060 may include a CPU, GPU, DSP, application-specific integrated circuit (ASIC), controller, FPGA, another hardware device, firmware device, or any combination thereof, configured to perform the operations described herein, for example, Figure 2-11 In terms of computing devices, the processor 1060 can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0093] Memory 1064 may include cache memory (e.g., cache memory of processor 1060), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory devices, hard disk drives, other forms of volatile and non-volatile memory, or combinations of different types of memory. In embodiments, memory 1064 includes a non-transient computer-readable medium. Memory 1064 may store instructions 1066. Instructions 1066 may include, when executed by processor 1060, causing processor 1060 to perform actions on reference probe 110 and / or host 130 (…). Figure 1 The operation described (e.g., Figure 2-9 Instructions (and aspects of 11). Instruction 1066 may also be referred to as code. The terms "instruction" and "code" should be interpreted broadly to include one or more computer-readable statements of any type. For example, the terms "instruction" and "code" can refer to one or more programs, routines, subroutines, functions, flows, etc. "Instructions" and "code" can include a single computer-readable statement or many computer-readable statements.

[0094] The communication module 1068 may include any electronic circuitry and / or logic circuitry to facilitate direct or indirect data communication between the processor circuitry 1000, the probe 110, and / or the display 132. In this respect, the communication module 1068 may be an input / output (I / O) device. In some instances, the communication module 1068 facilitates direct or indirect data communication between the processor circuitry 1000 and / or the probe 110. Figure 1 ), Probe 210 ( Figure 2 ) and / or host 130 ( Figure 1 Direct or indirect communication between various components.

[0095] Figure 11 This is a flowchart of a deep learning-based ultrasound image measurement method 1100 according to aspects of this disclosure. Method 1100 is implemented by system 100, for example, by processor circuitry (such as processor circuitry 1000) and / or other suitable components (such as probe 110 or 210, processor circuitry 116, host 130, and / or processor circuitry 134). In some examples, system 100 may include a computer-readable medium on which program code is recorded, including code for causing system 100 to perform the steps of method 1100. Method 1100 may employ methods as described above with respect to... Figure 2 , 3 Schemes 200, 300, 400, 500, 700, and / or 800 described in 4, 5, 7, and / or 8, above regarding Figure 6 The configuration described is 600 and the above is about Figure 9 A similar mechanism is described in the user interface 900. As shown in the figure, method 1100 includes a plurality of the listed steps, but embodiments of method 1100 may include additional steps before, after, and between the listed steps. In some embodiments, one or more of the listed steps may be omitted or performed in a different order.

[0096] At step 1110, method 1100 includes receiving a set of images (e.g., image 202) of a 3D volume of a patient’s anatomical structure, including anatomical features (e.g., fetus, heart chambers), at a processor circuit (e.g., processor circuits 134 and 1000) in communication with an ultrasound transducer array (e.g., array 112).

[0097] At step 1120, method 1100 includes obtaining first measurement data of anatomical features from a first image in the set of images.

[0098] At step 1130, method 1100 includes generating second measurement data of anatomical features in one or more images of the set of images by propagating first measurement data from a first image to one or more images at a processor circuit.

[0099] At step 1140, method 1100 includes outputting second measurement data for anatomical features to a display (e.g., display 132) that communicates with processor circuitry.

[0100] In some instances, step 1120 includes receiving first measurement data from a user interface (e.g., user interface 900) that communicates with processor circuitry, including at least two measurement markers (e.g., measurement markers 310a and 310b) across anatomical features.

[0101] In some instances, the set of images is associated with multiple imaging planes across a 3D volume of a patient’s anatomical structures, including anatomical features. Step 1130 includes determining 3D spatial data (e.g., 3D spatial dataset 242) of a first image and one or more images based on positional data of the ultrasound transducer array relative to the multiple imaging planes, and propagating first measurement data from the first image to one or more images based on the 3D spatial data.

[0102] In some instances, method 1100 further includes receiving inertial measurement data (e.g., position information 222) associated with the ultrasonic transducer array from an inertial measurement tracker (e.g., inertial measurement trackers 117 and 220) communicating with processor circuitry. Method 1100 also includes a transformation to an image based on the inertial measurement data and the inertial measurement results (e.g., the transformation matrix in equation (1)). ( ) to determine the position data of the ultrasonic transducer array relative to the first image and one or more images.

[0103] In some instances, the second measurement data includes propagated measurement marks (e.g., measurement marks 312a and 312b) propagated from measurement marks placed on the first image on one or more images.

[0104] In some instances, method 1100 includes generating third measurement data for an anatomical feature based on first and second measurement data, wherein the third measurement data is associated with at least one of a first imaging plane among a plurality of imaging planes or a second imaging plane within a 3D volume that differs from the plurality of imaging planes. Method 1100 also includes outputting the third measurement data to a display. In some instances, the second imaging plane intersects with the first imaging plane. For example, the first imaging plane (e.g., the measurement plane) is an intersecting plane (e.g., intersecting planes 340 and 840). In some instances, the third measurement data includes at least one of the second measurement data, the distance between two measurement markers across the anatomical feature, a confidence measure of the first measurement data, a confidence measure of the second measurement data, the mean of the first and second measurement data, the variance of the first and second measurement data, or the standard deviation of the first and second measurement data.

[0105] In some instances, method 1100 also includes providing a user with choices associated with a third measurement result (e.g., choices 930, 940, 950, and 960) via a user interface (e.g., user interface 900).

[0106] In some instances, step 1130 includes propagating the first measurement data from the first image to one or more images based on image segmentation. In some instances, step 1130 includes propagating the first measurement data from the first image to one or more images using a prediction network (e.g., deep learning networks 510, 610, and / or 810) trained for at least one of the image segmentation or feature measurement results.

[0107] In some instances, a prediction network is trained on a set of image-measurement pairs for feature measurement, wherein each image-measurement pair in the set includes images from a sequence of images representing 3D anatomical volumes and measurements of features for those 3D anatomical volumes. In some instances, a prediction network is trained on a set of image-segmentation pairs for image segmentation, wherein each image-segmentation pair in the set includes images from a sequence of images representing 3D anatomical volumes and segmentations of features for those 3D anatomical volumes.

[0108] In some instances, anatomical features include the fetal head, and the first and second measurements are associated with at least one of the circumference or length of the fetal head.

[0109] In some instances, anatomical features include the fetal head, and the first and second measurements are associated with at least one of the circumference or length of the fetal head.

[0110] The aspects of this disclosure can provide several benefits. For example, the use of deep learning-based frameworks for automated anatomical feature measurements can provide measurement assistance to clinicians, thereby reducing ultrasound examination time and / or user dependence. Therefore, the disclosed embodiments can provide more consistent and accurate measurement results compared to conventional measurements that rely on the user. Furthermore, reconstructing 3D volumes from acquired images to provide 3D information about anatomical features allows for more accurate measurement results. Additionally, the use of deep learning networks trained to create MPRs and perform measurements based on MPRs can further improve measurement accuracy, where measurements may not be limited to the imaging plane acquired during acquisition.

[0111] Those skilled in the art will recognize that the above-described apparatus, systems, and methods can be modified in various ways. Therefore, those skilled in the art will appreciate that the embodiments covered by this disclosure are not limited to the specific exemplary embodiments described above. In this regard, although illustrative embodiments have been shown and described, various modifications, alterations, and substitutions are contemplated in the foregoing disclosure. It should be understood that such changes can be made to the foregoing without departing from the scope of this disclosure. Therefore, it is appropriate that the appended claims be interpreted broadly in a manner consistent with this disclosure.

Claims

1. An ultrasound imaging system comprising: a processor circuit in communication with an ultrasound transducer array, the processor circuit configured to: receive, from the ultrasound transducer array, a set of images of a three-dimensional (3D) volume of an anatomical structure of a patient including an anatomical feature, wherein the set of images is associated with a plurality of imaging planes across the 3D volume of the patient’s anatomical structure including the anatomical feature; obtain first measurement data of the anatomical feature in a first image of the set of images; generate, based on position data of the ultrasound transducer array relative to the plurality of imaging planes, second measurement data for the anatomical feature in one or more images of the set of images by propagating the first measurement data from the first image to the one or more images; and output the second measurement data for the anatomical feature to a display in communication with the processor circuit.

2. The system of claim 1, wherein, the processor circuit configured to obtain the first measurement data is configured to: receive, from a user interface in communication with the processor circuit, the first measurement data including at least two measurement markers across the anatomical feature on the first image.

3. The system of claim 1, wherein, the processor circuit configured to generate the second measurement data is configured to: determine 3D spatial data for the first image and the one or more images based on the position data of the ultrasound transducer array; and propagate the first measurement data from the first image to the one or more images based on the 3D spatial data.

4. The system of claim 3, further comprising: a probe including the ultrasound transducer array and an inertial measurement tracker, wherein the processor circuit is configured to: receive, from the inertial measurement tracker, inertial measurement data associated with the ultrasound transducer array and the plurality of imaging planes, and wherein the processor circuit configured to determine the 3D spatial data is configured to: determine the position data of the ultrasound transducer array relative to the plurality of imaging planes based on the inertial measurement data and an inertial measurement to image transform.

5. The system of claim 1, wherein, the processor circuit is configured to: generate third measurement data for the anatomical feature based on the first measurement data and the second measurement data, wherein the third measurement data is associated with at least one of a first imaging plane of the plurality of imaging planes or a second imaging plane of the plurality of imaging planes that is different from the plurality of imaging planes within the 3D volume; and output the third measurement data to the display.

6. The system of claim 5, wherein, the second imaging plane intersects the first imaging plane.

7. The system of claim 6, wherein, the third measurement data includes at least one of the second measurement data, a distance between two measurement markers across the anatomical feature, a confidence metric of the first measurement data, a confidence metric of the second measurement data, an average of the first measurement data and the second measurement data, a variance of the first measurement data and the second measurement data, or a standard deviation of the first measurement data and the second measurement data.

8. The system of claim 7, further comprising a user interface in communication with the processor circuit and configured to provide a selection associated with the third measurement data.

9. The system of claim 1, wherein, the processor circuit configured to generate the second measurement data for the anatomical feature in the one or more images is configured to: propagate the first measurement data from the first image to the one or more images based on image segmentation.

10. The system of claim 1, wherein, the processor circuit configured to generate the second measurement data for the anatomical feature in the one or more images is configured to: propagate the first measurement data from the first image to the one or more images using a predictive network trained on at least one of image segmentation or feature measurements.

11. The system of claim 10, wherein, the predictive network is trained on a set of image-measurement pairs for the feature measurements, and wherein each image-measurement pair in the set of image-measurement pairs comprises an image in a sequence of images of a 3D anatomical volume and a measurement of a feature of the 3D anatomical volume for the image.

12. The system of claim 11, wherein, the predictive network is trained on a set of image-segmentation pairs for the image segmentation, wherein each image-segmentation pair in the set of image-segmentation pairs comprises an image in a sequence of images of a 3D anatomical volume and a segmentation of a feature of the 3D anatomical volume for the image.

13. An ultrasound imaging method, comprising: receiving, at a processor circuit in communication with an ultrasound transducer array, a set of images of a three-dimensional (3D) volume of an anatomical structure of a patient including an anatomical feature, wherein the set of images is associated with a plurality of imaging planes across the 3D volume of the patient’s anatomical structure including the anatomical feature; obtaining, at the processor circuit, first measurement data of the anatomical feature in a first image in the set of images; generating, at the processor circuit, second measurement data for the anatomical feature in one or more images of the set of images by propagating the first measurement data from the first image to the one or more images based on position data of the ultrasound transducer array relative to the plurality of imaging planes; and outputting the second measurement data for the anatomical feature to a display in communication with the processor circuit.

14. The method of claim 13, wherein, obtaining the first measurement data comprises: receiving, from a user interface in communication with the processor circuit, the first measurement data including at least two measurement markers across the anatomical feature.

15. The method of claim 13, wherein, generating the second measurement data comprises: determining 3D spatial data for the first image and the one or more images based on the position data of the ultrasound transducer array relative to the plurality of imaging planes; and propagating the first measurement data from the first image to the one or more images based on the 3D spatial data using a predictive network trained on at least one of image segmentation or feature measurements.

16. The method of claim 15, further comprising: receive, from an inertial measurement tracker in communication with the processor circuit, inertial measurement data associated with the ultrasound transducer array, and determine, based on the inertial measurement data and an inertial measurement to image transformation, the position data of the ultrasound transducer array relative to the first image and the one or more images.

Images