Automatic measurement point detection for anatomical measurements in anatomical images

Abstract

The system includes a display, a processor configured to communicate with the display, and a medical imaging device. The processor is configured to receive anatomical image frames acquired by the imaging device during live imaging and identify a plurality of measurement points of an anatomical feature within the anatomical image frames. The identification of the first measurement point is performed by a neural network trained to identify where the measurement points are located within the anatomical image frames, such that the measurement points are automatically identified without user input. The processor is also configured to generate measurements of the anatomical feature based on the measurement points and output a screen display based on the measurements.

Description

[Technical Field]

[0001] The subject matter described herein relates to devices, systems, and methods for automatically locating and measuring features (e.g., anatomical features or pathologies) during live imaging with a medical imaging device (e.g., an ultrasound probe). For example, a neural network can be trained to identify measurement points within an anatomical image, which can be used to generate measurements of the features. [Background technology]

[0002] Ultrasound images are often used for diagnostic purposes in clinic or hospital settings. For example, ultrasound is an imaging technology deployed at the point of care to aid in the assessment of fetal development. Important anatomical features such as head circumference, abdominal circumference, and femur length can be acquired in near real time by clinicians who either freeze a real-time ultrasound video stream at a particular image, place measurement points on the image, and then perform manual calculations or rely on software algorithms to perform calculations, for example, based on the distance between particular measurement points. The measurement points may be two endpoints (e.g., a measurement based on the linear distance between two points), or the measurement points may be three or more points. For example, three or more measurement points can define a curve, a curvilinear shape, a closed shape, etc. (e.g., an area, a circumference, a perimeter, etc.). In some cases, the measurement points may be referred to as calipers (e.g., in reference to physical calipers that may be used to obtain similar measurements from infants after birth). Summary of the Invention [Problem to be solved by the invention]

[0003] Such measurement processes can be imprecise, labor intensive, and highly dependent on the training level of the practitioner.

[0004] For example, an ultrasound examination protocol may require the clinician to scan the patient looking for specific anatomical structures of interest, pause the system, press the appropriate measurement button, and position the measurement calipers to measure the anatomical structure. The clinician typically takes three or more measurements for each anatomical structure and averages the results to arrive at an accurate measurement, which accounts for a large portion of the total examination time.

[0005] The information contained in this Background section of the specification, including any references cited herein and any description or discussion thereof, is included for technical reference purposes only and should not be considered as subject matter upon which the scope of the present disclosure should be bound. [Means for solving the problem]

[0006] An automatic measurement point detection system is disclosed that uses artificial intelligence / machine learning algorithms (e.g., neural networks). The neural network can be trained on anatomical image frames annotated by humans with measurement caliper positions and used to automatically identify and measure anatomical structures (depicted in medical images, such as ultrasound or x-ray images) in real time. For example, in an ultrasound image, a clinician or other user simply moves a transducer over the anatomical structure of interest, and the trained neural network identifies the anatomical structure, positions the measurement calipers in real time, and automatically calculates measurements without user intervention. Instead of the traditional three measurements performed for each anatomical structure during an examination, the automatic measurement point detection system completes hundreds of measurements at real-time frame rates. Once measurement convergence is identified, a user interface indicates that the most accurate measurement has been achieved.

[0007] The automatic measurement point detection system disclosed herein has particular, but not limited to, utility for measuring the size of anatomical features or pathologies in a real-time ultrasound video stream (e.g., live imaging as a clinician moves an ultrasound probe over a patient's body), such as may occur in a prenatal ultrasound examination. The automatic measurement point detection system detects features in individual frames of the video, automatically places measurement points on the video frames, performs calculations based on the measurement points, and generates and displays anatomical measurements based on the calculations.

[0008] One or more computer systems may be configured to perform particular operations or actions by having software, firmware, hardware, or a combination thereof installed on the system that, when in operation, causes or causes the system to perform the operation. One or more computer programs may be configured to perform particular operations or actions by containing instructions that, when executed by a data processing device, cause the device to perform the operation.

[0009] One general aspect includes a system including a display, a processor configured to communicate with the display, and a medical imaging device. The processor is configured to perform the steps of receiving first anatomical image frames acquired by the medical imaging device during live imaging, identifying a plurality of first measurement points of an anatomical feature in the first anatomical image frame while the live imaging is ongoing, the identification of the first measurement points being performed by a first neural network trained to identify where measurement points are to be located within the anatomical image frame such that the identification of the first measurement points is performed automatically without user input for positioning the plurality of first measurement points in the first anatomical image frame, generating first measurements of the anatomical feature for the first anatomical image frame based on the plurality of first measurement points, and outputting a screen display to the display based on the first measurements. Other aspects include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the operations of the method.

[0010] Implementations may include one or more of the following features. In some aspects, a processor, while live imaging is in progress, identifies a plurality of second measurement points of the anatomical feature in a second anatomical image frame acquired by the medical imaging device during live imaging, and generates a second measurement value of the anatomical feature in the second anatomical image frame based on the plurality of second measurement points. In some aspects, the second frame is acquired immediately after the first frame. In some aspects, the processor is configured to determine whether convergence of the measurements has occurred based on the first measurement value and the second measurement value, and the processor is configured to provide a screen display based on the determination of whether convergence of the measurements has occurred. In some aspects, the screen display may include progress of the convergence of the measurements. In some aspects, to determine whether convergence of the measurements has occurred, the processor is configured to determine whether the second measurement value is less than the first measurement value. In some aspects, if convergence of the measurements has occurred, the processor is configured to select the larger of the first measurement value or the second measurement value as the convergence measurement value, and the screen display may include the convergence measurement value. In some embodiments, the processor is configured to determine whether convergence of the measurements has occurred based on a difference between the first measurement and the second measurement. In some embodiments, the screen display may include a first anatomical image frame and a plurality of first measurement points overlaid on the first anatomical image frame. In some embodiments, the screen display may include the first anatomical image frame and an indication of the anatomical feature overlaid on the first anatomical image frame. In some embodiments, the processor is configured to identify an anatomical structure including the anatomical feature in the first anatomical image frame, and the processor is configured to generate the first measurement based on the identification of the anatomical structure. In some embodiments, the screen display may include the first anatomical image frame and an indication of the anatomical structure overlaid on the first anatomical image frame. In some embodiments, the identification of the anatomical structure is performed using a second neural network.In some aspects, the first neural network and the second neural network are the same neural network. In some aspects, the system may include a medical imaging device. Implementations of the described techniques may include hardware, a method or process on a computer-accessible medium, or computer software.

[0011] One general aspect includes a method that includes receiving, using a processor in communication with the medical imaging device, anatomical image frames acquired by the medical imaging device during live imaging. The method also includes identifying, while the live imaging is in progress, a plurality of measurement points of an anatomical feature in the anatomical image frames, where the identification of the measurement points is performed by a neural network trained to identify where the measurement points are located in the anatomical image frames, such that the identification of the first measurement points is performed automatically to identify the locations of the plurality of first measurement points in the anatomical image frames without user input. The method also includes generating measurements of the anatomical feature in the anatomical image frames based on the plurality of measurement points, and outputting a screen display based on the measurements on a display in communication with the processor. Other aspects include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the operations of the method.

[0012] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Forms for Carrying Out the Invention. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. A more extensive presentation of the features, details, utilities, and advantages of the automatic measure point detection system as defined in the claims is provided in the following description of various aspects of the disclosure and illustrated in the accompanying drawings.

[0013] Exemplary aspects of the present disclosure will now be described with reference to the accompanying drawings. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a schematic diagram of an ultrasound imaging system according to aspects of the present disclosure. [Figure 2] FIG. 2 is a schematic diagram of a processor circuit according to an aspect of the present disclosure. [Figure 3] 1 is a schematic diagram of a radiology video, video stream, or video clip, according to an aspect of the present disclosure. [Figure 4] 1 is a schematic diagram in flow diagram form of an exemplary ultrasound video feature measurement method according to aspects of the present disclosure. [Figure 5] 1 is a schematic diagram, in block diagram form, of at least a portion of an automatic measurement point detection system, in accordance with aspects of the present disclosure. [Figure 6] 1 is a schematic diagram in block diagram form of at least a portion of an automatic measurement point detection system according to an aspect of the present disclosure. [Figure 7A] FIG. 2 is a schematic diagram in block diagram form of a training mode for an object detector / analyzer, according to an aspect of the present disclosure. [Figure 7B] FIG. 1 is a schematic overview in block diagram form of a verification mode for an object detector / analyzer, according to an aspect of the present disclosure. [Figure 7C] FIG. 1 is a schematic diagram in block diagram form of an estimation mode or clinical use mode for an object detector / analyzer, according to an aspect of the present disclosure. [Figure 8] 1 is an exemplary screen display of an automatic measurement point detection system according to aspects of the present disclosure. [Figure 9] 1 is an exemplary training image for an automatic measurement point detection system, according to aspects of the present disclosure. [Figure 10] 1 is an exemplary object detection image of an automatic measurement point detection system according to aspects of the present disclosure. [Figure 11] 1 is an exemplary "detected objects" display of an automatic measurement point detection system, according to aspects of the present disclosure. [Figure 12]1 is an example of an estimated image generated by an automatic measurement point detection system, according to aspects of the present disclosure. [Figure 13] 1 is an exemplary object detection image of an automatic measurement point detection system according to aspects of the present disclosure. [Figure 14] 1 is an exemplary object detection image of an automatic measurement point detection system according to aspects of the present disclosure. [Figure 15] 1 is an exemplary filter control set for an automatic measurement point detection system, according to aspects of the present disclosure. [Figure 16] 1 is an exemplary training image of an automatic measurement point detection system, according to aspects of the present disclosure. [Figure 17] 1 is an exemplary object detection image of an automatic measurement point detection system according to aspects of the present disclosure. [Figure 18] 1 is an example of an estimated image generated by an automatic measurement point detection system, according to aspects of the present disclosure. [Figure 19] 1 is an exemplary training image for an automatic measurement point detection system, according to aspects of the present disclosure. [Figure 20] 1 is an example of an estimated image generated by an automatic measurement point detection system, according to aspects of the present disclosure. [Figure 21] 1 is an exemplary convergence progress display of an automatic measurement point detection system, according to aspects of the present disclosure. [Figure 22] 1 is an exemplary convergence progress display of an automatic measurement point detection system, according to aspects of the present disclosure. [Figure 23] 1 is an exemplary convergence progress display of an automatic measurement point detection system, according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0015] According to at least one aspect of the present disclosure, an automated measurement point detection system is provided that can measure the dimensions of anatomical features or pathologies in individual frames of a real-time ultrasound video stream, which may enable, for example, minimally trained users (including general practitioners, emergency physicians, and even patients) to obtain accurate anatomical measurements with minimal time investment and high confidence in the results.

[0016] The automatic measurement point detection system disclosed herein has particular utility for, but is not limited to, measuring anatomical structures in ultrasound procedures such as prenatal exams, pulmonary exams, etc. The automatic measurement point detection system detects features within individual frames of video, automatically places measurement points on the video frames, performs calculations based on the measurement points, generates anatomical measurements based on the calculations, and displays the measurements on the ultrasound console display, all in real time (e.g., at a high refresh rate of 60-100 Hz, such that the system completes measurements and annotations for one frame of a live video stream before the next frame is displayed).

[0017] The automatic measurement point detection system uses artificial intelligence (AI), deep learning (DL), or other machine learning (ML) techniques. The AI ​​model is trained using ultrasound video frames depicting specific anatomical structures (e.g., fetal head, abdomen, or femur) and annotated by humans with measurement points (calipers). Once the AI ​​model is trained, it can be incorporated into the ultrasound console's software to automatically identify and measure imaged anatomical structures in real time. The clinician or other user simply moves and / or reorients the ultrasound probe over the body part containing the anatomical structure of interest, and the trained AI model identifies the anatomical structure, calculates the location of the measurement points in real time (e.g., identifies the location of the measurement points (e.g., coordinates of the pixels that make up the measurement points among the pixels that form the anatomical image frame)), and automatically calculates the desired anatomical measurements. No intervention from the user is required beyond acquiring the relevant ultrasound images. The automatic measurement point detection system can complete hundreds of measurements at real-time frame rates, and these measurements are analyzed for convergence and plane alignment. Once convergence of the measurements is identified (e.g., through detection of maximum anatomical structure length or elliptical symmetry), the user interface will indicate that the most accurate measurement has been achieved, at which point the test may be completed or the user may move on to another anatomical feature for which measurement is desired.

[0018] The key elements of an automated measurement point detection system are (1) an AI neural network model trained on both ultrasound anatomical structures and human-generated measurement points (or their bounding boxes); (2) an error filtering step to ensure that measurement points are within and fit within the detected anatomical structure box (e.g., a femoral endpoint cannot be located in the head); (3) a computational engine to derive the desired anatomical measurements from the identified measurement points; (4) tests to detect measurement convergence (e.g., a distance measurement such as femoral length may be considered complete when a maximum distance is achieved (e.g., several seconds have passed without a longer measurement being detected). For more complex measurements such as circumference or area, the system may check for elliptical symmetry, maximum circumference, maximum area, and / or the presence of specific anatomical features indicative of an appropriate measurement plane); and (5) user interface graphics to display measurements and status.

[0019] The automated measurement point detection system can be used with live and / or recorded ultrasound video streams. An exemplary deep learning model has been trained to measure fetal femur length, head circumference, and abdominal circumference; the model can also be trained using live or recorded video from a medical imaging device using other imaging modalities, including, but not limited to, visible light, angiography / fluoroscopy (X-ray), computer-aided tomography (CAT) scan, magnetic resonance imaging (MRI), intravascular ultrasound (IVUS), etc.

[0020] Figures 9, 16, and 19 below are example images passed to the AI ​​training step, with fetal anatomical structures and measurement endpoints indicated by bounding boxes. The neural network is "trained" by feeding it many (e.g., tens, hundreds, or thousands) of images annotated with anatomical structure identification and measurement points. As the training process progresses, the neural network model "learns" the locations of anatomical structures and measurement points and can accurately predict this information on new ultrasound images.

[0021] One distinct advantage of training using human-annotated anatomical structures and measurement endpoints is the ability to validate results and filter potential errors. Upon receiving detection data from the AI ​​model, filtering rules are applied. For example, the anatomical box itself (e.g., head, abdomen, femur) and the measurement point must be detected, and the measurement point must actually fall within the anatomical box. For example, for a femur measurement to be complete, two femoral endpoints must be present within the femoral anatomical box. For a head circumference measurement to be complete, two biparietal diameter (BPD) and two occipitofrontal diameter (OFD) endpoints must be present within the head anatomical box, and the thalamic structures of the head must be detected to identify the correct plane location. These rules ensure that the neural network calculates measurements in the correct imaging plane. Other filtering rules may be used instead or in addition.

[0022] In one aspect, a user initiates an examination on the ultrasound system and begins scanning a patient. Once an anatomical structure is visualized, measurements of that anatomical structure are automatically displayed and updated in real time. As the user moves the ultrasound probe in and out of the imaging plane, the measurements stabilize and the user is presented with a final result. For anatomical structures requiring area measurements, alignment graphics indicating plane alignment can be used to display multiple components of the measurement, such as length and width, in real time, based on factors such as the ratio of length to width and the appearance of a reference anatomical structure in the desired imaging plane.

[0023] In another aspect, an automatic measurement point detection system can be configured to provide a single measurement: once the user visualizes the desired anatomy and freezes the image, the single measurement is automatically displayed and editable as in a traditional workflow.

[0024] The present disclosure substantially facilitates obtaining anatomical measurements from live or recorded radiology video by automatically placing measurement points and performing measurement calculations without requiring human intervention (e.g., without the need for a clinician to pause the live video stream to place measurement points and calculate measurements). The automated measurement point detection system disclosed herein, implemented on a processor in communication with the ultrasound probe, provides a practical improvement in the ability of untrained or inexperienced clinicians to obtain accurate anatomical measurements from radiology video. This improved anatomical structure measurement transforms a subjective, time-intensive process that relies heavily on expert experience into an objective and repeatable process, without the routine need to train clinicians, such as emergency department personnel, to recognize specific anatomical structures in the video stream and perform accurate measurements. This unconventional approach improves the functionality of ultrasound imaging systems by providing reliable measurements of anatomical features or pathologies.

[0025] The automatic measurement point detection system may be implemented as a process at least partially viewable on a display, accept user input from a keyboard, mouse, or touchscreen interface, and be operated by a control process executing on a processor that communicates with one or more sensor probes. In this regard, the control process performs certain operations in response to different inputs or selections made at different times. Specific structures, functions, and operations of processors, displays, sensors, and user input systems are known in the art; others are recited herein to specifically enable novel features or aspects of the present disclosure.

[0026] These descriptions are provided for illustrative purposes only and should not be considered to limit the scope of the automatic measurement point detection system. Certain features may be added, removed, or modified without departing from the spirit of the claimed subject matter.

[0027] For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the principles of the present disclosure. Nevertheless, it will be understood that no limitations on the scope of the present disclosure are intended. Any changes and further modifications to the described apparatus, systems, and methods, and any further applications of the principles of the present disclosure, as would normally occur to one skilled in the art to which the present disclosure pertains, are fully contemplated and included within the present disclosure. In particular, it is fully contemplated that features, components, and / or steps described with respect to one embodiment may be combined with features, components, and / or steps described with respect to other embodiments of the present disclosure. However, for the sake of brevity, multiple iterations of these combinations will not be described separately.

[0028] 1 is a schematic diagram of an ultrasound imaging system 100 according to an embodiment of the present disclosure. The ultrasound imaging system 100 may be used, for example, to train an automatic measurement point detection system or to acquire ultrasound video clips that may be analyzed and enhanced in a clinical setting (whether in real time, near real time, or as post-processing of stored video clips) by an automatic measurement point detection system.

[0029] The ultrasound imaging system 100 is used to scan a region or volume of a subject's body. The subject may include a patient undergoing an ultrasound imaging procedure, or any other person, or any suitable living or non-living object or structure. The ultrasound imaging system 100 includes an ultrasound imaging probe 110 that communicates with a host 130 via a communications interface or link 120. The probe 110 may include a transducer array 112, a beamformer 114, a processor circuit 116, and a communications interface 118. The host 130 may include a display 132, a processor circuit 134, a communications interface 136, and a memory 138 that stores subject information.

[0030] In some embodiments, the probe 110 is an external ultrasound imaging device including a housing 111 configured for handheld operation by a user. The transducer array 112 can be configured to acquire ultrasound data while a user grasps the housing 111 of the probe 110, with the transducer array 112 positioned adjacent to or in contact with the skin of the subject. The probe 110 is configured to acquire ultrasound data of anatomical structures within the subject's body while the probe 110 is positioned outside the subject's body, for general imaging such as for abdominal imaging, liver imaging, etc. In some embodiments, the probe 110 can be an external ultrasound probe, a transthoracic probe, and / or a curved array probe.

[0031] In other aspects, the probe 110 can be an internal ultrasound imaging device and can include a housing 111 configured to be placed within a lumen in a subject's body for general imaging, such as for abdominal imaging, liver imaging, etc. In some aspects, the probe 110 can be a curved array probe. The probe 110 can be in any suitable form for any suitable ultrasound imaging application, including both external and internal ultrasound imaging.

[0032] In some aspects, aspects of the present disclosure may be implemented using medical images of a subject acquired using any suitable medical imaging device and / or modality, including, for example, x-ray images (angiograms, fluoroscopic images, images with or without contrast) acquired by medical imaging devices such as ultrasound imaging devices, x-ray imaging (CT) images acquired by CT imaging devices, positron emission tomography (PET-CT) images acquired by PET-CT imaging devices, magnetic resonance images (MRI) acquired by MRI imaging devices, single photon emission computed tomography (SPECT) images acquired by SPECT imaging devices, optical coherence tomography (OCT) images acquired by OCT imaging devices, intravascular photoacoustic (IVPA) images acquired by IVPA imaging devices, etc. The medical imaging device may acquire medical images outside, spaced apart from, adjacent to, in contact with, and / or within the subject's body.

[0033] In an ultrasound imaging device, the transducer array 112 emits ultrasound signals toward an anatomical object 105 of a subject and receives echo signals that are reflected from the object 105 and return to the transducer array 112. The ultrasound transducer array 112 can include any suitable number of acoustic elements, including one or more acoustic elements and / or multiple acoustic elements. In some examples, the transducer array 112 includes a single acoustic element. In some examples, the transducer array 112 can include an array of acoustic elements having any number of acoustic elements in any suitable configuration. For example, the transducer array 112 can include between 1 and 10,000 acoustic elements, including values ​​such as 2 acoustic elements, 4 acoustic elements, 36 acoustic elements, 64 acoustic elements, 128 acoustic elements, 500 acoustic elements, 812 acoustic elements, 1,000 acoustic elements, 3,000 acoustic elements, 8,000 acoustic elements, and / or other greater and lesser values. In some cases, the transducer array 112 may include an array of acoustic elements having any number of acoustic elements in any suitable configuration, such as a linear array, a planar array, a curved array, a curvilinear array, a circumferential array, an annular array, a phased array, a matrix array, a one-dimensional (1D) array, a 1.x-dimensional array (e.g., a 1.5D array), or a two-dimensional (2D) array. The array of acoustic elements (e.g., one or more rows, one or more columns, and / or one or more orientations) may be controlled and activated uniformly or independently. The transducer array 112 may be configured to acquire one-dimensional, two-dimensional, and / or three-dimensional images of the subject's anatomy. In some embodiments, the transducer array 112 may include piezoelectric micromachined ultrasound transducers (PMUTs), capacitive micromachined ultrasound transducers (CMUTs), single crystal, lead zirconate titanate (PZT), PZT composites, other suitable transducer types, and / or combinations thereof.

[0034] The object 105 may include any anatomical structure or anatomical feature, such as a kidney, liver, and / or any other anatomical structure of a subject. The present disclosure may be implemented in the context of any number of anatomical locations and tissue types, including, but not limited to, organs including the liver, kidney, gallbladder, pancreas, and lungs, ducts, intestines, brain, dural sac, spinal cord, and nervous system structures including peripheral nerves, urinary tract, and blood vessels, blood, abdominal organs, and / or valves within other systems of the body. In some embodiments, the object 105 may include a tumor, cyst, lesion, hemorrhage, or malignancy such as a blood pool within any part of the human anatomy. The anatomical structure may be a blood vessel, such as an artery or vein, of the subject's vascular system, including the cardiovascular system, peripheral vasculature, neurovascular system, renal blood vessels, and / or any other suitable lumen within the body. In addition to natural structures, the present disclosure may be implemented in the context of artificial structures, such as, but not limited to, heart valves, stents, shunts, filters, implants, and other devices.

[0035] The beamformer 114 is coupled to the transducer array 112. The beamformer 114 controls the transducer array 112, for example, to transmit ultrasound signals and receive ultrasound echo signals. In some aspects, the beamformer 114 can apply time delays to signals transmitted to individual acoustic transducers in the array in the transducer 112 so that the acoustic signals are steered in any suitable direction propagating away from the probe 110. The beamformer 114 can further provide image signals to the processor circuit 116 based on the response of the received ultrasound echo signals. The beamformer 114 can include multiple stages of beamforming. Beamforming can reduce the number of signal lines for coupling to the processor circuit 116. In some aspects, the transducer array 112 in combination with the beamformer 114 may be referred to as an ultrasound imaging component.

[0036] The processor 116 is coupled to the beamformer 114. The processor 116 may also be described as a processor circuit, which may include other components in communication with the processor 116, such as memory, the beamformer 114, a communication interface 118, and / or other suitable components. The processor 116 may include a central processing unit (CPU), a graphical processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a controller, a field-programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 116 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 116 is configured to process the beamformed image signal. For example, the processor 116 may perform filtering and / or quadrature demodulation to condition the image signal. Processors 116 and / or 134 may be configured to control array 112 to acquire ultrasound data related to object 105 .

[0037] The communication interface 118 is coupled to the processor 116. The communication interface 118 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 118 may include hardware and / or software components that implement a particular communication protocol suitable for transferring signals to the host 130 over the communication link 120. The communication interface 118 may be referred to as a communication device or a communication interface module.

[0038] Communications link 120 may be any suitable communications link. For example, communications link 120 may be a wired link such as a Universal Serial Bus (USB) link or an Ethernet link. Alternatively, communications link 120 may be a wireless link such as an Ultra Wideband (UWB) link, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 WiFi link, or a Bluetooth link.

[0039] In the host 130, a communication interface 136 can receive the image signal. The communication interface 136 can be substantially similar to the communication interface 118. The host 130 can be any suitable computing and display device, such as a workstation, a personal computer (PC), a laptop, a tablet, or a mobile phone.

[0040] The processor 134 is coupled to the communication interface 136. The processor 134 may also be described as a processor circuit, which may include other components in communication with the processor 134, such as the memory 138, the communication interface 136, and / or other suitable components. The processor 134 may be implemented as a combination of software and hardware components. The processor 134 may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, an FPGA device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 134 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 134 may be configured to generate image data from image signals received from the probe 110. The processor 134 may apply advanced signal processing and / or image processing techniques to the image signals. In some embodiments, the processor 134 can form a three-dimensional (3D) volumetric image from the image data. In some embodiments, the processor 134 can perform real-time processing on the image data to provide streaming video of ultrasound images of the object 105. In some embodiments, the host 130 includes a beamformer. For example, the processor 134 can be part of and / or otherwise in communication with such a beamformer. The beamformer in the host 130 can be a system beamformer or a main beamformer (which provides one or more subsequent stages of beamforming), while the beamformer 114 is a probe beamformer or a microbeamformer (which provides one or more initial stages of beamforming).

[0041] The memory 138 is coupled to the processor 134. The memory 138 may be any suitable storage device, such as cache memory (e.g., cache memory of the processor 134), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), field programmable gate array read-only memory (PROM), erasable field programmable gate array read-only memory (EPROM), electrically erasable field programmable gate array read-only memory (EEPROM), flash memory, solid-state memory devices, hard disk drives, solid-state drives, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0042] Memory 138 may be configured to store subject information, measurements, data, or computer-readable instructions, such as files, code, software, or other applications, relating to the subject's medical history, history of procedures performed, anatomical or biological characteristics, properties, or medical conditions associated with the subject, as well as any other suitable information or data. Memory 138 may be located within host 130. Subject information may include other forms of medical history, such as, but not limited to, measurements, data, files, ultrasound images, ultrasound videos, and / or any imaging information relating to the subject's anatomy. Subject information may include parameters related to the imaging procedure, such as an anatomical scan window, probe orientation, and / or subject position during the imaging procedure. Memory 138 may also be configured to store information related to training and implementing machine learning algorithms (e.g., neural networks) and / or implementing image recognition algorithms for detecting / segmenting anatomical structures, image quantification algorithms, and / or image acquisition guidance algorithms, including those described herein.

[0043] The display 132 is coupled to the processor circuit 134. The display 132 may be a monitor or any suitable display. The display 132 is configured to display ultrasound images, image video, and / or any imaging information of the object 105.

[0044] The ultrasound imaging system 100 may be used to assist a sonographer in performing an ultrasound scan. The scan may be performed at a point-of-care location. In some examples, the host 130 is a console or a mobile cart. In some cases, the host 130 may be a mobile device, such as a tablet, a mobile phone, or a portable computer. During an imaging procedure, the ultrasound system may acquire ultrasound images of a particular region of interest within a subject's anatomy. The ultrasound imaging system 100 may then analyze the ultrasound images to identify various parameters related to the acquisition of the images, such as the scan window, the probe orientation, the subject's position, and / or other parameters. The ultrasound imaging system 100 may then store the images and their associated parameters in memory 138. During subsequent imaging procedures, the ultrasound imaging system 100 may retrieve previously acquired ultrasound images and associated parameters for display to the user, which may be used to guide the user of the ultrasound imaging system 100 to use the same or similar parameters in subsequent imaging procedures, as described in more detail below.

[0045] In some embodiments, the processor 134 may utilize a deep learning-based predictive network to identify parameters of the ultrasound image, including the anatomical scan window, probe orientation, subject position, and / or other parameters. In some embodiments, the processor 134 may receive metrics or perform various calculations regarding the physiological state of the imaged region of interest or subject during the imaging procedure. These metrics and / or calculations may also be displayed to the sonographer or other user via the display 132.

[0046] Before continuing, please note that the above examples are provided for illustrative purposes and are not intended to be limiting. Other apparatus and / or device configurations may be utilized to perform the operations described herein.

[0047] 2 is a schematic diagram of a processor circuit 250 according to an embodiment of the present disclosure. The processor circuit 250 may be implemented on the ultrasound imaging system 100, or on another device or workstation (e.g., a third-party workstation, a network router, etc.), or on a cloud processor or other remote processing unit as needed to implement the methods. As shown, the processor circuit 250 may include a processor 260, a memory 264, and a communication module 268. These elements may communicate with each other directly or indirectly, for example, via one or more buses.

[0048] Processor 260 may include any combination of a central processing unit (CPU), digital signal processor (DSP), ASIC, controller, or general-purpose computing device, reduced instruction set computing (RISC) device, application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other related logic device, including mechanical and quantum computers. Processor 260 may also comprise another hardware device, firmware device, or any combination thereof, configured to perform the operations described herein. Processor 260 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0049] The memory 264 may include cache memory (e.g., cache memory of the processor 260), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), field programmable gate array read-only memory (PROM), erasable field programmable gate array read-only memory (EPROM), electrically erasable field programmable gate array read-only memory (EEPROM), flash memory, solid-state memory devices, hard disk drives, other forms of volatile and non-volatile memory, or a combination of different types of memory. In one aspect, the memory 264 includes a non-transitory computer-readable medium. The memory 264 may store instructions 266. The instructions 266 may include instructions that, when executed by the processor 260, cause the processor 260 to perform the operations described herein. The instructions 266 may also be referred to as code. The terms "instructions" and "code" should be interpreted broadly to include any type of computer-readable statement. For example, the terms "instructions" and "code" may refer to one or more programs, routines, subroutines, functions, procedures, etc., and "instructions" and "code" may include a single computer-readable statement or many computer-readable statements.

[0050] The communications module 268 may include any electronic and / or logic circuitry for facilitating the direct or indirect communication of data between the processor circuit 250 and another processor or device. In this regard, the communications module 268 may be an input / output (I / O) device. In some instances, the communications module 268 facilitates direct or indirect communication between various elements of the processor circuit 250 and / or the ultrasound imaging system 100. The communications module 268 may communicate within the processor circuit 250 via a number of methods or protocols. Serial communications protocols include the United States Serial Protocol Interface (US SPI), the Interface Integrated Circuit (IIC), and the like. 2Serial and parallel protocols may include, but are not limited to, RS-232, RS-485, Controller Area Network (CAN), Ethernet, Airborne Radio, 429 (ARINC 429), MODBUS, Military Standard 1553 (MIL-STD-1553), or any other suitable method or protocol. Parallel protocols include, but are not limited to, Industry Standard Architecture (ISA), Advanced Technology Attachment (ATA), Small Computer System Interface (SCSI), Peripheral Component Interconnect (PCI), Institute of Electrical and Electronics Engineers 488 (IEEE-488), IEEE-1284, and other suitable protocols. If necessary, serial and parallel communications may be bridged by a universal asynchronous receiver transmitter (UART), universal synchronous receiver transmitter (USART), or other suitable subsystem.

[0051] External communication (including, but not limited to, software updates, firmware updates, model sharing between the processor and a central server, or reads from the ultrasound imaging system 100) is achieved using any suitable wireless or wired communication technology, such as a Universal Serial Bus (USB), micro USB, Lightning, or FireWire interface; a cable interface such as Bluetooth, Wi-Fi, ZigBee, or Li-Fi; or a cellular data connection such as 2G / GSM (Global System for Mobile Communications), 3G / UMTS (Universal Mobile Telecommunications System), 4G, Long Term Evolution (LTE), WiMax, or 5G. For example, a Bluetooth Low Energy (BLE) radio can be used to establish connectivity with cloud services, transmit data, and receive software patches. The controller may be configured to communicate with a remote server or a local device such as a laptop, tablet, or handheld device, or may include a display capable of showing status variables and other information. Information may also be transferred on physical media, such as a USB flash drive or memory stick.

[0052] FIG. 3 is a schematic diagram of a radiology (e.g., ultrasound) video stream 310 (whether live or recorded) according to an embodiment of the present disclosure. The ultrasound video stream 310 includes a number of frames 320. In one example, the ultrasound video stream 310 is captured at a frame rate of 30, 60, or 100 frames per second. Each frame has spatial dimensions representing a 2D cross-section of the object being imaged by the ultrasound imaging system as a Y-axis or height 330 and an X-axis or width 340. Additionally, the ultrasound video stream 310 includes a depth or time axis 350 representing the time at which each frame 320 of the video stream 310 was captured. Thus, the ultrasound video stream 310 may be considered a 3D data structure. The video stream 310 may be of any suitable modality having 2D image frames over time, such as x-ray, MRI, CT, etc.

[0053] 4 is a schematic diagram, in flow diagram form, of an exemplary automatic measurement point detection method 400 according to an aspect of the present disclosure. It should be understood that the steps of method 400 may be performed in a different order than that shown in FIG. 4, additional steps may be provided before, between, and after steps, and / or some of the described steps may be substituted or removed in other aspects. One or more of the steps of method 400 may be delivered by one or more devices and / or systems described herein, such as component elements of ultrasound system 100 and / or processor circuit 250.

[0054] In step 410, the method 400 includes receiving image frames from a live or recorded ultrasound video stream (or other radiology video stream).

[0055] In step 420, method 400 includes detecting / identifying an anatomical structure (e.g., fetal head, abdomen, or femur) using a neural network and placing measurement points on the image frame relative to the anatomical structure. Depending on the embodiment, the measurement points may be placed by the same neural network that detects the anatomical structure or by a different neural network.

[0056] Anatomical detection based on machine learning (ML) or other neural network (NN)-based artificial intelligence (AI) can be, for example, by object detection or bounding box detection, classification, segmentation, or other related means. Examples of classification networks can be found, for example, in U.S. Provisional Patent Application No. 63 / 293,232, filed December 23, 2021, entitled "Method and System for Clinically Assessing Lung Ultrasound," and U.S. Provisional Patent Application No. 63 / 294,501, filed December 29, 2021, entitled "Reconstruction Filter-Independent Machine Learning Image Processing," each of which is incorporated by reference as if fully set forth herein. An example of bounding box object detection can be found in Indian Patent Application No. 202141034243, filed July 29, 2021, entitled "Generation of Position Data" (International Application No. PCT / EP2022 / 070410). Examples of image or video segmentation can be found, for example, in U.S. Provisional Patent Application No. 63 / 325,660, filed March 31, 2022, entitled "Method and System for Ultrasound-Based Structure Localization Using Images and User Input," and U.S. Patent Publication No. 2022 / 0198669, entitled "Segmentation and View Guidance in Ultrasound Imaging and Related Devices, Systems, and Methods," each of which is incorporated by reference as if fully set forth herein.

[0057] At step 430, method 400 includes filtering the anatomical structure detection points and measurement points for reasonableness. For example, an anatomical structure detection may be considered valid if a specific relevant anatomical feature can be identified in the image frame, or if the eccentricity of an elliptical feature falls within a specific specified range; otherwise, it may be considered invalid. In some embodiments, an invalid anatomical structure detection can automatically invalidate any measurement point placed relative to that anatomical structure. A measurement point may be considered valid if it falls within the bounding box of a valid detected anatomical structure or within a specific region of the detected anatomical structure. For example, a femoral end point may be considered invalid if it occurs outside the femur bounding box or near the center of the femur rather than near one of the ends. Such filtering provides a check and balance against spurious predictions by the neural network, so that the system is configured to generate a first measurement based on the identification of the anatomical structure.

[0058] In step 440, method 400 includes calculating one or more measurements using the filtered measurement points. The measurements may include, for example, the length, width, area, diameter, or circumference of the identified anatomical structure, or other anatomical measurements. Measurements may be made, for example, directly using the measurement points (e.g., the length / distance between two measurement points). For example, femur length may be the length / distance between femoral endpoints. In other cases, measurements may combine multiple measurements. For example, head circumference may be a combination of biparietal diameter (BPD, a first length / distance between two measurement points) and occipitofrontal diameter (AFD, a second length / distance between two measurement points). For example, HC = 1.62 × (BPD + OFD). Similarly, abdominal circumference (AC) may be obtained from the anterior-posterior abdominal diameter (APAD) and transverse abdominal diameter (TAD) by AC = π(APAD + TAD) / 2 = 1.57(APAD + TAD). The displayed measurement may be the numeric value of the measurement itself or a derived value calculated using the first measurement. For example, OFD and BPD may be obtained as direct measurements (e.g., distance), while HC is a derived value calculated using OFD and BPD. The displayed measurement may also be a visual depiction of the measurement points (with or without lines / curves between the points) or may be or include a convergence progress indicator as discussed in Figures 21-23. The measurements may be two or more different measurements of the same anatomical feature. Two or more consecutive measurements of the same anatomical feature may be the same or different from each other. Values ​​may differ, for example, because different image frames show different portions of the same anatomical structure (e.g., different image planes) rather than different anatomical features being measured.

[0059] In one example, one or several metrics are calculated from the detections. The metrics may be hard-coded into the system or may be selectable in real time or near real time to represent clinically relevant parameters derived from the detections. For example, in a screening / triage context, an operator may be interested in picking up features of any size as long as they are detected with sufficient confidence. In this setting, a metric defined as the maximum confidence of all detections of a feature type (but regardless of the detection area) may be appropriate. Alternatively, in a diagnostic situation, an operator may already know that very small findings are not clinically significant, but larger findings may indicate pathology. Therefore, the metric may be defined as the maximum area of ​​all detection bounding boxes, or the maximum of the product of the area and confidence of all detections. In this way, the metric is not sensitive to small findings (even those with high confidence).

[0060] In step 450, method 400 includes determining convergence of the measurements and / or plane alignment of the image frames relative to the anatomical structure. For example, as different measurements are calculated with each cycle of the method (e.g., each image frame of an ultrasound video stream), the method may include tracking the maximum value for the measurement, and may consider the measurement "unconverged" when a certain time or number of frames has passed without a larger value being detected, or if the maximum value is still changing, and therefore the user needs to continue acquiring live ultrasound video to improve the measurement. In some aspects, the method may use cues such as the eccentricity of the ellipse or the presence of a particular anatomical feature to determine whether the image frame was captured at the desired position, alignment, or depth, and may consider the measurement "unconverged" if the image frame containing the maximum value is improperly positioned or aligned. Depending on the implementation, guidance may be provided to the user to move or reorient the ultrasound probe to obtain better measurements, or the user interface may simply indicate that the measurements have not yet converged, i.e., a cue to the user to continue moving the ultrasound probe.

[0061] In step 460, method 400 includes displaying the measurements to the user (e.g., adjacent to or superimposed on the current image frame of the video stream). Non-limiting example screen displays can be found in Figures 8-23 below. Because the method can be performed in real time, execution can then return to step 410 to await receipt of the next image frame in the video stream.

[0062] Flow diagrams are provided herein for illustrative purposes, and those skilled in the art will recognize numerous variations that nonetheless fall within the scope of the present disclosure. For example, the logic of a flow diagram may be shown sequentially. However, similar logic may be parallel, massively parallel, object-oriented, real-time, event-driven, cellular automaton, or otherwise, while achieving the same or similar functions. To perform the methods described herein, a processor may divide each of the steps described herein into multiple machine instructions, and these instructions may be executed within a single processor or across multiple processors at rates of hundreds, thousands, millions, or billions per second. Such rapid execution may be necessary to perform the methods described herein in real time or near real time. For example, to measure anatomical features in a real-time ultrasound video stream, steps 410 through 440 may require instruction execution faster than the frame rate of the video (e.g., 30, 60, or 100 instruction executions per second).

[0063] 5 is a schematic diagram, in block diagram form, of at least a portion of an automatic measurement point detection system, according to an embodiment of the present disclosure. Image frames 320 of an ultrasound video stream are received by an object detector / analyzer 520 (e.g., a deep learning network or other machine learning neural network or artificial intelligence), which performs feature detection (e.g., detection of specified anatomical features) on the image frames 320 and determines the locations of measurement points. In some embodiments, the object detector / analyzer 520 comprises two separate neural networks (e.g., an object detector and an object analyzer), although a single neural network may be easier to train and more computationally efficient. The object detector / analyzer can output annotated video frames 530 marking the measured anatomical features 540 and measurement points 550.

[0064] The object detector / analyzer 520 may implement or include any suitable type of learning network. For example, in some embodiments, the object detector / analyzer 520 may include a neural network such as a convolutional neural network (CNN). In addition, the convolutional neural network may additionally or alternatively be an encoder-decoder type network or may utilize a backbone architecture based on other types of neural networks, such as an object detection network, a classification network, etc. One example of a backbone network is the darknet YOLO backbone (e.g., Yolov3), which can be used for object detection. A CNN may, for example, include a set of N convolutional layers, where N may be any positive integer. Fully connected layers may be omitted when the CNN is the backbone. A CNN may also include a max-pooling layer and / or an activation layer. Each convolutional layer may include a set of filters configured to extract features from an input (e.g., from the image frame 320). The value N and size of the filters may vary depending on the embodiment. In some examples, convolutional layers can utilize any nonlinear activation function, such as a leaky-corrected nonlinear (ReLU) activation function and / or batch normalization. Max-pooling layers gradually reduce the high-dimensional output to the desired resulting dimensionality (e.g., bounding boxes of detected features).

[0065] The fully connected layer may be referred to as a perceptual layer or a perceptual layer. In some aspects, perceptual / perceptual and / or fully connected layers may be found in the object detector / analyzer 520 (e.g., a multi-layer perceptron) to enable downstream processing (e.g., detection, segmentation, etc.). The object detector / analyzer 520 may also include max pooling layers and / or activation layers. The max pooling layers gradually reduce the high-dimensional output to the desired result dimensionality (e.g., a bounding box for a region of interest).

[0066] These descriptions are included for illustrative purposes, and one skilled in the art will understand that other types of AI learning models having similar or dissimilar features to those described above may be used instead or in addition without departing from the spirit of the present disclosure.

[0067] The annotated image frame 530, or the information used to generate the annotations, is then sent to an error filtering block 560, which filters the anatomical structure detection points and measurement points for validity as described above. For example, an anatomical structure detection may be considered valid if a specified anatomical landmark is identified in the image frame, or if the eccentricity of an ellipse feature is within a certain specified range. A measurement point may be considered valid if it falls within the correct (and valid) anatomical structure.

[0068] The filtered measurement points are then sent to a measurement calculation block 570 which performs calculations such as determining the distance between two measurement points, the circumference or area of ​​an ellipse defined by four measurement points, or the like.

[0069] The measurement calculation block generates one or more measurements 580, which are then shown on a display 590 as annotated image frame 530, original image frame 320, original image frame 320 with bounding box, or otherwise, as shown below.

[0070] Block diagrams are provided herein for illustrative purposes, and those skilled in the art will recognize numerous variations that nonetheless fall within the scope of the present disclosure. For example, block diagrams may show a particular arrangement of components, modules, services, steps, processes, or layers that result in a particular data flow. It should be understood that some aspects of the systems disclosed herein may include additional components, some components shown may be absent from some aspects, and the arrangement of components may differ from that shown, resulting in different data flows while still performing the methods described herein.

[0071] 6 is a schematic diagram, in block diagram form, of at least a portion of an automatic measurement point detection system, according to an embodiment of the present disclosure. Image frames 320 of a radiology video stream 310 result in a series of measurements 580 (e.g., one measurement per image frame 320). The measurements 580 are received by a convergence and plane alignment block 630, which determines the convergence of the measurements and / or plane alignment of the image frame relative to the anatomical structure, as described above. For example, as different measurements are calculated for each image frame, the convergence and plane alignment block 630 can track the maximum value for the measurements and use cues such as the eccentricity of the ellipse or the presence of certain anatomical features to determine whether the image frame was captured at the desired position, alignment, or depth.

[0072] The convergence and plane alignment block 630 then generates a graphical display 640 that can be overlaid on the image frame or displayed adjacent to the image frame. For example, if the image frame containing the maximum value is improperly positioned or aligned, or if the maximum calculated value of the measurement is still increasing over time, the measurement may be indicated textually or graphically as "not converged" or "in progress." Depending on the embodiment, the graphical display 640 may also provide guidance to the user for moving or reorienting the ultrasound probe.

[0073] 7A is a schematic diagram, in block diagram form, of a training mode 700 for an object detector / analyzer 520, according to an embodiment of the present disclosure. A set of training data 705a, which in the example shown in FIG. 7A includes an ultrasound video stream with hand-marked anatomical structure localizations and measurement points (e.g., surrounded by bounding boxes), is fed to an untrained object detector / analyzer 710a in an iterative training process familiar to those skilled in the art.

[0074] In particular, for object detection and measurement point placement using convolutional neural networks, a large number of sample images are manually annotated by an expert to depict the localization of features of interest. The parameters of the network model (e.g., the weights in each artificial neuron) are initialized with an initial value A, which can be a random value, or with results from training on a previous dataset. In an iterative process, the network is used to make detection estimates on training images, the results are compared with the ground truth annotations, and an optimizer is used to adjust the network parameters B until a detection accuracy metric is maximized.

[0075] Thus, the output of this training process 700 is a trained object detector / analyzer 710b, where the parameters B (eg, weightings) are optimized for detection of features in the training video 705a.

[0076] 7B is a schematic diagram, in block diagram form, of a verification mode 702 for an object detector / analyzer 520, according to an aspect of the present disclosure. In verification mode, a set of manually annotated verification videos (e.g., videos that include bounding boxes around any anatomical structures or measurement points identified by experts in each frame of each video) are fed to the trained object detector / analyzer 710b to determine whether human-identified features in the verification videos 705b are detected to a desired level of accuracy by the trained object detector / analyzer 710b.

[0077] In some cases, the performance of the trained object detector / analyzer 710b may be deemed below a desired accuracy level. In this case, the parameters B (e.g., weighting) of the trained object detector / analyzer 710b may be adjusted until the detection accuracy on the validation data set (or the validation data set + the training data set) reaches the desired accuracy. In such a case, the output of the validation process may be a trained object detector / analyzer 520 that may be identical to the trained object detector / analyzer 710b except for the adjusted parameters C (e.g., weighting). In other cases, the performance of the trained object detector / analyzer 710b may be deemed adequate, and therefore no adjustments to the parameters are made, and the trained object detector / analyzer 520 may be identical to the trained object detector / analyzer 710b (e.g., using the same weighting).

[0078] 7C is a schematic diagram, in block diagram form, of an estimation mode or clinical use mode 704 for an object detector / analyzer 520, according to an embodiment of the present disclosure. In clinical use, an ultrasound video or video stream 310 is provided to a trained and validated object detector / analyzer 520 for analysis. In some cases, the video stream 310 may be acquired and analyzed in real time or near real time. In other cases, the video stream may be retrieved from memory, storage, or a network. The trained and validated object detector / analyzer 520 then produces as output an annotated version 720 of the video stream 310, including measurement points and detected anatomical structures.

[0079] Thus, for each frame of the video stream, the object detector / analyzer 520 is run to determine the localization and confidence value of features (e.g., anatomical structures and measurement points). Localization can be determined by forming a bounding box that tightly surrounds the feature. Other forms of localization are possible, such as a binary mask indicating the image pixels that are part of the feature, or a polygon or other shape that surrounds the feature. For any such localization, the center of detection and area can be determined. Confidence values ​​can be determined as normalized values ​​in the range [0,1], where 0 indicates the lowest confidence and 1 indicates the highest confidence that the feature is present at that location.

[0080] The detection step can be based on traditional image processing, including thresholding, filtering, and texture analysis, or it can be based on machine learning, especially using deep neural networks. One particularly useful implementation of detection is to use a Yolo-type network, such as Yolo3. An exemplary output of the detection step is a list of detections for one or several types of features of interest, for each frame of the video stream. Each element in the detection list can, for example, include at least the confidence and area (and typically the position, width, and height) of the detection. Thus, for each frame i and feature f, a list of detections {x, y, w, h; c} f, i where x, y represent the center coordinates, w, h represent the width and height, and c represents the confidence of the detection. Other means of representing the detection may be used instead or in addition without departing from the spirit of the invention.

[0081] 8 is an exemplary screen display 800 of an automatic measurement point detection system according to an aspect of the present disclosure. In the example shown in FIG. 8, the screen display 800 includes the location of the annotated image frame 720 with overlay measurements 580. The screen display 800 also includes a detected anatomy display area 810, a measurement report area 820 including final measurements 825, a convergence progress indicator 830, and two control settings 840 and 850.

[0082] 9 is an exemplary training image 900 for an automatic measurement point detection system according to an embodiment of the present disclosure. In the example shown in FIG. 9, the training image 900 is of a fetal head and includes a head bounding box 910, biparietal (BPD) measurement point bounding boxes 920L and 920R, and anteroposterior-occipital diameter (OFD) measurement point bounding boxes 930F and 930R. Each bounding box includes a measurement point 940, a region 950 occurring primarily on the outside of the head, and a region 960 occurring primarily on the inside of the head.

[0083] FIG. 10 is an exemplary object detection image 1000 of an automatic measurement point detection system according to an embodiment of the present disclosure. A head 1010, falx cerebri 1020, septum pellucidum (CSP) 1030, choroid plexus 1040, and posterior lateral ventricle (PLV) 1050 are visible. These anatomical features are generally found in the imaging plane best suited for performing head circumference measurements and may therefore serve as an indication that the imaging plane of the ultrasound probe is properly positioned and aligned for head circumference measurements. These specific anatomical landmarks are described herein for illustrative purposes. Depending on the anatomical structure being measured, other landmarks may be used instead or in addition, or an appropriate measurement plane may be detected based on geometry (e.g., the eccentricity of an ellipse) without reference to other anatomical landmarks.

[0084] 11 is an exemplary "Detected Objects" display 810 of an automatic measurement point detection system according to an embodiment of the present disclosure. In the example shown in FIG. 11, the detected objects display 810 includes several anatomical features 1110, each with its own check box 1120. The checked boxes 1140 may indicate either anatomical structures currently visible in the ultrasound video stream, previously detected anatomical structures, or anatomical structures that have allowed associated measurements to converge. In one example, an examination may be considered complete when all of the check boxes 1120 are checked.

[0085] 12 is an example of an estimated image 1200 generated by an automatic measurement point detection system according to aspects of the present disclosure. The image includes measured anatomical features 540 (in this case, ellipses representing the anatomical structure at the fetal head) and measurement points 550. Also visible are a biparietal diameter (BPD) line 1240 and an occipitofrontal diameter (OFD) line 1250. Depending on the implementation, these annotations may be displayed to the user, for example, along with the measurements derived from them.

[0086] 13 is an example object detection image 1300 of an automatic measurement point detection system according to an embodiment of the present disclosure. Detected anatomical structures 1310 and measurement points 1320L, 1320R, 1330F, and 1330R, as well as a detection confidence value 1340, are visible. The confidence threshold may be a user-selectable value, for example, such that an anatomical feature is only shown if its detection confidence 1340 exceeds 50%, 80%, 90%, 95%, etc. A higher confidence threshold may increase procedure time because acquiring the anatomical structures may be more difficult and therefore take longer. However, a higher confidence threshold may also improve the accuracy of the results.

[0087] 14 is an example object detection image 1400 of an automatic measurement point detection system according to aspects of the present disclosure. In this case, the detections are shown as bounding boxes 1410, 1420R, 1420L, 1430F, and 1430R. Depending on the embodiment, there may be a threshold associated with the center point of the measurement point box being inside the anatomical box. For example, the head measurement point does not necessarily have to be exactly inside the head (or head anatomical box), but can be within a threshold distance (+ / -) of the anatomical box.

[0088] 15 is an example filter control set for an automatic measurement point detection system according to an embodiment of the present disclosure. Confidence filter control 840 and intersection over union (IOU) filter control 850 are visible. In one example, confidence filter control 840 can be used to adjust (e.g., in real time) a confidence threshold for anatomical structure detection. The IOU filter control can be used to set a threshold that filters out duplicate detections. For example, if two femoral endpoints are detected within a few millimeters of each other, they may be considered a single femoral endpoint and therefore a single measurement point, but if two femoral endpoints are detected several centimeters apart, they may be considered two different endpoints of the same femur.

[0089] 16 is an exemplary training image 1600 for an automatic measurement point detection system according to an embodiment of the present disclosure. In the example shown in FIG. 16, the training image 1600 is of a fetal abdomen and includes an abdomen bounding box 1610, left and right transverse abdominal diameter (TAD) measurement point bounding boxes 1620L, 1620R, and anterior-posterior abdominal diameter (APAD) measurement point bounding boxes 1630F, 1630R.

[0090] 17 is an exemplary object detection image 1700 of an automatic measurement point detection system according to an embodiment of the present disclosure. The abdomen 1710, stomach 1720, spine 1730, and umbilical vein 1740 are visible. In one example, detection of these anatomical features may indicate that the imaging plane of the ultrasound probe is properly positioned and aligned for abdominal diameter or circumference measurement.

[0091] 18 is an example of an estimated image 1800 generated by an automatic measurement point detection system according to aspects of the present disclosure. The image includes the measured anatomical feature 540 (an ellipse representing the anatomical structure, in this case the fetal abdomen) and measurement points 550. Also visible are an anteroposterior abdominal diameter (APAD) line 1840 and a transverse abdominal diameter (TAD) line 1850. Depending on the implementation, these annotations may be displayed to the user, for example, along with the measurements derived therefrom.

[0092] 19 is an exemplary training image 1900 for an automatic measurement point detection system according to an embodiment of the present disclosure. The image shows an anatomical bounding box 1910 of a fetal femur 1915, as well as a bounding box 1920 of an endpoint of the femur 1915. The training image may be fed to an untrained neural network, for example, as described in FIG. 7A.

[0093] 20 is an example of an estimated image 2000 generated by an automatic measurement point detection system according to aspects of the present disclosure. The image includes a measured anatomical feature 540 (a line representing the anatomical structure, in this case the fetal femur) and a measurement point 550 indicating the end point of the femur. In this example, the measured anatomical feature 540 is simply a line connecting the two measurement points 550. Depending on the implementation, these annotations may be displayed to the user, for example, along with the measurements derived from them.

[0094] Depending on the implementation, the measured anatomical characteristic 540 may be or may include head circumference, BPD, OPD, femur length, abdominal circumference, TAD, APAD, or other anatomical characteristic of the patient. 21 is an example convergence progress display 830 of an automatic measurement point detection system according to aspects of the present disclosure. In the example shown in FIG. 21 , the convergence progress display 830 includes multiple measurements 580, measurement labels 2185 of the measurements 2180, each with a check box 2110, and a progress indicator 2120. A "waiting" or "processing" indicator 2130 and a checked check box 2115 are visible on one of the measurements 2180, indicating that this measurement 2180 is currently in progress and has not yet converged.

[0095] In one example, while waiting for a measurement 2180 to converge, each time a larger (or otherwise "more converged") value is received for that measurement 2180, the measurement value 580 for that measurement 2180 is updated and the progress indicator 2120 for that measurement 2180 becomes longer. In some cases, the progress indicator 2120 may also change color, e.g., when a measurement 580 is changing rapidly, the corresponding progress indicator 2120 is red; when a measurement 580 is changing less frequently, the corresponding progress indicator 2120 is yellow; and when a measurement 580 is no longer changing, the corresponding progress indicator 2120 is green. Thus, a green indicator 2120 for each measurement 2180 may indicate that all desired measurements have converged and the test is complete. In other cases, the color of the measurement values ​​themselves may similarly change from red to yellow to green, or other visual indicators of convergence or non-convergence may be provided instead or in addition.

[0096] In the example shown in FIG. 21 , the convergence progress display 830 also includes a plane alignment indicator 2140. The plane alignment indicator 2140 includes a desired imaging plane indicator 2150 and an actual imaging plane indicator 2160 in a bubble-level configuration. Thus, the position and orientation of the actual imaging plane indicator 2160 relative to the desired imaging plane indicator can provide guidance to a clinician or other user for moving or reorienting the ultrasound imaging probe. For example, when the transducer is aligned, a small circle can be centered inside a larger circle. For example, alignment can be based on the intersection of the BPD and OFD or the TAD / APAD. Other types of plane alignment indicators, including, but not limited to, straight lines, high point indicators, etc., can be used instead or in addition.

[0097] 22 is an example convergence progress display 830 of an automatic measurement point detection system according to an aspect of the present disclosure. In the example shown in FIG. 22, the convergence progress display 830 includes a measurement label 2185, a numerical scale 2220, a maximum measurement value 2230, and a current measurement value 2240. In one example, each time a new maximum value for the current measurement is received, the maximum measurement value is updated to a larger value. When a certain duration or number of measurements has occurred without a larger value being received, the maximum measurement value 2230 may be considered the converged, accurate measurement value corresponding to the measurement label 2185.

[0098] 23 is an example convergence progress display 830 of an automatic measurement point detection system according to aspects of the present disclosure. In the example shown in FIG. 23, the convergence progress display 830 includes a measurement label 2185, a numerical scale 2220, a maximum measurement value 2230, and a current measurement value 2240. In this case, however, the progress of the current measurement over time is represented as a graph 2320 whose X-axis 2310 indicates time or frame number and whose Y-axis indicates the numerical scale 2220. If a particular duration or number of measurements occurs without a current measurement value 2240 greater than the maximum measurement value 2230 being received, then the maximum measurement value 2230 may be considered the converged, accurate measurement value corresponding to the measurement label 2185.

[0099] Several variations are possible in the above examples and aspects. For example, the systems, methods, and devices described herein are not limited to neonatal ultrasound applications. Rather, the same techniques can be applied to images of other organs or anatomical systems, such as the heart, brain, digestive system, vascular system, or their pathologies. For example, in the case of cardiac ultrasound, the distance between the long and short axes of the left ventricle can be measured; in the case of vascular ultrasound, the width and length of the kidney and / or the length and width of the liver can be measured, etc. The measured anatomical structures can include either the CSP, PLV, etc. (as shown in FIG. 10 ), or the stomach, spine, umbilical vein, or any other anatomical structure of the patient on which the neural network was trained (as shown in FIG. 17 ).

[0100] Additionally, the techniques disclosed herein are applicable to other medical imaging modalities where 3D data is available, such as computed tomography (CT) scans, magnetic resonance imaging (MRI) scans, optical coherence tomography (OCT) scans, or other ultrasound applications such as intravenous ultrasound (IVUS) pullback sequences, camera-based video, X-ray video, and 3D volumetric images acquired from medical imaging devices. The techniques described herein can be used in a variety of hospital settings, including emergency departments, intensive care units, inpatient and outpatient settings.

[0101] Accordingly, the logical operations making up aspects of the technology described herein are referred to variously as operations, steps, objects, layers, elements, components, algorithms, or modules, and should be understood to be capable of occurring or being performed or arranged in any order unless expressly claimed or a particular order is inherently required by claim language.

[0102] All directional references, for example, above, below, inside, outside, upward, upward, left, down, right, lateral, front, rear, above, below, up, down, vertical, horizontal, clockwise, counterclockwise, proximal, and distal, are used for identification purposes only to aid the reader's understanding of the claimed subject matter and do not create limitations with respect to the location, orientation, or use, particularly of an automatic measurement point detection system. Connection references, such as attached, coupled, connected, joined, or "in communication with," should be interpreted broadly and, unless otherwise specified, may include intermediate members between a collection of elements and relative movement between the elements. Thus, a connection reference does not necessarily imply that two elements are directly connected and in a fixed relationship to one another. The term "or" should be interpreted to mean "and / or" rather than "exclusively or," the term "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. Unless otherwise stated in the claims, the values ​​recited are to be interpreted as examples only and not as limiting.

[0103] The above specification, examples, and data provide a complete description of the structure and use of exemplary embodiments of the automatic measure point detection system as defined in the claims. While various aspects of the claimed subject matter have been described above with a certain degree of particularity, or with reference to one or more individual embodiments, those skilled in the art could make numerous modifications to the disclosed embodiments without departing from the spirit or scope of the claimed subject matter.

[0104] Further embodiments are contemplated. All subject matter contained in the above description and shown in the accompanying drawings is to be construed only as an illustration of particular embodiments, and is not intended to be limiting. Changes in detail or structure may be made without departing from the basic elements of the subject matter defined in the following claims.

Claims

1. 1. A system comprising: The display and a processor configured to communicate with the display and the medical imaging device; wherein the processor: receiving a first anatomical image frame acquired by the medical imaging device during live imaging; identifying a plurality of first measurement points of an anatomical feature in the first anatomical image frame while the live imaging is in progress, wherein the identification of the first measurement points is performed by a first neural network trained to identify where measurement points are to be located within the anatomical image frame, such that the identification of the first measurement points is performed automatically without user input for locating the plurality of first measurement points in the first anatomical image frame; generating a first measurement of an anatomical feature for the first anatomical image frame based on the plurality of first measurement points; outputting a screen display to the display based on the first measurement value; A system configured to run

2. The processor: While the live imaging is in progress, identifying a plurality of second measurement points of an anatomical feature within second anatomical image frames acquired by the medical imaging device during the live imaging; generating second measurements of anatomical features for the second anatomical image frame based on the plurality of second measurement points; configured to run the screen display is based on the first measurement value and the second measurement value; The system of claim 1.

3. The system of claim 2 , wherein the second frame is acquired immediately after the first frame.

4. the processor is configured to determine whether convergence of measurements has occurred based on the first measurement and the second measurement; The system of claim 2 , wherein the processor is configured to provide the screen display based on a determination of whether convergence of the measurements has occurred.

5. The system of claim 4 , wherein the screen display includes a progress of convergence of measurements.

6. The system of claim 4 , wherein the processor is configured to determine whether convergence of the measurements has occurred based on a difference between the first measurement and the second measurement.

7. 6. The system of claim 5, wherein to determine whether convergence of the measurements has occurred, the processor is configured to determine whether the second measurement is less than the first measurement.

8. If convergence of the measurements occurs, the processor is configured to select the larger of the first measurement or the second measurement as a converged measurement; the screen display having the converged measurements. The system of claim 7.

9. The screen display is the first anatomical image frame; a plurality of first measurement points superimposed on the first anatomical image frame; The system of claim 1 , comprising:

10. The screen display is the first anatomical image frame; an indication of an anatomical feature superimposed on the first anatomical image frame; and The system of claim 1 , comprising:

11. the processor is configured to identify, in the first anatomical image frame, an anatomical structure having the anatomical feature; the processor is configured to generate the first measurement based on an identification of the anatomical structure. The system of claim 1.

12. the first anatomical image frame; an indication of an anatomical structure superimposed on the first anatomical image frame; and The system of claim 11 , comprising:

13. The system of claim 11 , wherein the identification of the anatomical structure is performed using a second neural network.

14. 14. The system of claim 13, wherein the first neural network and the second neural network are the same neural network.

15. The system of claim 1 further comprising the medical imaging device.

16. receiving, by a processor in communication with a medical imaging device, anatomical image frames acquired by the medical imaging device during live imaging; identifying a plurality of measurement points of anatomical features within the anatomical image frames while the live imaging is in progress, wherein the identification of the measurement points is performed by a neural network trained to identify where measurement points are to be located within the anatomical image frames, such that the identification of the first measurement points is performed automatically without user input for positioning the first plurality of measurement points within the anatomical image frames; generating anatomical feature measurements for the anatomical image frame based on the plurality of measurement points; outputting a screen display based on the measurements on a display in communication with the processor; A method comprising:

Images