Automatic measurement point detection for anatomical structure measurements in anatomical images

Abstract

A system includes a display and a processor configured to communicate with the display and a medical imaging device. The processor is configured to receive an anatomical image frame obtained by the imaging device during live imaging and identify a plurality of measurement points of an anatomical feature in the anatomical image frame. The identification of a first measurement point is performed by a neural network trained to identify where a measurement point is located within the anatomical image frame such that the measurement point is automatically identified without user input. The processor is further configured to generate a measurement value of the anatomical feature based on the measurement point, and output a screen display based on the measurement value.

Description

Technical Field

[0001] The subject matter described herein relates to devices, systems, and methods for automatically locating and measuring features (e.g., anatomical features or pathologies) during live imaging performed by a medical imaging device (e.g., an ultrasound probe). For example, a neural network is trained to identify measurement points in an anatomical image, which can be used to generate measurements of the features. Background Art

[0002] Ultrasound imaging is commonly used for diagnostic purposes in an office or hospital setting. For example, ultrasound is an imaging technique that is deployed at the point of care to assist in the assessment of fetal development. Important anatomical features (such as head circumference, abdominal circumference, and femur length) can be obtained by a clinician in near real-time by freezing a live ultrasound video stream at a particular image, placing measurement points on the image, and then performing manual calculations or relying on software algorithms to perform calculations based on, for example, the distances between certain measurement points. The measurement points can be two endpoints (e.g., for a measurement value based on the distance of a line between two points). The measurement points can also be more than two points. For example, more than two measurement points can define a curve, a curved shape, a closed shape, etc. (e.g., area, circumference, perimeter, etc.). In some cases, the measurement points can be referred to as calipers (e.g., referring to physical calipers that can be used to collect similar measurements from an infant after birth). This measurement process can be imprecise, labor-intensive, and highly dependent on the level of training of the practitioner.

[0003] For example, an ultrasound examination protocol may require a clinician to scan a patient to locate a specific anatomical structure of interest, pause the system, press the appropriate measurement button, and position measurement calipers to measure the anatomical structure. The clinician typically takes 3 or more measurements for each anatomical structure and averages the results to derive an accurate measurement result, which occupies a large portion of the total examination time.

[0004] The information included in this background art section of the specification (including any references cited herein and any description or discussion thereof) is included for technical reference purposes only and should not be regarded as subject matter by which the scope of the present disclosure is to be bound. Summary of the Invention

[0005] An automatic measurement point detection system using artificial intelligence / machine learning algorithms (e.g., neural networks) is disclosed. The neural network is trained on anatomical image frames that are manually annotated with the positions of measurement calipers, and the neural network can be used to automatically identify and measure anatomical structures (depicted in medical images such as ultrasound images or X-ray images) in real time. For example, in ultrasound imaging, a clinician or other user simply moves the transducer over the anatomical structure of interest, and the trained neural network identifies the anatomical structure and locates the measurement calipers in real time to automatically calculate the measurement results without user intervention. Instead of performing the traditional three measurements during an examination for each anatomical structure, the automatic measurement point detection system completes hundreds of measurement results at the real-time frame rate. Once the measurement results are identified as converging, the user interface indicates that the most accurate measurement results have been achieved.

[0006] The automatic measurement point detection system disclosed herein has a specific but non-exclusive utility for measuring the dimensions of anatomical features or pathologies in a real-time ultrasound video stream (e.g., live imaging as a clinician moves an ultrasound probe over a patient's body), as may occur for example in prenatal ultrasound examinations. The automatic measurement point detection system detects features in individual frames of the video, automatically places measurement points on the video frames, performs calculations based on the measurement points, and generates and displays anatomical measurement results based on the calculations.

[0007] A system of one or more computers can be configured to perform particular operations or actions by virtue of software, firmware, hardware, or a combination of them installed in the system that, in operation, cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by including instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0008] One general aspect includes a system including a display and a processor configured to communicate with the display and a medical imaging device. The processor is configured to: receive a first anatomical image frame obtained by the medical imaging device during live imaging; during the live imaging, identify a plurality of first measurement points of anatomical features in the first anatomical image frame, wherein the identification of the first measurement points is performed by a first neural network trained to identify where the measurement points are located within the anatomical image frame such that the identification of the first measurement points is automatically performed without user input for positioning the plurality of first measurement points in the first anatomical image frame; generate a first measurement value of the anatomical features for the first anatomical image frame based on the plurality of first measurement points; and output a screen display to the display based on the first measurement value. Other aspects include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, all configured to perform the actions of the method.

[0009] Embodiments may include one or more of the following features. In some aspects, the processor is configured to: identify, while live imaging is in progress, a plurality of second measurement points of an anatomical feature in a second anatomical image frame obtained by a medical imaging device during the live imaging; and generate, based on the plurality of second measurement points, a second measurement value of the anatomical feature for the second anatomical image frame, wherein the screen display is based on a first measurement result and a second measurement result. In some aspects, the second frame is obtained immediately after the first frame. In some aspects, the processor is configured to determine whether convergence of the measurement values has occurred based on the first measurement value and the second measurement value, wherein the processor is configured to provide the screen display based on determining whether convergence of the measurement values has occurred. In some aspects, the screen display may include the progress of the convergence of the measurement values. In some aspects, to determine whether convergence of the measurement values has occurred, the processor is configured to determine whether the second measurement result is less than the first measurement result. In some aspects, if convergence of the measurement values has occurred, the processor is configured to select the larger value of the first measurement value or the second measurement value as the converged measurement value; and wherein the screen display may include the converged measurement value. In some aspects, the processor is configured to determine whether convergence of the measurement values has occurred based on the difference between the first measurement result and the second measurement result. In some aspects, the screen display may include: the first anatomical image frame; and the plurality of first measurement points, which are superimposed on the first anatomical image frame. In some aspects, the screen display may include: the first anatomical image frame; and an indication of the anatomical feature, which is superimposed on the first anatomical image frame. In some aspects, the processor is configured to identify an anatomical structure including the anatomical feature in the first anatomical image frame, wherein the processor is configured to generate the first measurement value based on the identification of the anatomical structure. In some aspects, the screen display may include the first anatomical image frame and an indication of the anatomical structure superimposed on the first anatomical image frame. In some aspects, a second neural network is used to perform the identification of the anatomical structure. In some aspects, the first neural network and the second neural network are the same neural network. In some aspects, the system may include a medical imaging device. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0010] One general aspect includes a method that includes receiving, by a processor in communication with a medical imaging device, anatomical image frames obtained by the medical imaging device during live imaging. The method further includes identifying, while the live imaging is in progress, a plurality of measurement points of anatomical features in the anatomical image frames, wherein the identification of the measurement points is performed by a neural network that is trained to identify where the measurement points are located within the anatomical image frames such that the identification of the first measurement points is automatically performed without user input for positioning the plurality of first measurement points in the anatomical image frames. The method further includes generating, based on the plurality of measurement points, a measurement value for the anatomical features of the anatomical image frame, and outputting a screen display to a display in communication with the processor based on the measurement value. Other aspects include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, all of which are configured to perform the actions of the method.

[0011] The present invention content is provided to introduce, in a simplified form, a selection of concepts that will be further described in the detailed description below. The present invention content is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. A more extensive presentation of the features, details, utilities, and advantages of the automatic measurement point detection system as defined in the claims is provided in the following written description of various aspects of the present disclosure and is illustrated in the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Exemplary aspects of the present disclosure will be described with reference to the drawings, in which:

[0013] Figure 1 is a schematic diagrammatic representation of an ultrasound imaging system according to aspects of the present disclosure.

[0014] Figure 2 is a schematic diagram of a processor circuit according to aspects of the present disclosure.

[0015] Figure 3 is a schematic diagrammatic representation of a radiology video, video stream, or video clip according to aspects of the present disclosure.

[0016] Figure 4 is a schematic diagrammatic representation in flowchart form of an exemplary ultrasound video feature measurement method according to aspects of the present disclosure.

[0017] Figure 5 is a schematic diagrammatic representation in block diagram form of at least a portion of an automatic measurement point detection system according to aspects of the present disclosure.

[0018] Figure 6 is a schematic diagrammatic representation in block diagram form of at least a portion of an automatic measurement point detection system according to aspects of the present disclosure.

[0019] Figure 7A It is a schematic diagrammatic overview in the form of a block diagram of a training mode for an object detector / analyzer according to aspects of the present disclosure.

[0020] Figure 7B It is a schematic diagrammatic overview in the form of a block diagram of a verification mode for an object detector / analyzer according to aspects of the present disclosure.

[0021] Figure 7C It is a schematic diagrammatic overview in the form of a block diagram of an inference mode or a clinical use mode for an object detector / analyzer according to aspects of the present disclosure.

[0022] Figure 8 It is an example screen display of an automatic measurement point detection system according to aspects of the present disclosure.

[0023] Figure 9 It is an example training image for an automatic measurement point detection system according to aspects of the present disclosure.

[0024] Figure 10 It is an example object detection image of an automatic measurement point detection system according to aspects of the present disclosure.

[0025] Figure 11 It is an example "detected object" display of an automatic measurement point detection system according to aspects of the present disclosure.

[0026] Figure 12 It is an example inference image generated by an automatic measurement point detection system according to aspects of the present disclosure.

[0027] Figure 13 It is an example object detection image of an automatic measurement point detection system according to aspects of the present disclosure.

[0028] Figure 14 It is an example object detection image of an automatic measurement point detection system according to aspects of the present disclosure.

[0029] Figure 15 It is an example filtering control setting of an automatic measurement point detection system according to aspects of the present disclosure.

[0030] Figure 16 It is an example training image for an automatic measurement point detection system according to aspects of the present disclosure.

[0031] Figure 17 It is an example object detection image of an automatic measurement point detection system according to aspects of the present disclosure.

[0032] Figure 18 It is an example inference image generated by an automatic measurement point detection system according to aspects of the present disclosure.

[0033] Figure 19 is an example training image for an automatic measurement point detection system according to aspects of the present disclosure.

[0034] Figure 20 is an example inference image generated by an automatic measurement point detection system according to aspects of the present disclosure.

[0035] Figure 21 is an example convergence progress display of an automatic measurement point detection system according to aspects of the present disclosure.

[0036] Figure 22 is an example convergence progress display of an automatic measurement point detection system according to aspects of the present disclosure.

[0037] Figure 23 is an example convergence progress display of an automatic measurement point detection system according to aspects of the present disclosure. Detailed Description

[0038] According to at least one aspect of the present disclosure, an automatic measurement point detection system is provided that can measure the dimensions of anatomical features or pathologies in individual frames of a real-time ultrasound video stream. This can allow, for example, minimally trained users (including general practitioners, paramedics, and even patients) to obtain accurate anatomical measurements with a minimal time investment and a high confidence in the results.

[0039] The automatic measurement point detection system disclosed herein has a specific but non-exclusive utility for measuring anatomical structures during ultrasound procedures (such as prenatal examinations, lung examinations, etc.). The automatic measurement point detection system detects features in individual frames of the video, automatically places measurement points on the video frames, performs calculations based on the measurement points, generates anatomical measurement results based on the calculations, and displays the measurement results on an ultrasound console display, all in real time (e.g., at a refresh rate of up to 60 - 100 Hz such that the system has completed the measurement and annotation of one frame of the real-time video stream before displaying the next frame).

[0040] The described automatic measurement point detection system uses artificial intelligence (AI) deep learning (DL) or other machine learning (ML) techniques. Ultrasound video frames showing a specific anatomical structure (e.g., fetal head, abdomen, or femur) and manually annotated with measurement points (calipers) are used to train the AI model. Once the AI model is trained, it can be incorporated into the software of the ultrasound console to automatically identify and measure the imaged anatomical structure in real time. The clinician or other user simply moves and / or reorients the ultrasound probe over the body part containing the anatomical structure of interest, and the trained AI model identifies the anatomical structure and calculates the position of the measurement points in real time (e.g., identifies the position of the measurement points (e.g., the coordinates of the pixel(s) that make up the measurement point in the pixels forming the anatomical image frame)) to automatically calculate the desired anatomical measurement results. No intervention from the user is required other than acquiring the relevant ultrasound images. The automatic measurement point detection system can complete hundreds of measurement results at a real-time frame rate, which are analyzed for convergence and plane alignment. Once the measurement results are identified to have converged (e.g., by detecting the maximum anatomical structure length or elliptical symmetry), the user interface indicates that the most accurate measurement results have been achieved, at which point the examination can be completed, or the user can move on to another anatomical feature that needs to be measured.

[0041] The main elements of the described automatic measurement point detection system include: (1) an AI neural network model trained on both ultrasound anatomical structures and manually generated measurement points (or their bounding boxes). (2) An error filtering step to confirm that the measurement points are within the detected anatomical structure box and are compatible. (e.g., the femur endpoints cannot be located within the head) (3) A calculation engine to derive the desired anatomical measurement results based on the identified measurement points. (4) A test for detecting convergence of the measurement results. For example, when a maximum distance is reached (e.g., several seconds have passed without detecting a longer measurement result), a distance measurement such as femur length can be considered complete. For more complex measurement results such as perimeter or area, the system can examine elliptical symmetry, maximum perimeter, maximum area, and / or the presence of certain anatomical features indicating the appropriate measurement plane. (5) A user interface graphic showing the measurement results and status.

[0042] The automatic measurement point detection system can be used with live and / or recorded ultrasound video streams. Example deep learning models have been trained to measure fetal femur length, head circumference, and abdominal circumference, and the model can also be trained using live or recorded video from medical imaging devices using other imaging modalities, including but not limited to visible light, angiography / fluoroscopy (X-ray), computed tomography (CAT) scan, magnetic resonance imaging (MRI), intravascular ultrasound (IVUS), etc.

[0043] The following Figure 9 、16 and Figure 19 are example images passed to the AI training step, where fetal anatomical structures and measurement endpoints are shown using bounding boxes. The neural network is "trained" by feeding it multiple (e.g., dozens, hundreds, or thousands) of images annotated with anatomical structure identifications and measurement points. As the training process progresses, the neural network model "learns" the anatomical structures and measurement point locations and can accurately predict this information on new ultrasound images.

[0044] An obvious advantage of training with manually annotated anatomical structures and measurement endpoints is the ability to validate results and filter potential errors. When detection data is received from the AI model, filtering rules are applied. For example, the anatomical structure box itself (e.g., head, abdomen, femur) and the measurement points must be detected, and furthermore the measurement points must actually fall within the anatomical structure box. Example: For a femur measurement to be completed, there must be two femur endpoints located within the femur anatomical box. To complete a head circumference measurement, there must be two biparietal diameter (BPD) and two occipitofrontal diameter (OFD) endpoints located within the head anatomical structure box, and we must detect the thalamus structure within the head to identify the correct planar location. These rules ensure that the neural network calculates measurements at the precise imaging plane. Alternatively or additionally, other filtering rules can be used.

[0045] In one aspect, the user begins an examination of the ultrasound system and starts scanning a patient. When an anatomical structure is visualized, the measurement results for that anatomical structure are automatically displayed and updated in real time. When the user moves the ultrasound probe into and out of the imaging plane, the measurements are stable, and the final results are presented to the user. For anatomical structures that require area measurements, multiple components of the measurement, such as length and width, can be displayed in real time using alignment graphics indicating plane alignment, based on factors such as the ratio of length and width, the appearance of the reference anatomical structure in the desired imaging plane, etc.

[0046] In another aspect, the automatic measurement point detection system can be configured to provide a single measurement result. When the user visualizes the desired anatomical structure and freezes the image, the single measurement result is automatically displayed and is editable as in a traditional workflow.

[0047] The present disclosure substantially facilitates obtaining anatomical measurement results from live or recorded radiology videos by automatically placing measurement points and performing measurement result calculations without the need for manual intervention (e.g., without a clinician pausing the live video stream to place measurement points and calculate the measurement results). Implemented on a processor in communication with an ultrasound probe, the automatic measurement point detection system disclosed herein provides a practical improvement in the ability of untrained or inexperienced clinicians to obtain accurate anatomical measurement results from radiology videos. This improved anatomical structure measurement transforms a subjective, time-consuming process that heavily relies on professional experience into an objective and repeatable process, typically without the need to train clinicians (such as emergency department personnel) to identify specific anatomical structures in the video stream and perform accurate measurements. This unconventional approach improves the functionality of the ultrasound imaging system by providing reliable measurement results of anatomical features or pathologies.

[0048] The automatic measurement point detection system can be implemented as a process that is at least partially visible on a display and is operated by a control process executed on a processor that receives user input from a keyboard, mouse, or touchscreen interface and communicates with one or more sensor probes. In this regard, the control process performs certain specific operations in response to different inputs or selections made at different times. Certain structures, functions, and operations of the processor, display, sensors, and user input system are known in the art, while other structures, functions, and operations are described herein to implement the novel features or aspects of the present disclosure.

[0049] These descriptions are provided for illustrative purposes only and should not be considered to limit the scope of the automatic measurement point detection system. Certain features may be added, removed, or modified without departing from the spirit of the claimed subject matter.

[0050] To facilitate an understanding of the principles of the present disclosure, aspects shown in the accompanying drawings will now be referred to, and these aspects will be described using specific language. However, it should be understood that no limitation of the scope of the present disclosure is intended. Any changes and further modifications to the described devices, systems, and methods, as well as any further applications of the principles of the present disclosure, are fully contemplated and included within the present disclosure as would typically be thought of by a person skilled in the art to which the present disclosure pertains. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one aspect may be combined with the features, components, and / or steps described with respect to other aspects of the present disclosure. However, for the sake of brevity, multiple iterations of these combinations will not be described separately.

[0051] Figure 1FIG. 0 is a schematic graphical representation of an ultrasound imaging system 100 in accordance with aspects of the present disclosure. For example, the ultrasound imaging system 100 can be used to acquire ultrasound video clips, which can be used to train an automatic measurement point detection system or can be analyzed and highlighted by the automatic measurement point detection system in a clinical setting (either in real-time, near real-time, or as post-processing of a stored video clip).

[0052] The ultrasound imaging system 100 is configured to scan an area or volume of an object's body. The object can include a patient undergoing an ultrasound imaging procedure, or any other person, or any suitable living or non-living entity or structure. The ultrasound imaging system 100 includes an ultrasound imaging probe 110 that communicates with a host 130 via a communication interface or link 120. The probe 110 can include a transducer array 112, a beamformer 114, a processor circuit 116, and a communication interface 118. The host 130 can include a display 132, a processor circuit 134, a communication interface 136, and a memory 138 that stores object information.

[0053] In some aspects, the probe 110 is an external ultrasound imaging device that includes a housing 111 configured for hand-held operation by a user. The transducer array 112 can be configured to obtain ultrasound data when the user grasps the housing 111 of the probe 110 such that the transducer array 112 is positioned adjacent to or in contact with the skin of the object. The probe 110 is configured to obtain ultrasound data of anatomical structures within the object's body when the probe 110 is positioned external to the object's body for general imaging (such as for abdominal imaging, liver imaging, etc.). In some aspects, the probe 110 can be an external ultrasound probe, a trans-thoracic probe, and / or a curved array probe.

[0054] In other aspects, the probe 110 can be an internal ultrasound imaging device and can include a housing 111 configured to be positioned within a lumen of the object's body for general imaging, such as for abdominal imaging, liver imaging, etc. In some aspects, the probe 110 can be a curved array probe. The probe 110 can be in any suitable form for any suitable ultrasound imaging application including both external and internal ultrasound imaging.

[0055] In some aspects, aspects of the present disclosure can be implemented using medical images of an object obtained using any suitable medical imaging device and / or modality. Examples of medical images and medical imaging devices include X-ray images (angiographic images, fluoroscopic images, images with or without contrast) obtained by a medical imaging device such as an ultrasound imaging device, an X-ray imaging device, computed tomography (CT) images obtained by a CT imaging device, positron emission tomography-computed tomography (PET-CT) images obtained by a PET-CT imaging device, magnetic resonance images (MRI) obtained by an MRI imaging device, single photon emission computed tomography (SPECT) images obtained by a SPECT imaging device, optical coherence tomography (OCT) images obtained by an OCT imaging device, and intravascular photoacoustic (IVPA) images obtained by an IVPA imaging device. The medical imaging device can obtain medical images when positioned outside the object's body, spaced apart from the object's body, adjacent to the object's body, in contact with the object's body, and / or inside the object's body.

[0056] The transducer array 112 transmits an ultrasonic signal towards the anatomical object 105 of the patient and receives an echo signal reflected from the object 105 back to the transducer array 112. The transducer array 112 may include any suitable number of acoustic elements, including one or more acoustic elements and / or multiple acoustic elements. In some cases, the transducer array 112 includes a single acoustic element. In some cases, the transducer array 112 may include an array of acoustic elements having any number of acoustic elements in any suitable configuration. For example, the transducer array 112 may include values between 1 acoustic element and 10,000 acoustic elements, including values such as 2 acoustic elements, 4 acoustic elements, 36 acoustic elements, 64 acoustic elements, 128 acoustic elements, 500 acoustic elements, 812 acoustic elements, 1,000 acoustic elements, 3,000 acoustic elements, 8,000 acoustic elements, and / or other values larger and smaller. In some cases, the transducer array 112 may include an array of acoustic elements having any number of acoustic elements in any suitable configuration, such as a linear array, a planar array, a curved array, a curvilinear array, a circumferential array, an annular array, a phased array, a matrix array, a one-dimensional (1D) array, a 1.x-dimensional array (e.g., 1.5D array), or a two-dimensional (2D) array. The array of acoustic elements may be controlled and activated uniformly or independently (e.g., one or more rows, one or more columns, and / or one or more orientations). The transducer array 112 may be configured to obtain one-dimensional, two-dimensional, and / or three-dimensional images of the anatomical structure of the patient. In some embodiments, the transducer array 112 may include piezoelectric micromachined ultrasonic transducers (PMUTs), capacitive micromachined ultrasonic transducers (CMUTs), single crystals, lead zirconate titanate (PZT), PZT composites, other suitable transducer types, and / or combinations thereof.

[0057] The object 105 may include any anatomical structure or anatomical feature, such as a kidney, a liver, and / or any other anatomical structure of the object. The present disclosure may be implemented in the context of any number of anatomical locations and tissue types, including but not limited to organs, including the liver, the kidney, the gallbladder, the pancreas, the lungs; ducts; intestines; nervous system structures, including the brain, the dural sac, the spinal cord, and peripheral nerves; the urinary tract; and valves within blood vessels, blood, abdominal organs, and / or other systems of the body. In some aspects, the object 105 may include a malignant tumor, such as a tumor, a cyst, a lesion, bleeding, or a blood pool within any part of an artificial anatomical structure. The anatomical structure may be a blood vessel, an artery or a vein of the vascular system of the object, including the cardiac vasculature, the peripheral vasculature, the neurovascular system, the renal vasculature, and / or any other suitable lumen within the body. In addition to natural structures, the present disclosure may be implemented in the context of artificial structures, such as but not limited to heart valves, stents, shunts, filters, implants, and other devices.

[0058] The beamformer 114 is coupled to the transducer array 112. The beamformer 114 controls the transducer array 112, for example, for the transmission of ultrasonic signals and the reception of ultrasonic echo signals. In some embodiments, the beamformer 114 may apply time delays to signals sent to individual acoustic transducers within the array in the transducer array 112 such that the acoustic signals are steered in any suitable direction away from the probe 110. The beamformer 114 may also provide an image signal to the processor 116 based on the response to the received ultrasonic echo signals. The beamformer 114 may include multi-stage beamforming. Beamforming may reduce the number of signal lines for coupling to the processor 116. In some embodiments, the transducer array 112 in combination with the beamformer 114 may be referred to as an ultrasonic imaging component. The beamformer 114 may also be a microwave beamformer.

[0059] The processor 116 is coupled to the beamformer 114. The processor 116 may also be described as a processor circuit, which may include other components communicating with the processor 116, such as a memory, the beamformer 114, the communication interface 118, and / or other suitable components. The processor 116 may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 116 may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 116 is configured to process the beamformed image signal. For example, the processor 116 may perform filtering and / or quadrature demodulation to condition the image signal. The processor 116 and / or 134 may be configured to control the array 112 to obtain ultrasonic data associated with the object 105.

[0060] The communication interface 118 is coupled to the processor 116. The communication interface 118 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 118 may include hardware components and / or software components implementing a specific communication protocol suitable for conveying signals over the communication link 120 to the host 130. The communication interface 118 may be referred to as a communication device or a communication interface module.

[0061] The communication link 120 can be any suitable communication link. For example, the communication link 120 can be a wired link, such as a Universal Serial Bus (USB) link or an Ethernet link. Alternatively, the communication link 120 can be a wireless link, such as an Ultra-Wideband (UWB) link, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 WiFi link, or a Bluetooth link.

[0062] At the host 130, the communication interface 136 can receive the image signal. The communication interface 136 can be substantially similar to the communication interface 118. The host 130 can be any suitable computing and display device, such as a workstation, a personal computer (PC), a laptop computer, a tablet computer, or a mobile phone.

[0063] The processor 134 is coupled to the communication interface 136. The processor 134 can also be described as a processor circuit, which can include other components that communicate with the processor 134, such as the memory 138, the communication interface 136, and / or other suitable components. The processor 134 can be implemented as a combination of software components and hardware components. The processor 134 can include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a controller, an FPGA device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 134 can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. The processor 134 can be configured to generate image data based on the image signal received from the probe 110. The processor 134 can apply advanced signal processing and / or image processing techniques to the image signal. In some aspects, the processor 134 can form a three-dimensional (3D) volume image based on the image data. In some aspects, the processor 134 can perform real-time processing on the image data to provide a streaming video of the ultrasonic image of the object 105. In some aspects, the host 130 includes a beamformer. For example, the processor 134 can be part of such a beamformer and / or otherwise communicate with such a beamformer. The beamformer in the host 130 can be a system beamformer or a main beamformer (providing one or more subsequent stages of beamforming), while the beamformer 114 is a probe beamformer or a microbeamformer (providing one or more initial stages of beamforming).

[0064] Memory 138 is coupled to processor 134. Memory 138 can be any suitable storage device, such as cache memory (e.g., the cache memory of processor 134), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory devices, hard disk drives, solid-state drives, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0065] Memory 138 can be configured to store object information, measurements, data, or files related to the medical history of an object, the history of procedures performed, anatomical or biometric characteristics, properties, or medical conditions associated with the object, computer-readable instructions (such as code, software, or other applications), and any other suitable information or data. Memory 138 can be located within host 130. Object information can include measurements, data, files, other forms of medical history, such as, but not limited to, ultrasound images, ultrasound videos, and / or any imaging information related to the anatomical structure of the object. Object information can include parameters related to the imaging procedure, such as anatomical scan windows, probe orientations, and / or the position of the object during the imaging procedure. Memory 138 can also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to the implementation of image recognition algorithms, image quantization algorithms, and / or image acquisition guidance algorithms for detecting / segmenting anatomical structures, including those described herein.

[0066] Display 132 is coupled to processor circuit 134. Display 132 can be a monitor or any suitable display. Display 132 is configured to display ultrasound images, image videos, and / or any imaging information of object 105.

[0067] The ultrasound imaging system 100 can be used to assist an ultrasound physician in performing an ultrasound scan. The scan can be performed in a point-of-care environment. In some cases, the host 130 is a console or a mobile cart. In some cases, the host 130 can be a mobile device, such as a tablet computer, a mobile phone, or a portable computer. During the imaging process, the ultrasound system can acquire ultrasound images of a specific region of interest within the anatomical structure of an object. The ultrasound imaging system 100 can then analyze the ultrasound images to identify various parameters associated with the acquisition of the images, such as the scan window, the probe orientation, the object position, and / or other parameters. The ultrasound imaging system 100 can then store the images and these associated parameters in the memory 138. During a subsequent imaging process, the ultrasound imaging system 100 can retrieve the previously acquired ultrasound images and the associated parameters for display to the user, and these parameters can be used to guide the user of the ultrasound imaging system 100 to use the same or similar parameters during the subsequent imaging process, as will be described in more detail below.

[0068] In some aspects, the processor 134 can utilize a deep learning-based prediction network to identify parameters of the ultrasound images, including the anatomical scan window, the probe orientation, the object position, and / or other parameters. In some aspects, the processor 134 can receive metrics related to the region of interest being imaged or the physiological state of the object during the imaging process or perform various calculations related to the region of interest being imaged or the physiological state of the object. These metrics and / or calculations can also be displayed to the ultrasound physician or other users via the display 132.

[0069] Prior to proceeding, it should be noted that the above examples are provided for illustrative purposes and are not intended to be limiting. Other devices and / or device configurations can be utilized to perform the operations described herein.

[0070] Figure 2 is a schematic diagram of a processor circuit 250 according to aspects of the present disclosure. The processor circuit 250 can be implemented in the ultrasound imaging system 100 or other devices or workstations (e.g., third-party workstations, network routers, etc.), or on a cloud processor or other remote processing unit as needed to implement the method. As shown, the processor circuit 250 can include a processor 260, a memory 264, and a communication module 268. These elements can communicate directly or indirectly with each other, for example, via one or more buses.

[0071] The processor 260 may include a central processing unit (CPU), a digital signal processor (DSP), an ASIC, a controller, or any combination of general-purpose computing devices, reduced instruction set computing (RISC) devices, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other related logic devices (including mechanical and quantum computers). The processor 260 may also include another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 260 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0072] The memory 264 may include cache memory (e.g., cache memory of the processor 260), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid state memory devices, hard disk drives, other forms of volatile and non-volatile memory, or a combination of different types of memory. In one aspect, the memory 264 includes non-transitory computer-readable media. The memory 264 may store instructions 266. The instructions 266 may include instructions that, when executed by the processor 260, cause the processor 260 to perform the operations described herein. The instructions 266 may also be referred to as code. The terms "instructions" and "code" should be construed broadly to include any type of (one or more) computer-readable statements. For example, the terms "instructions" and "code" may refer to one or more programs, routines, subroutines, functions, procedures, etc. "Instructions" and "code" include single computer-readable statements or multiple computer-readable statements.

[0073] The communication module 268 may include any electronic circuit and / or logic circuit to facilitate direct or indirect communication of data between the processor circuit 250 and other processors or devices. In this regard, the communication module 268 may be an input / output (I / O) device. In some cases, the communication module 268 facilitates direct or indirect communication between the processor circuit 250 and / or various elements of the ultrasound imaging system 100. The communication module 268 may communicate within the processor circuit 250 by a variety of methods or protocols. Serial communication protocols may include, but are not limited to, the United States Serial Protocol Interface (US SPI), Inter-Integrated Circuit (I 2C), Recommended Standard 232 (RS-232), RS-485, Controller Area Network (CAN), Ethernet, Aeronautical Radio Incorporated 429 (ARINC429), MODBUS, Military Standard 1553 (MIL-STD-1553), or any other suitable method or protocol. Parallel protocols include, but are not limited to, Industry Standard Architecture (ISA), Advanced Technology Attachment (ATA), Small Computer System Interface (SCSI), Peripheral Component Interconnect (PCI), Institute of Electrical and Electronics Engineers 488 (IEEE-488), IEEE-1284, and other suitable protocols. In appropriate cases, serial and parallel communications can be bridged by a Universal Asynchronous Receiver-Transmitter (UART), Universal Synchronous Receiver-Transmitter (USART), or other appropriate subsystems.

[0074] External communications (including, but not limited to, software updates, firmware updates, model sharing between the processor and a central server, or readings from the ultrasound imaging system 100) can be implemented using any suitable wireless or wired communication technology, such as a cable interface (such as Universal Serial Bus (USB), Micro-USB, Lightning, or FireWire interface), Bluetooth, Wi-Fi, ZigBee, Li-Fi, or cellular data connection (such as 2G / GSM (Global System for Mobile Communications), 3G / UMTS (Universal Mobile Telecommunications System), 4G, Long-Term Evolution (LTE), WiMax, or 5G). For example, a Bluetooth Low Energy (BLE) radio can be used to establish a connection to a cloud service for transmitting data and receiving software patches. The controller can be configured to communicate with a remote server or a local device (such as a laptop, tablet, or handheld device), or can include a display capable of showing status variables and other information. Information can also be transferred on a physical medium such as a USB flash drive or memory stick.

[0075] Figure 3 is a schematic graphical representation of a radiological (e.g., ultrasound) video stream 310 (whether live or recorded) in accordance with aspects of the present disclosure. The ultrasound video stream 310 includes a plurality of frames 320. In an example, the ultrasound video stream 310 is captured at a frame rate of 30, 60, or 100 frames per second. Each frame as the Y-axis or height 330 and X-axis or width 340 is a spatial dimension representing a 2D cross-section of an object imaged by the ultrasound imaging system. Additionally, the ultrasound video stream 310 includes a depth or time axis 350, which represents the time at which each frame 320 of the video stream 310 is captured. Thus, the ultrasound video stream 310 can be considered a 3D data structure. The video stream 310 can be any suitable modality with 2D image frames over time, such as X-ray, MRI, CT, etc.

[0076] Figure 4is a schematic graphical representation in flowchart form of an exemplary automatic measurement point detection method 400 according to aspects of the present disclosure. It should be understood that the steps of method 400 may be performed in a different order than Figure 4 shown, additional steps may be provided before, during, and after the steps, and / or in other respects some of the described steps may be replaced or eliminated. One or more steps of method 400 may be performed by one or more of the devices and / or systems described herein, such as components of ultrasound system 100 and / or processor circuitry 250.

[0077] In step 410, method 400 includes receiving an image frame from a live or recorded ultrasound video stream (or other radiology video stream).

[0078] In step 420, method 400 includes detecting / identifying an anatomical structure (e.g., fetal head, abdomen, or femur) using a neural network and placing measurement points relative to the anatomical structure on the image frame. Depending on the implementation, the measurement points may be placed by the same neural network that detects the anatomical structure or by a different neural network.

[0079] Anatomical structure detection based on machine learning (ML) or other neural network (NN)-based artificial intelligence (AI) can be, for example, through object detection or bounding box detection, classification, segmentation, or other related means. Examples of classification networks can be found, for example, in U.S. Provisional Patent Application No. 63 / 293,232, titled "Methods and systems for clinical scoring a lung ultrasound," filed on December 23, 2021, and U.S. Provisional Patent Application No. 63 / 294,501, titled "Machine-learning image processing independent of reconstruction filter," filed on December 29, 2021, each of which is incorporated by reference as if fully set forth herein. Examples of bounding box object detection can be found in Indian Patent Application No. 202141034243 (International Application No. PCT / EP2022 / 070410), titled "Generating location data," filed on July 29, 2021, which is incorporated by reference as if fully set forth herein. Examples of image or video segmentation can be found, for example, in U.S. Provisional Patent Application 63 / 325,660, titled "Methods and systems for ultrasound-based structure localization using image and user input," filed on March 31, 2022, and U.S. Publication No. 2022 / 0198669, titled "Segmentation and view guidance in ultrasound imaging and associated devices, systems, and methods," each of which is incorporated by reference as if fully set forth herein.

[0080] In step 430, method 400 includes filtering anatomical structure detection and measurement points for reasonableness. For example, if certain relevant anatomical features can be identified in an image frame, or if the eccentricity of an elliptical feature falls within a certain specified range, the anatomical structure detection can be considered valid; otherwise, it can be considered invalid. In some aspects, an invalid anatomical structure detection can automatically invalidate any measurement points placed with reference to that anatomical structure. A measurement point can be considered valid if it falls within the bounding box of a validly detected anatomical structure or within a specific region of the detected anatomical structure. For example, if a femoral endpoint occurs outside the femoral bounding box or near the middle of the femur rather than near one of the ends, the femoral endpoint can be considered invalid. This filtering provides a check and balance against false predictions of the (one or more) neural networks, such that the system is configured to generate a first measurement value based on the identification of anatomical structures.

[0081] In step 440, method 400 includes using the filtered measurement points to calculate one or more measurement values. For example, the measurement results can include the length, width, area, diameter, or perimeter of the identified anatomical structure, or other anatomical measurement results. For example, the measurement can be made directly using the measurement points (e.g., the length / distance between two measurement points). For example, the femoral length can be the length / distance between the femoral endpoints. In other cases, the measurement results can combine multiple measurement values. For example, the head circumference can be a combination of the biparietal diameter (BPD, the first length / distance between two measurement points) and the occipitofrontal diameter (OFD, the second length / distance between two measurement points). For example, HC = 1.62×(BPD + OFD). Similarly, the abdominal circumference (AC) can be obtained from the anteroposterior abdominal diameter (APAD) and the transverse abdominal diameter (TAD) by AC = π(APAD + TAD) / 2 = 1.57(APAD + TAD). The displayed measurement value can be the numerical value of the measurement itself, or it can be a derived value calculated using the first measurement value. For example, OFD and BPD can be obtained as direct measurement results (e.g., distances), while HC is a derived value calculated using OFD and BPD. The displayed measurement result can also be a visual depiction of the measurement points (with or without a line / curve between the points), or it can be or include Figures 21 - 23 the convergence progress indication discussed in. The measurement values can be two or more different measurement results of the same anatomical feature. Two or more consecutive measurement values of the same anatomical feature can be the same or different from each other. For example, the values can be different because different image frames show different parts of the same anatomical structure (e.g., different image planes), rather than because different anatomical features are being measured.

[0082] In an example, one or more metrics are calculated based on detections. The metrics can be hard-coded into the system or can be selected in real-time or near real-time to represent clinically relevant parameters derived from the detections. For example, in a screening / classification context, an operator may be interested in picking up features of any size as long as they have been detected with sufficient confidence. In this setting, a metric defined as the maximum confidence of all detections (but independent of the detection area) of a feature type may be appropriate. Alternatively, in a diagnostic context, the operator may already know that very small findings are not clinically significant while larger findings may indicate pathology. Thus, the metric can be defined as the maximum area of all detection bounding boxes or the maximum value of the product of the area and confidence of all detections. In this way, the metric will be insensitive to small findings (even those with high confidence).

[0083] In step 450, method 400 includes determining the convergence of the measurement(s) and / or the alignment of the image frame relative to the plane of the anatomical structure. For example, when different measurements are calculated in each cycle of the method (e.g., each image frame of an ultrasound video stream), the method can include keeping track of the maximum value of the measurement results and can consider the measurement results to be "converged" when no larger value has been detected after a certain amount of time or a certain number of frames, or "not converged" if the maximum value is still changing, and thus the user needs to keep acquiring the live ultrasound video to improve the measurement results. In some aspects, the method uses cues (such as the eccentricity of an ellipse or the presence of a particular anatomical feature) to determine whether the image frame has been captured at the desired location, alignment, or depth, and if the image frame containing the maximum value is incorrectly placed or aligned, the measurement results can be considered "not converged". According to an embodiment, guidance can be provided to the user to move or reorient the ultrasound probe to obtain better measurement values, or the user interface can simply indicate that the measurement values have not converged, i.e., prompt the user to keep moving the ultrasound probe.

[0084] In step 460, method 400 includes displaying the measurement results to the user (e.g., adjacent to or superimposed on the current image frame of the video stream). Non-limiting example screen displays can be found below in Figures 8 - 23 Since the method can be performed in real-time, the execution can then return to step 410 to wait for the next image frame in the video stream to be received.

[0085] This document provides flowcharts for illustrative purposes; those of ordinary skill in the art will recognize countless variations that still fall within the scope of the present disclosure. For example, the logic of the flowchart may be shown as sequential. However, similar logic may be parallel, massively parallel, object-oriented, real-time, event-driven, cellular automata, etc., while achieving the same or similar functions. To perform the methods described herein, a processor may divide each step described herein into multiple machine instructions and may execute these instructions at a rate of hundreds, thousands, millions, or billions per second in a single processor or across multiple processors. Such rapid execution may be necessary to perform the methods as described herein in real time or near real time. For example, to measure anatomical features in a real-time ultrasound video stream, it may be necessary to execute steps 410-440 faster than the frame rate of the video (e.g., 30, 60, or 100 times per second).

[0086] Figure 5 is a schematic graphical representation in block diagram form of at least a portion of an automatic measurement point detection system in accordance with aspects of the present disclosure. Image frames 320 of an ultrasound video stream are received by an object detector / analyzer 520 (e.g., a deep learning network or other machine learning neural network or artificial intelligence), which performs feature detection (e.g., detecting a specified anatomical feature) on the image frames 320 and determines the locations of measurement points. In some aspects, the object detector / analyzer 520 includes two separate neural networks (e.g., an object detector and an object analyzer), although a single neural network may be easier to train and more computationally efficient. The object detector / analyzer may output an annotated video frame 530 that marks the measured anatomical features 540 and the measurement points 550.

[0087] The object detector / analyzer 520 may implement or include any suitable type of learning network. For example, in some aspects, the object detector / analyzer 520 may include a neural network, such as a convolutional neural network (CNN). Additionally, the convolutional neural network may additionally or alternatively be an encoder-decoder type network, or may utilize a backbone architecture based on other types of neural networks, such as object detection networks, classification networks, etc. An example backbone network is the Darknet YOLO backbone (e.g., Yolov3) that can be used for object detection. For example, the CNN may include a set of N convolutional layers, where N may be any positive integer. When the CNN is a backbone, the fully connected layers may be omitted. The CNN may also include max pooling layers and / or activation layers. Each convolutional layer may include a set of filters configured to extract features from the input (e.g., from the image frame 320). The value of N and the size of the filters may vary according to aspects. In some cases, the convolutional layers may utilize any non-linear activation function, such as the leaky rectified non-linear (ReLU) activation function and / or batch normalization. The max pooling layer gradually reduces the high-dimensional output to the dimension of the desired result (e.g., the bounding box of the detected feature).

[0088] The fully connected layer may be referred to as a perceptron or perception layer. In some aspects, the perceptron / perception layer and / or the fully connected layer may be found in the object detector / analyzer 520 (e.g., a multi-layer perceptron) to allow downstream processing (e.g., detection, segmentation, etc.). The object detector / analyzer 520 may also include max pooling layers and / or activation layers. The max pooling layer gradually reduces the high-dimensional output to the dimension of the desired result (e.g., the bounding box of the region of interest).

[0089] Including these descriptions is for illustrative purposes; those of ordinary skill in the art will understand that other types of AI learning models with features similar to or different from the above features may be alternatively or additionally used without departing from the spirit of the present disclosure.

[0090] Then the annotated image frame 530 or the information used to generate the annotation is sent to the error filtering module 560, which filters the anatomical structure detections and measurement points for reasonableness as described above. For example, if a specified anatomical landmark is identified in the image frame, or if the eccentricity of the elliptical feature falls within a certain specified range, the anatomical structure detection may be considered valid. If the measurement point falls within the correct (and valid) anatomical structure, the measurement point may be considered valid.

[0091] Then the filtered measurement points are sent to the measurement result calculation module 570, which performs calculations such as determining the distance between two measurement points, the perimeter or area of an ellipse defined by four measurement points, etc.

[0092] The measurement result calculation module generates one or more measurement results 580, which are then displayed on a display 590 together with the annotated image frame 530, the original image frame 320, the original image frame 320 with bounding boxes, or others, as follows.

[0093] Block diagrams are provided herein for exemplary purposes; those of ordinary skill in the art will recognize numerous variations that still fall within the scope of the present disclosure. For example, a block diagram may illustrate a particular arrangement of components, modules, services, steps, processes, or layers that generate a particular data stream. It should be understood that some aspects of the systems disclosed herein may include additional components, some aspects may not include some of the components shown, and the arrangement of components may be different from that shown, resulting in different data streams while still performing the methods described herein.

[0094] Figure 6 is a schematic graphical representation in the form of a block diagram of at least a portion of an automatic measurement point detection system according to aspects of the present disclosure. Image frames 320 of a radiology video stream 310 generate a sequence of measurement results 580 (e.g., one measurement result per image frame 320). The measurement results 580 are received by a convergence and plane alignment module 630, which determines the convergence of the (one or more) measurements and / or the plane alignment of the image frames relative to the anatomical structure, as described above. For example, when different measurements are calculated for each image frame, the convergence and plane alignment module 630 may keep track of the maximum value for the measurement results and may use cues such as the eccentricity of an ellipse or the presence of specific anatomical features to determine whether an image frame has been captured at the desired location, alignment, or depth.

[0095] Then, the convergence and plane alignment module 630 generates a graphical display 640 that can be superimposed on or displayed adjacent to the image frames. For example, if the image frame containing the maximum value is incorrectly placed or aligned, or if the maximum calculated value of the measurement results is still increasing over time, the measurement results can be shown textually or graphically as "not converged" or "in progress". According to an embodiment, the graphical display 640 may also provide guidance to the user to move or reorient the ultrasound probe.

[0096] Figure 7A is a schematic graphical overview in the form of a block diagram of a training mode 700 for an object detector / analyzer 520 according to aspects of the present disclosure. In Figure 7A the example shown, a training data set 705a including an ultrasound video stream with manually marked anatomical structure localizations and measurement points (e.g., surrounded by bounding boxes) is fed into an untrained object detector / analyzer 710a during an iterative training process familiar to those of ordinary skill in the art.

[0097] In particular, for object detection and measurement point placement using a convolutional neural network, a large number of sample images are manually annotated by experts to depict the localization of features of interest. The parameters of the network model (e.g., the weights at each artificial neuron) are initialized with an initial value A which can be a random value or with a result from training on a previous dataset. During the iterative process, the network is used to perform detection inference on the training images, the results are compared with the ground truth annotations, and the optimizer is used to adjust the network parameters B until the measure of detection accuracy is maximized.

[0098] Thus, the output of the training process 700 is a trained object detector / analyzer 710b, where the parameters B (e.g., weights) are optimized for detecting features in the training video 705a.

[0099] Figure 7B is a schematic diagrammatic overview in block diagram form of the validation mode 702 of the object detector / analyzer 520 according to aspects of the present disclosure. In the validation mode, a set of manually annotated validation videos (e.g., videos including bounding boxes around any anatomical structures or measurement points identified by experts in each frame of each video) are fed into the trained object detector / analyzer 710b to determine whether the trained object detector / analyzer 710b detects the manually identified features in the validation video 705b at a desired level of accuracy.

[0100] In some cases, the performance of the trained object detector / analyzer 710b can be considered to be below the desired level of accuracy. In such cases, the parameters B (e.g., weights) of the trained object detector / analyzer 710b can be adjusted until the detection accuracy for the validation dataset (or the validation dataset plus the training dataset) reaches the desired accuracy. In such cases, the output of the validation process can be a trained object detector / analyzer 520 which can be the same as the trained object detector / analyzer 710b except for the adjusted parameters C (e.g., weights). In other cases, the performance of the trained object detector / analyzer 710b can be considered to be sufficient and thus the parameters are not adjusted, and the trained object detector / analyzer 520 can be the same as the trained object detector / analyzer 710b (e.g., using the same weights).

[0101] Figure 7CSchematic diagrammatic overview in block diagram form of the inference mode or clinical usage mode 704 of the object detector / analyzer 520 according to aspects of the present disclosure. In clinical use, an ultrasound video or video stream 310 is fed to the trained and validated object detector / analyzer 520 for analysis. In some cases, the video stream 310 can be acquired and analyzed in real time or near real time. In other cases, the video stream can be retrieved from memory, a storage device, or a network. Then, the trained and validated object detector / analyzer 520 produces an annotated version 720 of the video stream 310 as output, the annotated version including measurement points and detected anatomical structures.

[0102] Thus, in each frame of the video stream, the object detector / analyzer 520 is run to determine the localization and confidence value of features (e.g., anatomical structures and measurement points). The localization can be determined in the form of a bounding box tightly enclosing the feature. Other forms of localization are possible, such as a binary mask indicating the image pixels that are part of the feature, or a polygon or other shape enclosing the feature. For any such localization, the center and area of the detection can be determined. The confidence value can be determined as a normalized value in the range [0, 1], where 0 indicates the lowest confidence and 1 indicates the highest confidence that the feature is present at that location.

[0103] The detection step can be based on conventional image processing including thresholding, filtering, and texture analysis, or can be based on machine learning, particularly using a deep neural network. A particular advantageous implementation of the detection is to use a Yolo-type network such as Yolo3. For each frame of the video stream, an exemplary output of the detection step is a list of detections for one or more types of features of interest. For example, each element in the detection list can include at least the confidence and area of the detection (and typically also the location, width, and height). Thus, for each frame i and feature f, there is a list of detections {x, y, w, h; c} f,i where x, y represent the center coordinates, w, h represent the width and height, and c represents the confidence value of the detection. Other means of representing the detection can alternatively or additionally be used without departing from the spirit of the present disclosure.

[0104] Figure 8 is an example screen display 800 of an automatic measurement point detection system according to aspects of the present disclosure. In Figure 8 the example shown, the screen display 800 includes the location of an annotated image frame 720 with superimposed measurement results 580. The screen display 800 also includes a detected anatomical structure display area 810, a measurement report area 820 containing the final measurement results 825, a convergence progress indicator 830, and two control settings 840 and 850.

[0105] Figure 9 is an example training image 900 for an automatic measurement point detection system according to aspects of the present disclosure. In Figure 9 the example shown, the training image 900 is an image of a fetal head and includes a head bounding box 910, left and right biparietal diameter (BPD) measurement point bounding boxes 920L and 920R, and anterior-posterior occipitofrontal diameter (OFD) measurement point bounding boxes 930F and 930R. Each bounding box includes a measurement point 940 and a region 950 that appears primarily outside the head and a region 960 that appears primarily inside the head.

[0106] Figure 10 is an example object detection image 1000 of an automatic measurement point detection system according to aspects of the present disclosure. Visible are the head 1010, the cerebral falx 1020, the cavum septum pellucidum (CSP) 1030, the choroid plexus 1040, and the posterior lateral ventricles (PLV) 1050. These anatomical features are typically found in the imaging plane most suitable for head circumference measurement and can thus be used as indicators that the imaging plane of the ultrasound probe is correctly positioned and aligned for head circumference measurement. These specific anatomical landmarks are described here for exemplary purposes. Depending on the anatomical structure being measured, other landmarks may alternatively or additionally be used, or an appropriate measurement plane may be detected based on geometry (e.g., the eccentricity of an ellipse) without reference to other anatomical landmarks.

[0107] Figure 11 is an example "detected object" display 810 of an automatic measurement point detection system according to aspects of the present disclosure. In Figure 11 the example shown, the detected object display 810 includes a plurality of anatomical features 1110, each having its own checkbox 1120. The checkbox 1140 may indicate the anatomical structures currently visible in the ultrasound video stream, the anatomical structures previously detected, or the anatomical structures whose detection has enabled the associated measurement results to converge. In the example, when all checkboxes 1120 are checked, the check is considered complete.

[0108] Figure 12 is an example inference image 1200 generated by an automatic measurement point detection system according to aspects of the present disclosure. The image includes measured anatomical features 540 (in this case, an ellipse representing the anatomical structure, which is the fetal head) and measurement points 550. Also visible are the biparietal diameter (BPD) line 1240 and the occipitofrontal diameter (OFD) line 1250. Depending on the implementation, these annotations may be displayed to the user, for example, together with the measurement values derived from them.

[0109] Figure 13An example object detection image 1300 of an automatic measurement point detection system according to aspects of the present disclosure is shown. Visible are detected anatomical structures 1310 and measurement points 1320L, 1320R, 1330F, and 1330R, as well as confidence values 1340 for the detections. The confidence threshold can be a user-selectable value such that, for example, anatomical features are shown only if the detection confidence 1340 of the anatomical feature exceeds 50%, 80%, 90%, 95%, etc. A higher confidence threshold may increase the process time because it may be more difficult to acquire the anatomical structure and thus take longer. However, a higher confidence threshold can also improve the accuracy of the results.

[0110] Figure 14 An example object detection image 1400 of an automatic measurement point detection system according to aspects of the present disclosure is shown. In this case, the detections are shown as bounding boxes 1410, 1420R, 1420L, 1430F, and 1430R. Depending on the implementation, there may be a threshold associated with the center point of the measurement point box inside the anatomical structure box. For example, the measurement points of the head may not necessarily have to be exactly inside the head (or the head anatomical structure box), but can be within a threshold distance (+ / -) of the anatomical structure box.

[0111] Figure 15 An example filter control setting of an automatic measurement point detection system according to aspects of the present disclosure is shown. Visible are a confidence filter control 840 and an intersection over union (IOU) filter control 850. In the example, the confidence filter control 840 can be used to (e.g., in real-time) adjust the confidence threshold for anatomical structure detection. The IOU filter control can be used to set a threshold for clearing duplicate detections. For example, if two femoral endpoints are detected within a few millimeters of each other, this can be considered a single femoral endpoint and thus a single measurement point, while if two femoral endpoints are detected several centimeters apart, they can be considered two different endpoints of the same femur.

[0112] Figure 16 An example training image 1600 of an automatic measurement point detection system according to aspects of the present disclosure is shown. In Figure 16 the example shown, the training image 1600 is an image of a fetal abdomen and includes an abdominal bounding box 1610, left and right transverse abdominal diameter (TAD) measurement point bounding boxes 1620L and 1620R, and anterior-posterior abdominal diameter (APAD) measurement point bounding boxes 1630F and 1630R.

[0113] Figure 17Example object detection image 1700 of an automatic measurement point detection system according to aspects of the present disclosure. Visible are the abdomen 1710, the stomach 1720, the spine 1730, and the umbilical vein 1740. In the example, detection of these anatomical features can indicate that the imaging plane of the ultrasound probe is properly positioned and aligned for measuring the abdominal diameter or circumference.

[0114] Figure 18 Example inference image 1800 generated by an automatic measurement point detection system according to aspects of the present disclosure. The image includes a measured anatomical feature 540 (in this case, an ellipse representing an anatomical structure, which is the fetal abdomen) and measurement points 550. Also visible are the anterior-posterior abdominal diameter (APAD) line 1840 and the transverse abdominal diameter (TAD) line 1850. Depending on the implementation, these annotations can be displayed to the user, for example, together with the measurements derived from them.

[0115] Figure 19 Example training image 1900 for an automatic measurement point detection system according to aspects of the present disclosure. The image shows a bounding box 1910 of the anatomical structure of the fetal femur bone 1915, and a bounding box 1920 of the endpoints of the femur bone 1915. The training image can be fed into an untrained neural network, such as, for example Figure 7A as described.

[0116] Figure 20 Example inference image 2000 generated by an automatic measurement point detection system according to aspects of the present disclosure. The image includes a measured anatomical feature 540 (in this case, a straight line representing an anatomical structure, which is the fetal femur bone) and measurement points 550 indicating the endpoints of the femur. In this example, the measured anatomical feature 540 is simply a line connecting two measurement points 550. Depending on the implementation, these annotations can be displayed to the user, for example, together with the measurements derived from them.

[0117] Depending on the implementation, the measured anatomical feature 540 can be or include the patient's head circumference, BPD, OPD, femur length, abdominal circumference, TAD, APAD, or other anatomical features.

[0118] Figure 21 Example convergence progress display 830 of an automatic measurement point detection system according to aspects of the present disclosure. In the Figure 21 example shown, the convergence progress display 830 includes a plurality of measurements 580 and measurement labels 2185 for the measurement results 2180, each measurement result having a checkbox 2110 and a progress indicator 2120. A "waiting" or "in progress" indicator 2130 and a selected checkbox 2115 are visible on one of the measurement results 2180, indicating that this measurement result 2180 is currently in progress and has not yet converged.

[0119] In the example, while the measurement result 2180 is waiting to converge, each time a larger (or otherwise "more converged") value of the measurement result 2180 is received, the measured value 580 for the measurement result 2180 is updated, and the progress indicator 2120 for the measurement result 2180 becomes longer. In some cases, the progress indicator 2120 can also change color, such that for example when the measured value 580 changes rapidly, the corresponding progress indicator 2120 is red, and when the measured value 580 changes less frequently, the corresponding progress indicator 2120 is yellow, and when the measured value 580 no longer changes, the corresponding progress indicator 2120 is green. Thus, a green indicator 2120 for the measurement result 2180 can indicate that all desired measurement results have converged and the check is complete. In other cases, the color of the measured value itself can change from red to yellow and from yellow to green in the same manner, or alternatively or additionally other visual indicators of convergence or non-convergence can be provided.

[0120] In Figure 21 the example shown, the convergence progress display 830 also includes a plane alignment indicator 2140. The plane alignment indicator 2140 includes a desired imaging plane indicator 2150 and an actual imaging plane indicator 2160 in a bubble level configuration. Thus, the position and orientation of the actual imaging plane indicator 2160 relative to the desired imaging plane indicator can provide guidance to a clinician or other user to move or re-orient the ultrasound imaging probe. For example, when the transducer is aligned, the small circle can be centered within the larger circle. For example, the alignment can be based on the intersection of BPD and OFD, or based on the intersection of TAD / APAD. Other types of plane alignment indicators can be used alternatively or additionally, including but not limited to lines, high watermark indicators, etc.

[0121] Figure 22 is an example convergence progress display 830 of an automatic measurement point detection system according to aspects of the present disclosure. In Figure 22 the example shown, the convergence progress display 830 includes a measurement result label 2185, a numerical scale 2220, a maximum measurement result 2230, and a current measurement result 2240. In the example, each time a new maximum value of the current measurement result is received, the maximum measurement result is updated to a larger value. When a certain duration or a certain number of measurement results have occurred without receiving a larger value, the maximum measurement result 2230 can be considered the converged and accurate measurement result corresponding to the measurement result label 2185.

[0122] Figure 23 is an example convergence progress display 830 of an automatic measurement point detection system according to aspects of the present disclosure. In Figure 23In the example shown, the convergence progress display 830 includes a measurement result label 2185, a numerical scale 2220, a maximum measurement value 2230, and a current measurement value 2240. However, in this case, the progress of the current measurement result over time is represented as a graph 2320, whose X-axis 2310 indicates time or the number of frames, and whose Y-axis represents the numerical scale 2220. When a certain duration or a certain number of measurement results have occurred without receiving a current measurement value 2240 greater than the maximum measurement value 2230, the maximum measurement value 2230 can be considered a convergent and accurate measurement result corresponding to the measurement result label 2185.

[0123] In the above examples and aspects, many variations are possible. For example, the systems, methods, and devices described herein are not limited to neonatal ultrasound applications. Instead, the same techniques can be applied to images of other organs or anatomical systems (such as the heart, brain, digestive system, vascular system, etc.) or their pathologies. For example, for cardiac ultrasound, the left ventricular long-axis and short-axis distances can be measured, and for vascular ultrasound, the kidney width and length and / or the liver length and width can be measured, and so on. The anatomical structures measured can include any one of CSP, PLV, etc. (as Figure 10 shown), or the stomach, spine, umbilical vein (as Figure 17 shown), or any other anatomical structure of a patient for which a neural network has been trained.

[0124] In addition, the techniques disclosed herein are also applicable to other medical imaging modalities obtained from medical imaging devices where 3D data is available, such as other ultrasound applications, camera-based video, X-ray video, and 3D volume images, such as computed tomography (CT) scans, magnetic resonance imaging (MRI) scans, optical coherence tomography (OCT) scans, or intravascular ultrasound (IVUS) pullback sequences. The techniques described herein can be used in various environments, including the emergency department, intensive care, inpatient, and out-of-hospital settings.

[0125] Accordingly, the logical operations constituting aspects of the techniques described herein are variously referred to as operations, steps, objects, layers, elements, components, algorithms, or modules. Moreover, it should be understood that these can occur or be executed or arranged in any order, unless explicitly stated otherwise or the claim language inherently requires a specific order.

[0126] All directional references (e.g., above, below, inside, outside, upward, downward, left, right, lateral, front, rear, top, bottom, over, under, vertical, horizontal, clockwise, counterclockwise, proximal, and distal) are for identification purposes only to assist the reader in understanding the claimed subject matter and do not create a limitation, particularly as to the position, orientation, or use of the automated measurement point detection system. Connecting references (e.g., attached, coupled, connected, joined, or "in communication") shall be construed broadly and may, unless otherwise stated, include intermediate members between elements and relative movement between elements. Thus, a connecting reference does not necessarily imply that two elements are directly connected and in a fixed relationship to each other. The term "or" shall be construed to mean "and / or" rather than "exclusive or". The word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. Unless otherwise stated in the claims, the stated values should be construed as illustrative only and not as restrictive.

[0127] The foregoing specification, examples, and data provide a complete description of the structure and use of the exemplary aspects of the automated measurement point detection system as defined in the claims. While the various aspects of the claimed subject matter have been described above to a certain degree of particularity or with reference to one or more individual aspects, many modifications may be made to the disclosed aspects without departing from the spirit or scope of the claimed subject matter.

[0128] Other aspects are also contemplated. All subject matter intended to be included in the above description and shown in the drawings should be construed as illustrative of particular aspects and not as limiting. Changes may be made to the details or structure without departing from the basic elements of the subject matter defined in the appended claims.

Claims

1. A system, comprising: A display; And A processor configured to communicate with the display and a medical imaging device, wherein the processor is configured to: Receive a first anatomical image frame obtained by the medical imaging device during live imaging; During the live imaging, identify a plurality of first measurement points of anatomical features in the first anatomical image frame, wherein the identification of the first measurement points is performed by a first neural network trained to identify where the measurement points are located within the anatomical image frame, such that the identification of the first measurement points is automatically performed without user input for locating the plurality of first measurement points in the first anatomical image frame; Generate a first measurement value for the anatomical features of the first anatomical image frame based on the plurality of first measurement points; and Output a screen display to the display based on the first measurement value.

2. The system according to claim 1, wherein The processor is configured to: During the live imaging, identify a plurality of second measurement points of the anatomical features in a second anatomical image frame obtained by the medical imaging device during the live imaging; And Generate a second measurement value for the anatomical features of the second anatomical image frame based on the plurality of second measurement points, wherein the screen display is based on the first measurement result and the second measurement result.

3. The system according to claim 2, wherein The second frame is obtained immediately after the first frame.

4. The system according to claim 2, Among them, The processor is configured to determine whether convergence of the measurement values has occurred based on the first measurement value and the second measurement value, Wherein the processor is configured to provide the screen display based on determining whether convergence of the measurement values has occurred.

5. The system according to claim 4, wherein, The screen display includes the progress of the convergence of the measurement values.

6. The system according to claim 4, wherein, The processor is configured to determine whether convergence of the measurement values has occurred based on the difference between the first measurement result and the second measurement result.

7. The system according to claim 5, wherein, To determine whether convergence of the measurement values has occurred, the processor is configured to determine whether the second measurement result is less than the first measurement result.

8. The system according to claim 7, Among them, If convergence of the measurement values has occurred, the processor is configured to select the larger value of the first measurement value or the second measurement value as the converged measurement value; And Wherein the screen display includes the converged measurement value.

9. The system according to claim 1, wherein The screen display includes: The first anatomical image frame; and The plurality of first measurement points superimposed on the first anatomical image frame.

10. The system according to claim 1, wherein, The screen display includes: The first anatomical image frame; and An indication of the anatomical features superimposed on the first anatomical image frame.

11. The system according to claim 1, Among them, The processor is configured to identify an anatomical structure including the anatomical features in the first anatomical image frame, Wherein the processor is configured to generate the first measurement value based on the identification of the anatomical structure.

12. The system according to claim 11, wherein, The screen display includes: The first anatomical image frame; and An indication of the anatomical structure, which is superimposed on the first anatomical image frame.

13. The system according to claim 11, wherein Use a second neural network to perform the recognition of the anatomical structure.

14. The system according to claim 13, wherein The first neural network and the second neural network are the same neural network.

15. The system according to claim 1, further comprising the medical imaging device.

16. A method, comprising: Receiving, by a processor in communication with a medical imaging device, an anatomical image frame obtained by the medical imaging device during live imaging; When the live imaging is in progress, identifying a plurality of measurement points of anatomical features in the anatomical image frame, wherein the identification of the measurement points is performed by a neural network that is trained to identify where the measurement points are located within the anatomical image frame such that the identification of the first measurement points is automatically performed without user input for positioning the plurality of first measurement points in the anatomical image frame; Generating a measurement value for the anatomical feature of the anatomical image frame based on the plurality of measurement points; Outputting a screen display to a display in communication with the processor based on the measurement value.

Images