Multi-frame ultrasound video with video level feature classification based on frame level detection

Abstract

A system is provided that includes a display and a processor in communication with the display. The processor is configured to receive an ultrasound video comprising a plurality of frames of an anatomical structure obtained by the ultrasound probe. The processor is further configured to generate at least one frame level metric for each frame detected to include a pathology based on the ultrasound video, and generate at least one video level metric related to the pathology based on the frame level metrics for the frames including the pathology. The processor is further configured to generate a video level classification of the pathology based on the video level metric related to the pathology and output it to the display.

Description

Technical Field

[0001] The subject matter described herein relates to devices, systems, and methods for automatically locating and classifying features (eg, anatomical features such as pathologies) in ultrasound videos. Background Art

[0002] Ultrasound imaging is often used for diagnostic purposes in office or hospital settings. For example, lung ultrasound (LUS) is an imaging technology deployed at the point of care to help evaluate lung diseases and infectious diseases, including COVID-19 pneumonia. Important clinical features—such as B-lines, fused B-lines, pleural line changes, consolidation, and pleural effusions—can be visualized under LUS, but accurately identifying these clinical features can be a challenging skill to learn and involves reviewing a video or "image loop" of the entire acquisition. The effectiveness of LUS use can depend on operator experience, image quality, and the choice of imaging settings.

[0003] Point-of-care lung ultrasound is increasingly accepted for detecting a variety of lung conditions. However, identifying clinically relevant ultrasound features and artifacts requires expertise and can be time-consuming, particularly because lung ultrasounds are typically acquired in image loops.

[0004] The information included in this background section of the specification, including any references cited herein and any description or discussion thereof, is included for technical reference purposes only and is not to be construed as subject matter by which the scope of the present disclosure is to be limited. Summary of the Invention

[0005] An ultrasound video feature classification system with a machine learning algorithm (e.g., a neural network) is disclosed. The ultrasound video feature classification system disclosed herein has a specific but non-exclusive utility for identifying the presence, likelihood, and / or severity of pathology in ultrasound videos (such as lung ultrasound videos). The ultrasound video feature classification system detects features in individual frames of a video, obtains per-frame metrics based on the identification, obtains video-level metrics based on the per-frame metrics, and then obtains a classification for the entire video (e.g., a video-level classification) based on the video-level metrics. The video-level classification can, for example, help a user (e.g., a clinician) understand the video and / or identify which video(s) are clinically relevant from many videos collected from a patient. The ultrasound video feature classification system includes a training mode in which the machine learning algorithm is trained using labeled ultrasound video data. The ultrasound video feature classification system also includes an inference mode in which the machine learning algorithm generates a classification of features identified in the video. These classifications can, for example, be superimposed on the video or displayed near the video.

[0006] A system of one or more computers can be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system, which, when in operation, causes or enables the system to perform the actions. One or more computer programs can be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions. One general aspect includes a system comprising a display and a processor configured to communicate with the display, wherein the processor is configured to: receive an ultrasound video of an anatomical structure acquired by an ultrasound probe, wherein the ultrasound video may include a plurality of frames; generate at least one frame-level metric for each frame in the plurality of frames that is detected as including a pathology based on the ultrasound video; generate at least one video-level metric related to the pathology based on the at least one frame-level metric for the frames in the plurality of frames that are detected as including the pathology; generate a video-level classification of the pathology based on the at least one video-level metric related to the pathology; and provide a screen display to the display that may include the video-level classification. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0007] Implementations may include one or more of the following features. In some embodiments, the at least one frame level metric is calculated by an object detection machine learning network. In some embodiments, the screen display may further include the ultrasound video or a frame of the ultrasound video. In some embodiments, the at least one frame level metric may include: the number of detections of the pathology within the frame; or the area or confidence level of a bounding box, binary mask, or polygon representing the pathology. In some embodiments, the pathology may include at least one of the following: B-lines, fused B-lines, pleural line changes, consolidation, or pleural effusion. In some embodiments, the at least one video level metric is selected from a list that may include: a maximum confidence of all detections of the pathology, a maximum area of all detections of the pathology, a number of detections of the pathology exceeding a minimum confidence level, a number of detections of the pathology exceeding a minimum area, a maximum product of confidence and area of the pathology, an average of the highest confidence in each frame of the pathology, an average of the maximum area in each frame of the pathology, an average of the maximum product of confidence and area in each frame of the pathology, an average number of detections of the pathology exceeding a minimum confidence in each frame, a number of frames or a percentage of frames containing a detection of the pathology exceeding a minimum confidence, a product of confidences of the highest confidence detections of the pathology, a maximum product of confidences of the highest confidence detections of the pathology, or any combination thereof. In some embodiments, generating the video level classification of the pathology involves at least one of: a threshold, a regression, or a classification machine learning network. In some embodiments, the video level classification may include at least one of: a binary classification, a discrete classification, or a numerical classification. In some embodiments, the processor is further configured to: generate at least one second frame-level metric for each frame in the plurality of frames detected as including a second pathology based on the ultrasound video; generate at least one second video-level metric related to the second pathology based on the at least one second frame-level metric for the frame in the plurality of frames detected as including the second pathology; generate a second video-level classification of the pathology based on the at least one second video-level metric related to the second pathology; and provide a screen display to the display that may include the second video-level classification. In some embodiments, the processor is further configured to: generate a classification of a disease state associated with the pathology and the second pathology based on the video-level classification of the pathology and the second video-level classification of the second pathology; and provide a screen display to the display that may include the classification of the disease state. Implementations of the described technology may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0008] One general aspect includes a method comprising, using a processor configured to communicate with a display, performing the following operations: receiving an ultrasound video of an anatomical structure acquired by an ultrasound probe, wherein the ultrasound video may include a plurality of frames; generating, based on the ultrasound video, at least one frame level metric for each frame in the plurality of frames that is detected as including a pathology; generating, based on the at least one frame level metric for the frames in the plurality of frames that are detected as including the pathology, at least one video level metric related to the pathology; generating, based on the at least one video level metric related to the pathology, a video level classification of the pathology; and providing, to the display, a screen display that may include the video level classification. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0009] Implementations may include one or more of the following features. In some embodiments, calculating the at least one frame-level metric involves an object detection machine learning network. In some embodiments, the screen display may further include the ultrasound video or a frame of the ultrasound video. In some embodiments, the at least one frame-level metric may include: the number of detections of the pathology within the frame; or the area or confidence level of a bounding box, binary mask, or polygon representing the pathology. In some embodiments, the at least one video level metric is selected from a list that may include: a maximum confidence of all detections of the pathology, a maximum area of all detections of the pathology, a number of detections of the pathology exceeding a minimum confidence level, a number of detections of the pathology exceeding a minimum area, a maximum product of confidence and area of the pathology, an average of the highest confidence in each frame of the pathology, an average of the maximum area in each frame of the pathology, an average of the maximum product of confidence and area in each frame of the pathology, an average number of detections of the pathology exceeding a minimum confidence in each frame, a number of frames or a percentage of frames containing a detection of the pathology exceeding a minimum confidence, a product of confidences of the highest confidence detections of the pathology, a maximum product of confidences of the highest confidence detections of the pathology, or any combination thereof. In some embodiments, the pathology may include at least one of: B-lines, fused B-lines, pleural line changes, consolidation, or pleural effusion. Generating the video level classification of the pathology involves at least one of: a threshold, a regression, or a classification machine learning network. In some embodiments, the video level classification may include at least one of the following: a binary classification, a discrete classification, or a numerical classification. In some embodiments, the method may include: generating at least one second frame level metric for each frame of the plurality of frames detected as including a second pathology based on the ultrasound video; generating at least one second video level metric related to the second pathology based on the at least one second frame level metric for the frame of the plurality of frames detected as including the second pathology; generating a second video level classification of the pathology based on the at least one second video level metric related to the second pathology; and providing a screen display to the display that may include the second video level classification. In some embodiments, the method may include: generating a classification of a disease state associated with the pathology and the second pathology based on the video level classification of the pathology and the second video level classification of the second pathology; and providing a screen display to the display that may include the classification of the disease state. Implementations of the described technology may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0010] This summary is provided to introduce a series of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. A more comprehensive presentation of the features, details, utilities, and advantages of the ultrasound video feature classification system, as defined in the claims, is provided in the following written description of various aspects of the present disclosure and is illustrated in the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Illustrative aspects of the present disclosure will be described with reference to the accompanying drawings, in which:

[0012] Figure 1 is a schematic diagrammatic representation of an ultrasound imaging system according to aspects of the present disclosure.

[0013] Figure 2 is a schematic diagram of a processor circuit according to aspects of the present disclosure.

[0014] Figure 3 is a schematic diagrammatic representation of a radiology video, image loop, or video clip according to aspects of the present disclosure.

[0015] Figure 4 is a schematic diagrammatic representation of a labeled ultrasound dataset according to aspects of the present disclosure.

[0016] Figure 5 is a schematic diagrammatic representation in the form of a flow chart of an example ultrasound video feature classification method according to aspects of the present disclosure.

[0017] Figure 6 is a schematic diagrammatic representation in the form of a block diagram of an ultrasound video feature classification system according to aspects of the present disclosure.

[0018] Figure 7 is a schematic graphical illustration in the form of a block diagram of the calculation of a per-frame metric according to aspects of the present disclosure.

[0019] Figure 8A is a schematic, graphical overview in the form of a block diagram of a training mode 800 for an object detector according to aspects of the present disclosure.

[0020] Figure 8B is a schematic, graphical overview in the form of a block diagram of a verification mode 802 of an object detector according to aspects of the present disclosure.

[0021] Figure 8C is a schematic diagram overview in the form of a block diagram of an inference mode or clinical usage mode of an object detector according to aspects of the present disclosure.

[0022] Figure 9Ais a schematic diagram overview in the form of a block diagram of a training mode for a classifier according to aspects of the present disclosure.

[0023] Figure 9B is a schematic diagram overview in the form of a block diagram of a validation mode of a classifier according to aspects of the present disclosure.

[0024] Figure 9C is a schematic diagram overview in the form of a block diagram of an inference mode or clinical usage mode of an ultrasound video classifier according to aspects of the present disclosure.

[0025] Figure 10 Example screen displays of an ultrasound video feature classification system according to aspects of the present disclosure are shown.

[0026] Figure 11 Example screen displays of an ultrasound video feature classification system according to aspects of the present disclosure are shown.

[0027] Figure 12 Example screen displays of an ultrasound video feature classification system according to aspects of the present disclosure are shown.

[0028] Figure 13 Example screen displays of an ultrasound video feature classification system according to aspects of the present disclosure are shown.

[0029] Figure 14 is a schematic diagrammatic representation in the form of a flow chart of an example diagnostic method according to aspects of the present disclosure. DETAILED DESCRIPTION

[0030] According to at least one aspect of the present disclosure, an ultrasound video feature classification system is provided that can identify pathologies and disease states at the level of an entire imaging cycle, rather than at the level of individual frames within the imaging cycle. This can allow, for example, small or low-confidence detections present across multiple frames of an imaging cycle to be given greater weight than if those same features were detected only in a single frame.

[0031] Point-of-care lung ultrasound is increasingly being accepted for detecting a variety of lung conditions. However, identification of clinically relevant ultrasound features and artifacts requires expertise, can be time-consuming, and would benefit from automation, particularly because lung ultrasound is typically acquired in image loops. Findings from many frames of an image loop need to be quickly summarized into a simple binary or multi-class classification of the presence / absence and / or severity of a clinical condition. Disclosed herein are systems, devices, and methods for automatically creating summary metrics from multiple frames of an image loop that are relevant to a clinical condition and for using these metrics in image loop classification.

[0032] Image processing methods, including AI-based processing, exist for processing individual images to classify or localize (detect) abnormalities. The challenge in lung ultrasound is to combine the findings from the entire imaging cycle (typically about 60 to 200 image frames), taking into account the type, severity, and size of the different abnormalities detected in the individual frames, and combining them into an overall actionable assessment for the physician.

[0033] Reviewing and recording LUS examinations can be time-consuming because the examination includes multiple image loops (up to 12 or 14 image loops in a complete standardized lung examination), and each image loop must be played back—often multiple times—to view all images and understand the dynamic changes between images in order to identify abnormalities. Automatically providing an overall classification of an image loop regarding the presence and / or severity of one or several features or abnormalities would help the physician in several ways because it would speed up their workflow and increase their confidence in using and evaluating lung ultrasound. At the same time, the physician needs to be able to see visual evidence of what information in the image loop the automatic classification is based on in order to increase their confidence in the output of the automatic processing.

[0034] Therefore, one challenge is to identify and localize potential features and anomalies in the many frames of a lung ultrasound image cycle and combine multiple possible detections into a single (binary or multi-class) classification scheme for the entire image cycle that matches the ground truth classification provided by expert physicians with high accuracy. Furthermore, this automated processing needs to occur very quickly (ideally in real time) so that the results are ready for display immediately after (or within seconds of) the image cycle is acquired.

[0035] Furthermore, automated classification results need to be user-interpretable. Therefore, it is important to visualize the clinically relevant features in the image loop frames that the algorithm uses for classification.

[0036] The present disclosure provides systems, devices, and methods for automated classification of ultrasound image cycles, wherein detection (ie, localization) results from each frame of an image cycle are aggregated to generate a single (ideally interpretable) metric for the image cycle as a whole.

[0037] The ultrasound video feature classification system includes the following elements: 1. Acquisition of at least one ultrasound image cycle. Acquiring an ultrasound image cycle and providing it to an image cycle classification processor. The image cycle includes a plurality of image frames, typically 10 to 300, acquired continuously over a period of several seconds (typically 1 to 10). 2. Providing the acquired image cycle to the image cycle classification processor for analysis. 3. An image cycle classification processor includes the following steps:

[0038] a. Process each frame of the image loop for the purpose of detecting (i.e., localizing) one or more features of interest (e.g., lung consolidation, pleural effusion). For each frame, the output of this processing is a list of one or more "detections," each of which includes at least localization information (e.g., the coordinates of a rectilinear box containing the detected feature) and a confidence score (reflecting the likelihood that the detection actually corresponds to the feature of interest).

[0039] b. Compute one or more metrics based on the detections in all frames of the image loop. Metrics can include the average or maximum confidence score of all detections, the average or maximum area of all detections, the average or maximum confidence-weighted area of all detections, or other handcrafted metrics, as described in detail below.

[0040] c. Automated processing of all calculated metrics from the image loop to derive a classification of the image loop as a whole into two or more categories. If a single metric was calculated in the previous step, classification can be achieved using simple thresholding. If multiple metrics are used, classification can be achieved using other known methods including logistic regression and machine learning methods.

[0041] The image loop classification results are then typically displayed in conjunction with a display of the image loop itself, providing an output of the determined image loop class and displaying the location of the detected features.

[0042] The present disclosure substantially aids in the classification of pathologies in radiology videos (such as ultrasound image loops) by improving the detection system's ability to understand ambiguous features that exist across multiple frames of the video. Implemented on a processor in communication with an ultrasound probe, the ultrasound video feature classification system disclosed herein provides a practical improvement in the ability of untrained or inexperienced clinicians to provide accurate diagnoses based on radiology videos. This improved pathology classification converts a subjective process that relies heavily on professional experience into an objective and repeatable process without the usual conventional need for training clinicians (such as emergency room personnel) to identify specific abnormalities in various organ systems of the body. This unconventional approach improves the functionality of ultrasound imaging systems by providing reliable feature classification and even diagnosis of certain disease states (rather than just raw image or frame-by-frame classification).

[0043] The ultrasound video feature classification system can be implemented as a process that is at least partially visible on a display and is operated by a control process running on a processor that accepts user input from a keyboard, mouse, or touch screen interface and communicates with one or more sensor probes. In this regard, the control process performs certain specific operations in response to different inputs or selections made at different times. Certain structures, functions, and operations of the processor, display, sensor, and user input system are known in the art, while other structures, functions, and operations are described herein to specifically implement novel features or aspects of the present disclosure.

[0044] These descriptions are provided for exemplary purposes only and should not be considered as limiting the scope of the ultrasound video feature classification system. Certain features may be added, removed, or modified without departing from the spirit of the claimed subject matter.

[0045] For the purpose of promoting understanding of the principles of the present disclosure, reference will now be made to the aspects shown in the accompanying drawings, and specific language will be used to describe these aspects. However, it should be understood that this is not intended to limit the scope of the present disclosure. Any changes and further modifications to the described devices, systems and methods and any further applications of the principles of the present disclosure are fully anticipated and included in the present disclosure, as would be generally expected by those skilled in the art to which the present disclosure relates. In particular, it should be fully anticipated that the features, components and / or steps described in relation to one aspect can be combined with the features, components and / or steps described in relation to other aspects of the present disclosure. However, for the sake of brevity, multiple iterations of these combinations will not be described separately.

[0046] Figure 1 is a schematic diagrammatic representation of an ultrasound imaging system 100 according to aspects of the present disclosure. The ultrasound imaging system 100 can be used, for example, to acquire ultrasound video clips that can be used to train an ultrasound video feature classification system or that can be analyzed and highlighted by an ultrasound video feature classification system in a clinical setting (whether in real time, near real time, or as post-processing of stored video clips).

[0047] The ultrasound imaging system 100 is used to scan a region or volume of a subject's body. The subject may include a patient undergoing an ultrasound imaging procedure, or any other person, or any suitable living or non-living organism or structure. The ultrasound imaging system 100 includes an ultrasound imaging probe 110 that communicates with a host computer 130 via a communication interface or link 120. The probe 110 may include a transducer array 112, a beamformer 114, a processor circuit 116, and a communication interface 118. The host computer 130 may include a display 132, a processor circuit 134, a communication interface 136, and a memory 138 for storing subject information.

[0048] In some aspects, the probe 110 is an external ultrasound imaging device that includes a housing 111 configured for handheld operation by a user. The transducer array 112 can be configured to obtain ultrasound data when the user grasps the housing 111 of the probe 110 so that the transducer array 112 is positioned adjacent to or in contact with the skin of the subject. The probe 110 is configured to obtain ultrasound data of anatomical structures within the subject's body when the probe 110 is positioned external to the subject's body for general imaging, such as for abdominal imaging, liver imaging, etc. In some aspects, the probe 110 can be an external ultrasound probe, a transthoracic probe, and / or a curved array probe.

[0049] In other aspects, the probe 110 can be an internal ultrasound imaging device and can include a housing 111 configured to be positioned within a lumen of a subject's body for general imaging, such as for abdominal imaging, liver imaging, etc. In some aspects, the probe 10 can be a curved array probe. The probe 110 can be any suitable form for any suitable ultrasound imaging application, including both external and internal ultrasound imaging.

[0050] In some aspects, aspects of the present disclosure can be implemented using medical images of a subject obtained using any suitable medical imaging device and / or modality. Examples of medical images and medical imaging devices include X-ray images (angiographic images, fluoroscopic images, images with or without angiography) obtained by an X-ray imaging device, CT images obtained by a computed tomography (CT) imaging device, PET-CT images obtained by a positron emission tomography-computed tomography (PET-CT) imaging device, MRI images obtained by a magnetic resonance imaging (MRI) device, SPECT images obtained by a single photon emission computed tomography (SPECT) imaging device, OCT images obtained by an optical coherence tomography (OCT) imaging device, and IVPA images obtained by an intravascular photoacoustic (IVPA) imaging device. The medical imaging device can obtain medical images while being positioned outside the subject's body, spaced apart from the subject's body, adjacent to the subject's body, in contact with the subject's body, and / or inside the subject's body.

[0051] For ultrasound imaging, the transducer array 112 transmits ultrasound signals toward the anatomical object 105 of the subject and receives echo signals reflected from the object 105 back to the transducer array 112. The ultrasound transducer array 112 can include any suitable number of acoustic elements, including one or more acoustic elements and / or multiple acoustic elements. In some examples, the transducer array 112 includes a single acoustic element. In some examples, the transducer array 112 can include an array of acoustic elements having any number of acoustic elements in any suitable configuration. For example, the transducer array 112 can include 1 to 10,000 acoustic elements, including values such as 2 acoustic elements, 4 acoustic elements, 36 acoustic elements, 64 acoustic elements, 128 acoustic elements, 500 acoustic elements, 812 acoustic elements, 1,000 acoustic elements, 3,000 acoustic elements, 8,000 acoustic elements, and / or other values both larger and smaller. In some instances, the transducer array 112 can include an array of acoustic elements having any number of acoustic elements in any suitable configuration, such as a linear array, a planar array, a curved array, a curvilinear array, a circular array, an annular array, a phased array, a matrix array, a one-dimensional (1D) array, a 1.x-dimensional array (e.g., a 1.5D array), or a two-dimensional (2D) array. The array of acoustic elements can be controlled and activated in unison or independently (e.g., one or more rows, one or more columns, and / or one or more orientations). The transducer array 112 can be configured to obtain one-dimensional, two-dimensional, and / or three-dimensional images of the anatomical structure of the subject. In some aspects, the transducer array 112 can include piezoelectric micromachined ultrasonic transducers (PMUTs), capacitive micromachined ultrasonic transducers (CMUTs), single crystals, lead zirconate titanate (PZT), PZT composites, other suitable transducer types, and / or combinations thereof.

[0052] The object 105 may include any anatomical structure or anatomical feature, such as the kidneys, liver, and / or any other anatomical structure of the subject. The present disclosure may be implemented in the context of any number of anatomical locations and tissue types, including but not limited to organs, including the liver, kidneys, gallbladder, pancreas, lungs; ducts; intestines; nervous system structures, including the brain, thecal sac, spinal cord, and peripheral nerves; the urinary tract; and valves within blood vessels, blood, abdominal organs, and / or other systems of the body. In some aspects, the object 105 may include a malignancy, such as a tumor, cyst, lesion, hemorrhage, or blood pool within any part of the human anatomy. The anatomical structure may be a blood vessel, such as an artery or vein of the subject's vascular system, including the cardiac vasculature, peripheral vasculature, neural vasculature, renal vasculature, and / or any other suitable lumen within the body. In addition to natural structures, the present disclosure may be implemented in the context of artificial structures, such as, but not limited to, heart valves, stents, shunts, filters, implants, and other devices.

[0053] The beamformer 114 is coupled to the transducer array 112. For example, the beamformer 114 controls the transducer array 112 for transmission of ultrasound signals and reception of ultrasound echo signals. In some aspects, the beamformer 114 can apply time delays to signals sent to individual acoustic transducers within the array of transducers 112 so that the acoustic signals are steered in any suitable direction to propagate away from the probe 110. The beamformer 114 can also provide image signals to the processor circuit 116 based on responses to received ultrasound echo signals. The beamformer 114 can include multiple stages of beamforming. Beamforming can reduce the number of signal lines used to couple to the processor circuit 116. In some aspects, the transducer array 112 in combination with the beamformer 114 can be referred to as an ultrasound imaging component.

[0054] The processor 116 is coupled to the beamformer 114. The processor 116 may also be described as a processor circuit, which may include other components in communication with the processor 116, such as memory, the beamformer 114, a communication interface 118, and / or other suitable components. The processor 116 may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a controller, a field-programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof, configured to perform the operations described herein. The processor 116 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. The processor 116 is configured to process the beamformed image signals. For example, the processor 116 may perform filtering and / or quadrature demodulation to condition the image signals. The processors 116 and / or 134 may be configured to control the array 112 to obtain ultrasound data associated with the subject 105.

[0055] The communication interface 118 is coupled to the processor 116. The communication interface 118 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for sending and / or receiving communication signals. The communication interface 118 may include hardware components and / or software components that implement a specific communication protocol suitable for transmitting signals to the host 130 via the communication link 120. The communication interface 118 may be referred to as a communication device or a communication interface module.

[0056] The communication link 120 may be any suitable communication link. For example, the communication link 120 may be a wired link, such as a Universal Serial Bus (USB) link or an Ethernet link. Alternatively, the communication link 120 may be a wireless link, such as an Ultra Wideband (UWB) link, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 WiFi link, or a Bluetooth link.

[0057] At host 130, image signals may be received by communication interface 136. Communication interface 136 may be substantially similar to communication interface 118. Host 130 may be any suitable computing and display device, such as a workstation, personal computer (PC), laptop, tablet, or mobile phone.

[0058] The processor 134 is coupled to the communication interface 136. The processor 134 can also be described as a processor circuit, which may include other components that communicate with the processor 134, such as a memory 138, a communication interface 136, and / or other suitable components. The processor 134 can be implemented as a combination of software components and hardware components. The processor 134 may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a controller, an FPGA device, another hardware device, a firmware device, or any combination thereof, which is configured to perform the operations described herein. The processor 134 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. The processor 134 can be configured to generate image data based on the image signals received from the probe 110. The processor 134 can apply advanced signal processing and / or image processing techniques to the image signals. In some aspects, the processor 134 can form a three-dimensional (3D) volume image based on the image data. In some aspects, processor 134 can perform real-time processing on the image data to provide a streaming video of an ultrasound image of subject 105. In some aspects, host computer 130 includes a beamformer. For example, processor 134 can be part of such a beamformer and / or otherwise communicate with such a beamformer. The beamformer in host computer 130 can be a system beamformer or a main beamformer (providing one or more subsequent stages of beamforming), while beamformer 114 can be a probe beamformer or a microbeamformer (providing one or more initial stages of beamforming).

[0059] Memory 138 is coupled to processor 134. Memory 138 may be any suitable storage device, such as cache memory (e.g., cache memory of processor 134), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, a solid-state memory device, a hard disk drive, a solid-state drive, other forms of volatile and nonvolatile memory, or a combination of different types of memory.

[0060] The memory 138 can be configured to store: subject information, measurements, data, or files relating to the subject's medical history, a history of procedures performed, anatomical or biological characteristics, traits, or medical conditions associated with the subject; computer-readable instructions, such as code, software, or other applications; and any other suitable information or data. The memory 138 can be located within the host computer 130. The subject information can include measurements, data, files, other forms of medical history, such as, but not limited to, ultrasound images, ultrasound videos, and / or any imaging information related to the subject's anatomy. The subject information can include parameters related to the imaging procedure, such as the anatomical scan window, probe orientation, and / or subject position during the imaging procedure. The memory 138 can also be configured to store information related to the training and implementation of machine learning algorithms (e.g., neural networks) and / or information related to the implementation of image recognition algorithms, image quantification algorithms, and / or image acquisition guidance algorithms (including those described herein) for detecting / segmenting anatomical structures.

[0061] The display 132 is coupled to the processor circuit 134. The display 132 may be a monitor or any suitable display. The display 132 is configured to display ultrasound images, image videos, and / or any imaging information of the subject 105.

[0062] The ultrasound imaging system 100 can be used to assist sonographers in performing ultrasound scans. Scans can be performed in a point-of-care setting. In some instances, the host computer 130 is a console or a mobile cart. In some cases, the host computer 130 can be a mobile device, such as a tablet, mobile phone, or portable computer. During an imaging procedure, the ultrasound system can acquire ultrasound images of a specific region of interest within the subject's anatomy. The ultrasound imaging system 100 can then analyze the ultrasound images to identify various parameters associated with the acquisition of the images, such as the scan window, probe orientation, subject position, and / or other parameters. The ultrasound imaging system 100 can then store the images and these associated parameters in the memory 138. During a subsequent imaging procedure, the ultrasound imaging system 100 can retrieve previously acquired ultrasound images and associated parameters for display to the user, which can be used to guide the user of the ultrasound imaging system 100 in using the same or similar parameters during a subsequent imaging procedure, as described in more detail below.

[0063] In some aspects, the processor 134 can utilize a deep learning-based prediction network to identify parameters of the ultrasound image, including anatomical scan window, probe orientation, subject position, and / or other parameters. In some aspects, the processor 134 can receive metrics related to the imaged region of interest or the subject's physiological state or perform various calculations related to the imaged region of interest or the subject's physiological state during the imaging process. These metrics and / or calculations can also be displayed to the ultrasound physician or other user via the display 132.

[0064] Before continuing, it should be noted that the above examples are provided for illustration purposes and are not intended to be limiting. Other devices and / or device configurations may be utilized to perform the operations described herein.

[0065] Figure 2 is a schematic diagram of a processor circuit 250 according to aspects of the present disclosure. The processor circuit 250 can be implemented in the ultrasound imaging system 100 or other devices or workstations (e.g., third-party workstations, network routers, etc.) as needed, or on a cloud processor or other remote processing unit to implement the method. As shown, the processor circuit 250 may include a processor 260, a memory 264, and a communication module 268. These elements can communicate with each other directly or indirectly, for example, via one or more buses.

[0066] The processor 260 may include any combination of a central processing unit (CPU), a digital signal processor (DSP), an ASIC, a controller or general-purpose computing device, a reduced instruction set computing (RISC) device, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other related logic devices (including mechanical and quantum computers). The processor 260 may also include another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 260 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0067] Memory 264 may include cache memory (e.g., cache memory of processor 260), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory devices, hard drives, other forms of volatile and non-volatile memory, or a combination of different types of memory. In one aspect, memory 264 includes non-transitory computer-readable media. Memory 264 may store instructions 266. Instructions 266 may include instructions that, when executed by processor 260, cause processor 260 to perform the operations described herein. Instructions 266 may also be referred to as code. The terms "instructions" and "code" should be broadly interpreted to include any type of computer-readable statement(s). For example, the terms "instructions" and "code" may refer to one or more programs, routines, subroutines, functions, processes, etc. "Instructions" and "code" may include a single computer-readable statement or a plurality of computer-readable statements.

[0068] The communication module 268 may include any electronic circuitry and / or logic circuitry to facilitate direct or indirect data communication between the processor circuit 250 and other processors or devices. In this regard, the communication module 268 may be an input / output (I / O) device. In some cases, the communication module 268 facilitates direct or indirect communication between the processor circuit 250 and / or various components of the ultrasound imaging system 100. The communication module 268 may communicate within the processor circuit 250 via a number of methods or protocols. Serial communication protocols may include, but are not limited to, the United States Serial Protocol Interface (US SPI), an Inter-Integrated Circuit (IIC), and a Serial Bus. 2C), Recommended Standard 232 (RS-232), RS-485, Controller Area Network (CAN), Ethernet, Aeronautical Radio Incorporated 429 (ARINC429), MODBUS, Military Standard 1553 (MIL-STD-1553), or any other suitable method or protocol. Parallel protocols include, but are not limited to, Industry Standard Architecture (ISA), Advanced Technology Attachment (ATA), Small Computer System Interface (SCSI), Peripheral Component Interconnect (PCI), Institute of Electrical and Electronics Engineers 488 (IEEE-488), IEEE-1284, and other suitable protocols. Where appropriate, serial and parallel communications can be bridged by a Universal Asynchronous Receiver / Transmitter (UART), a Universal Synchronous Receiver / Transmitter (USART), or other suitable subsystem.

[0069] External communications (including but not limited to software updates, firmware updates, model sharing between the processor and a central server, or readings from the ultrasound imaging system 100) can be implemented using any suitable wireless or wired communication technology, such as a cable interface such as a Universal Serial Bus (USB), micro USB, Lightning or FireWire interface, Bluetooth, Wi-Fi, ZigBee, Li-Fi, or a cellular data connection such as 2G / GSM (Global System for Mobile), 3G / UMTS (Universal Mobile Telecommunications System), 4G, Long Term Evolution (LTE), WiMax or 5G. For example, a Bluetooth Low Energy (BLE) radio can be used to establish a connection with a cloud service for transmitting data and for receiving software patches. The controller can be configured to communicate with a remote server or a local device such as a laptop, tablet or handheld device, or can include a display capable of showing state variables and other information. Information can also be transmitted on a physical medium such as a USB flash drive or memory stick.

[0070] Figure 3 is a schematic diagrammatic representation of a radiology video, image loop, or video clip 310 (e.g., an ultrasound video clip) according to aspects of the present disclosure. The ultrasound image loop 310 includes a plurality of frames 320. In an example, the ultrasound image loop 310 is between 1 second and 60 seconds in length, at a frame rate of 30 frames per second, and thus may include between 30 frames and 1800 frames 320. Each frame serves as a Y-axis or height 330 and an X-axis or width 340, which are spatial dimensions representing a 2D cross-section of an object imaged by the ultrasound imaging system. Additionally, the ultrasound image loop 310 includes a depth or time axis 350, which represents the time at which each frame 320 of the image loop 310 is captured. Thus, the ultrasound image loop 310 can be considered a 3D data structure. The image loop 310 can be of any suitable modality having 2D image frames over time, such as X-ray, MRI, CT, etc.

[0071] In some aspects, the image loop 310 may include 4D data (X, Y, Z, time). For example, the 4D data may be 3D ultrasound (X, Y, Z are spatial dimensions) + time or another imaging modality that is 3D (X, Y, Z are spatial dimensions) + time, such as MRI, CT, etc. In other examples, the image loop 310 may include a 4D multi-modality / multi-imaging type image (X, Y are spatial dimensions in one imaging type of the modality + Z is an imaging type dimension in a modality whose imaging type is different from the X, Y dimensions + time). For example, the 4D multi-modality / multi-imaging type image may be 2D ultrasound (X, Y are spatial dimensions in B-mode ultrasound) + color Doppler ultrasound (Z) + time. Generally, the "Z" dimension may be any suitable imaging type (e.g., Doppler, elastography, etc.) different from the X, Y dimensions (e.g., B-mode).

[0072] Figure 4 4 is a schematic diagrammatic representation of a labeled ultrasound dataset 400 according to aspects of the present disclosure. The labeled ultrasound dataset 400 includes a plurality of image loops 405. Each image loop 405 includes a header 410 and a plurality of frames 420. Each frame 420 includes a frame number 430 and an annotation 440. The annotation 440 may, for example, indicate whether there is visible pathology in the frame 420. If pathology is present, the annotation 440 may also include one or more pathology locations 450, and the frame 420 may include one or more bounding boxes 460 indicating those locations on the image. Such frame-by-frame labeling is typically performed manually by highly skilled clinicians in order to generate training data for machine learning (ML) models.

[0073] Each image cycle of the labeled ultrasound dataset 400 may also include Figure 5 The video level metric 460 is generated (e.g., algorithmically) by the method described in Figure 6 In this regard, the video level metric 460 and the video level classification 470 are outputs of the systems, devices, and methods disclosed herein. Both the training data and the validation data for the ML network can be or include labeled ultrasound data.

[0074] Figure 5 is a schematic diagrammatic representation in the form of a flow chart of an exemplary ultrasound video feature classification method 500 according to aspects of the present disclosure. It should be understood that the steps of the method 500 may be performed in the same manner as Figure 5The steps of method 500 may be performed in a different order than shown, may include additional steps before, during, and after the steps, and / or may be replaced or eliminated in other embodiments. One or more of the steps of method 500 may be performed by one or more devices and / or systems described herein, such as components of the ultrasound system 100 and / or the processor circuit 250.

[0075] In step 510 , method 500 begins.

[0076] At step 520, method 500 includes acquiring an ultrasound image cycle (eg, Figure 3 The image loop 310 may be retrieved from a memory, received over a network, etc. In some cases, the image loop is received in real time or near real time during a medical procedure. Figure 1 In the case of executing the method on a host processor 134 , an image loop may be received from the probe 110 , although other arrangements may alternatively or additionally be used.

[0077] At step 530, method 500 includes determining, in each frame of the imaging loop, the location, size (e.g., bounding box), and confidence score of a feature of interest that can be found in the frame image. This step can be performed, for example, by software and / or hardware of a processor circuit for object detection. The object detector module can be, for example, a neural network (e.g., a convolutional neural network or CNN) trained for object detection.

[0078] In step 540 , method 500 includes computing a video level metric based on the location, size, bounding box, and / or confidence score of the frame level features identified in step 530 .

[0079] In the example, one or more metrics are calculated based on the detection. The metric can be hard-coded into the system, or can be selectable in real time or near real time to represent clinically relevant parameters derived from the detection. For example, in the screening / triage context, the operator may be interested in picking up features of any size, as long as they have been detected with sufficient confidence. In this setting, a metric defined as the maximum confidence of all detections of the feature type (but independent of the detection zone) may be appropriate. Alternatively, in the diagnostic context, the operator may already know that very small findings are clinically unimportant, but larger findings may indicate pathology. Therefore, the metric can be defined as the maximum area of all detection bounding boxes, or the maximum value of the product of the area and confidence of all detections. In this way, the metric will be insensitive to small findings (even findings with high confidence). In another context, the operator may be more interested in deep or shallow findings, and a metric can be used based on the mean, minimum or maximum value of the depth of all detections.

[0080] Several metrics can be considered, each appropriate for different clinical settings, including but not limited to:

[0081] The maximum confidence across all detections for a feature type.

[0082] The maximum area of all detections for the feature type.

[0083] The number of detections that exceed a minimum confidence and / or minimum area.

[0084] The maximum product of confidence and area for the feature type.

[0085] The average of the highest confidence scores in each frame for the feature type.

[0086] The average of the maximum areas in each frame for the feature type.

[0087] The average of the maximum product of confidence and area in each frame (or group of frames) for the feature type.

[0088] The average number of detections of a feature type that exceed a minimum confidence level per frame.

[0089] The number of frames or percentage of frames that contain detections of the feature type that exceed a minimum confidence level.

[0090] The product of the confidences of the highest confidence detections of multiple feature types.

[0091] The maximum product of the confidences of the highest confidence detections of multiple feature types in each frame or group of frames.

[0092] Combinations of the above metrics may be used.

[0093] Other metrics besides the non-limiting examples listed above may be used without departing from the spirit of the present disclosure.

[0094] In step 550, method 500 includes classifying the image loop based on the one or more video level metrics identified in step 540. This step can be performed, for example, by hardware and / or software of a processor circuit to perform the classification. The classifier can be, for example, a threshold, linear / logistic regression, a neural network, etc., as described below.

[0095] At step 560, method 500 includes displaying the classification results to the user, possibly along with the location of each frame identified in the image loop. A non-limiting example screen display can be found below. Figure 10-13 Found in.

[0096] In step 570 , method 500 is complete.

[0097] The flowcharts herein are provided for illustrative purposes; those skilled in the art will recognize numerous variations that still fall within the scope of the present disclosure. For example, the logic of the flowchart may be shown as sequential. However, similar logic may be parallel, massively parallel, object-oriented, real-time, event-driven, cellular automata, etc., while implementing the same or similar functionality. To perform the methods described herein, a processor may divide each of the steps described herein into multiple machine instructions and execute these instructions at a rate of hundreds, thousands, millions, or billions per second in a single processor or across multiple processors. Such rapid execution may be necessary to perform the methods as described herein in real time or near real time. For example, to identify features of an ultrasound image cycle, step 530 may need to be executed faster than the frame rate of the video (e.g., 30 Hz or 30 executions per second), and to classify the entire image cycle, steps 540 and 550 may need to be executed in the time gap between the acquisition of consecutive image cycles.

[0098] Figure 6 is a schematic diagrammatic representation in the form of a block diagram of an ultrasound video feature classification system according to aspects of the present disclosure. An image loop 310 comprising a plurality of frames 320 is received by an object detector 610, which performs feature detection on the image loop 310 and outputs an annotated image loop 620 comprising a plurality of frames 622. Each frame 622 may include a detection or bounding box 625, each of which is defined by parameters such as an x-axis position (e.g., in pixels or millimeters), a y-axis position (e.g., in pixels or millimeters), a width (e.g., in pixels or millimeters), a height (e.g., in pixels or millimeters), and a confidence level (e.g., a score or percentage). Thus, each possible detection within the image loop may be represented as {x, y, w, h; c} f,i , where f is the frame number and i is the detection number within the frame. The likelihood of detection can be considered a frame level metric.

[0099] Based on the frame level metrics, the system 600 calculates the video level metrics 630 as described above in Figure 5 The video level metrics are then received by a classifier 640, which classifies the detections for the entire image cycle (video level classification 650). For example, the classifier can determine: (a) whether a particular pathology is present, (b) the severity of the particular pathology, (c) the probability or confidence that the particular pathology is present, (d) the size of the particular pathology, (e) the number of detected sites of the particular pathology, or any combination thereof.

[0100] The most straightforward approach to video level classification is to threshold one or more metrics. One advantage of this approach is that it has good interpretability and explainability, which can allow users to easily understand and trust the video level classification results. Alternatively, better performance can be achieved by using a combination of metrics (e.g., utilizing linear or logistic regression and then thresholding the regression results), or by using more advanced machine learning algorithms that combine metrics. (See the "Algorithm Training" section below). Although the interpretability of the video level rules is reduced here, the overall approach is still interpretable because it relates the final video level results to individual frame-by-frame detections that can be shown to the user.

[0101] The block diagrams are provided herein for illustrative purposes; those skilled in the art will recognize numerous variations that still fall within the scope of the present disclosure. For example, the block diagrams may illustrate a specific arrangement of components, modules, services, steps, processes, or layers that result in a particular data flow. It should be understood that some embodiments of the systems disclosed herein may include additional components, some components shown may not be present in some aspects, and the arrangement of components may differ from that shown, resulting in different data flows while still performing the methods described herein.

[0102] Figure 7 is a schematic diagrammatic illustration in the form of a block diagram of the computation of a per-frame metric 625 according to aspects of the present disclosure. An image loop 310 comprising a plurality of frames 320 is fed into an object detector 610 .

[0103] The object detector 610 can implement or include any suitable type of learning network. For example, in some aspects, the object detector 610 can include a neural network, such as a convolutional neural network (CNN). In addition, the convolutional neural network can additionally or alternatively be an encoder-decoder type network, or can utilize a backbone architecture based on other types of neural networks (such as object detection networks, classification networks, etc.). An example backbone network is the Darknet YOLO backbone (e.g., Yolov3), which can be used for object detection. The CNN can, for example, include a set of N convolutional layers, where N can be any positive integer. When the CNN is the backbone, the fully connected layer can be omitted. The CNN can also include a maximum pooling layer and / or an activation layer. Each convolutional layer may include a set of filters configured to extract features from the input (e.g., from a frame of ultrasound video). The value N and the size of the filter can vary depending on the aspect. In some cases, the convolutional layer can utilize any nonlinear activation function, such as a leaky rectified nonlinear (ReLU) activation function and / or batch normalization. The max pooling layer gradually reduces the high-dimensional output to the dimension of the desired result (e.g., the bounding box of the detected feature). The output of the detection network may include many bounding boxes, most of which have very low confidence scores and are therefore filtered out or ignored. The fully connected layer may be referred to as a perception layer or perceptual layer. In some aspects, perception / perceptual and / or fully connected layers (e.g., multi-layer perception) may be found in the object detector 610.

[0104] These descriptions are included for exemplary purposes; one of ordinary skill in the art will understand that other types of learning models having similar or different features than those described above may be used instead or in addition without departing from the spirit of the present disclosure.

[0105] The output of the object detector 610 may include an annotated image loop 620 consisting of a plurality of annotated image frames 622 and per-frame metrics 625. Figure 7 In the example shown, the per-frame metrics include the number of bounding boxes identified in the frame, the corresponding area of the bounding boxes, and the corresponding confidence score for each box. These per-frame metrics can then be used to calculate video level metrics. For example, in Figure 7 In the example shown, the maximum number of detections in a frame is 3, the maximum area of a detection is 40,182 pixels, the maximum confidence of a detection is 74%, and the average of all detection confidences is 56%. One or more of these values can be used as a video-level metric for classifying the entire image loop 310.

[0106] The systems and methods disclosed herein are broadly applicable to different types of features and can, for example, draw boxes around suspicious B-lines or other features (including suspicious imaging artifacts). Object detectors can be single-class or multi-class, depending on how the model is constructed. If the B-line detector is trained separately, both models can be run separately (e.g., one model for each feature type). Otherwise, multiple feature classes can be identified and enclosed in detection boxes at the same time. The ML model for B-line detection can use exactly the same structure as the model for real change detection. A single detector can be trained / run that detects multiple feature types (a "multi-class detector") and provides their locations as well as a confidence score and feature type (class) for each detection as output. Alternatively, several "single-class" detectors can be run, each trained to detect a single feature type / class. These separate single-class detectors can have the same architecture (e.g., layers and connections), but will have been trained using different data (e.g., different images and / or annotations).

[0107] The system can illustrate the detection of different feature types in the image by adding boxes with a black outline color, for example. The system can then calculate two separate metrics based on each type of detection to arrive at a video-level classification for the feature type. Alternatively, the system can calculate metrics based on two / several types of features to arrive at a single video-level classification. For example, a detector can be trained to detect three features: "normal pleural line (PL)", "thickened PL", and "irregular PL". The system can then calculate a single metric for the video-level classification of the entire video as having a "normal pleural line" or an "abnormal pleural line".

[0108] Figure 8A is a schematic diagram overview in block diagram form of a training mode 800 of an object detector 610 according to aspects of the present disclosure. Figure 8A In the example shown, a set of training data 805a comprising loops of ultrasound images with manually labeled pathology locations (e.g., bounding boxes) is fed into an untrained object detector 810a in an iterative training process familiar to one of ordinary skill in the art.

[0109] In particular, for object detection using convolutional neural networks, experts manually annotate a large number of sample images to depict the localization of features of interest. The parameters of the network model (e.g., the weights at each artificial neuron) are initialized with initial values A, which can be random values or the results from training on a previous dataset. In an iterative process, the network is used to perform detection inference on the training images, the results are compared with the ground-truth annotations, and an optimizer is used to adjust the network parameters B until the measure of detection accuracy is maximized.

[0110] Thus, the output of the training process 800 is a trained object detector 810b, where parameters B (eg, weights) are optimized for detecting features in the training video 805a.

[0111] Figure 8B is a schematic diagrammatic overview in the form of a block diagram of a validation mode 802 of an object detector 610 according to aspects of the present disclosure. In the validation mode, a set of manually annotated validation videos (e.g., videos that include bounding boxes around any pathology identified by an expert in each frame of each video) are fed into a trained object detector 810b to determine whether the trained object detector 810b detects the human-identified features in the validation video 805b with a desired level of accuracy.

[0112] In some cases, the performance of trained object detector 810b may be considered to be below the desired level of accuracy. In this case, the parameters B (e.g., weights) of trained object detector 810b may be adjusted until the detection accuracy on the validation dataset (or validation dataset plus training dataset) reaches the desired accuracy. In such cases, the output of the validation process may be trained object detector 610, which may be identical to trained object detector 810b except for the adjusted parameters C (e.g., weights). In other cases, the performance of trained object detector 810b may be considered sufficient, so the parameters are not adjusted, and trained object detector 610 may be identical to trained object detector 810b (e.g., using the same weights).

[0113] Figure 8C is a schematic, pictorial overview in the form of a block diagram of an inference mode or clinical use mode 804 of an object detector 610 according to aspects of the present disclosure. In clinical use, an ultrasound video or image loop 310 is fed to a trained and validated object detector 610 for analysis. In some cases, the image loop 310 may be acquired and analyzed in real time or near real time. In other cases, the image loop may be retrieved from a memory, storage device, or network. The trained and validated object detector 610 then produces as output an annotated version 620 of the patient video 310 that includes frame-level object detections (e.g., bounding boxes superimposed on frames of the image loop at the location of suspected pathology). The annotated video 620 with frame-level object detections may then be used as input to a video-level metric calculator or metric calculation step 520, as described above in Figure 5 As described in .

[0114] Thus, in each frame of the imaging loop, the object detector 610 is run to determine the location and confidence value of the occurrence of multiple clinically relevant features. The location can be determined in the form of a bounding box that tightly surrounds the feature. Other forms of location are possible, such as a binary mask indicating image pixels that are part of the feature, or a polygon or other shape that surrounds the feature. For any such location, the center and area of the detection can be determined. The confidence value can be determined as a normalized value in the range [0..1], where 0 indicates the lowest confidence and 1 indicates the highest confidence that the feature is present at that location.

[0115] The detection algorithm may be based on conventional image processing (including thresholding, filtering and texture analysis), or may be based on machine learning (in particular using deep neural networks). A particularly advantageous implementation of the detection is the use of a Yolo-type network such as Yolo3. An exemplary output of the detection step is, for each frame of the imaging loop, a list of detections for one or several types of features of interest. Each element in the detection list may, for example, comprise at least the confidence and area of the detection (and typically also position, width and height). Thus, for each frame i and feature f, there is a detection {x, y, w, h; c} f,i Where x, y represent the center coordinates, w, h represent the width and height, and c represents the confidence value of the detection. Other means of representing detection may be used alternatively or additionally without departing from the spirit of the present disclosure.

[0116] Figure 9A is a schematic diagrammatic overview in block diagram form of a training mode 900 of a classifier 640 according to aspects of the present disclosure. Figure 9A In the example shown, video level metrics are included (e.g., Figure 5 A set of training data 905a (the metric calculated in the video level metric calculation step 540) is fed into an untrained classifier 910a in an iterative training process that will be familiar to those skilled in the art.

[0117] In the example, the input to the classifier is a video-level metric calculated from the output of the detector (rather than frame-level human annotations), and the output of the classifier is a video-level classification or labeling. In other words, the classifier only uses the previously calculated video-level metric as input.

[0118] A simple approach to classification using a single metric is to choose a threshold that best separates two classes (such as the presence and absence of a clinical feature of interest in an imaging cycle). For better performance, a combination of metrics can be employed using, for example, simple linear or logistic regression to determine a continuous variable, which in turn can be thresholded for optimal separation of two or several classes. In addition, machine learning algorithms can be used to combine metrics. Examples include support vector machines, decision trees, random forests, boosted trees, etc., which will also be trained using training data 905a. Thus, the untrained classifier 920a can include any combination of machine learning (ML) networks, regression, and thresholds. In any of these aspects, the parameters required for classification (e.g., weights and thresholds) can start with random or inherited values A, for which new values B can be determined as part of the imaging cycle classification training phase. Thus, the trained classifier 920b is the output of the training process.

[0119] The fully connected layers within the ML network may be referred to as perception layers or perceptual layers. In some aspects, perception / perceptual and / or fully connected layers (e.g., multi-layer perception) may be found in the classifier 640 to allow for classification, regression, thresholding, segmentation, etc. The classifier 640 may also include a max pooling layer and / or an activation layer. The max pooling layer gradually reduces the high-dimensional output to the dimension of the desired result (e.g., classification output or regression output).

[0120] Figure 9B is a schematic diagrammatic overview in block diagram form of a verification mode 902 of the classifier 640 according to aspects of the present disclosure. Figure 9B In the example shown, video level metrics are included (e.g., Figure 5 A set of validation data 905b of the video level metric calculation step 540 (the metrics calculated) is fed into the trained classifier 920b. Similarly, in this example, frame level labels are not used in classifier training.

[0121] Ideally, the validation data set 905b is completely independent of the training data set 905a. For example, two data sets can be derived from different patients using different equipment or equipment settings. Parameters B (e.g., weights and thresholds) required for classification are used during validation. In some cases, in an iterative process, the network is used to perform detection reasoning on the validation images, frame-level classification, and video-level metrics, the results are compared with the true situation annotations, and the optimizer is used to adjust the parameters (e.g., weights and / or thresholds) until the metric of classification accuracy is maximized in the separate annotated validation video, thereby generating updated parameters C and trained classifier 640. In other cases, the classification accuracy of the trained classifier 920b with parameter B is considered to be sufficient for the validation data set 905b, and therefore parameter B is not adjusted. Therefore, parameter C can be the same as parameter B, and the fully trained and validated classifier 640 can be the same as the trained classifier 920b.

[0122] Figure 9C is a schematic diagrammatic overview in block diagram form of an inference mode or clinical use mode 904 of the ultrasound video classifier 640 according to aspects of the present disclosure. Figure 9C In the example shown, a patient video image loop (eg, real-time ultrasound video) 310 is provided along with a metric calculator 510 (see Figure 8C ) are fed together with the video level metrics calculated to a trained and validated classifier 640, which produces as output an annotated version 930 of the patient video including the video level classification.

[0123] In some aspects, the classification weights and / or thresholds can be hard-coded into the trained and validated classifier 640. In other aspects, the classification thresholds for the metrics can be provided to the user in the form of user-adjustable settings (e.g., as a slider bar that is part of an application interface shown on a display). An advantage of using adjustable settings over fixed settings is that the user can have control over the tradeoff between algorithm sensitivity and specificity. For example, a rule such as "number of frames containing detected features whose size (area) exceeds Y centimeters" can include up to two adjustable settings: a first setting regarding the number of frames, and a second setting regarding the size of the detections.

[0124] The results of the image loop classification can then be displayed to the user along with the metric used and / or the detections used to calculate the metric. In particular, the detections that have the greatest (or exclusive) influence on the metric can be highlighted to "explain" the entire image loop classification result ("Explainable AI"). For example, if "maximum confidence" or "maximum area" is used as the metric and a simple threshold for the classification is employed, detections whose confidence values or areas exceed the threshold can be displayed or highlighted.

[0125] It should be noted that in some aspects (particularly where the training dataset is large and / or diverse), Figure 8B and 9B The verification steps shown in can be considered unnecessary.

[0126] Figure 10 An example screen display 1000 of an ultrasound video feature classification system according to aspects of the present disclosure is shown. The screen display includes both an annotated image frame 1010 and a classification output 1020. The classification output 1020 may, for example, be a binary classification that reports whether a given anatomical structure or pathology (in this example, consolidation) is believed to be present in the image frame. The annotations to the image frame 1010 may, for example, include one or more metrics 1030 and one or more bounding boxes 1040. Figure 10 In the example shown, the screen display 1000 also includes a frame counter 1050, a pause control 1060, a play control 1070, and a pair of step buttons 1080L and 1080R. Other types of video controls may be used instead or in addition, and in some aspects, there may be no video controls at all, as the screen display may include only the image frames 1010 deemed most important in determining the classification 1020.

[0127] In the example, step buttons 1080L and 1080R can be used to scroll through the frames that contribute most to the classification. For example, if frames 2, 5, and 13 have the greatest contribution to the video level metric (in Figure 10 , confidence level), which in turn leads to object detection, which in turn leads to classification. If the user pushes the right arrow, the screen display is updated to show frame 13 (along with the size of the corresponding box and the confidence level). Similarly, if the user pushes the left arrow, the screen display is updated to show frame 2 (along with the size of the corresponding bounding box and the confidence level).

[0128] Figure 11 An example screen display 1100 of an ultrasound video feature classification system according to aspects of the present disclosure is shown. The screen display includes both an annotated image frame 1010 and a classification output 1120. The classification output 1120 may, for example, be a discrete classification that reports one of a plurality of specified severity levels for a given anatomical structure or pathology (in this example, a consolidation) believed to be present in the image frame. Metrics 1030, a bounding box 1040, a frame counter 1050, a pause control 1060, a play control 1070, and step buttons 1080L and 1080R are also visible.

[0129] Figure 12An example screen display 1200 of an ultrasound video feature classification system according to aspects of the present disclosure is shown. The screen display includes both annotated image frame 1010 and classification output 1220. Classification output 1220 may, for example, be a numerical classification reporting the severity of a given anatomical structure or pathology (in this example, consolidation) believed to be present in the image frame. The severity may, for example, be a fractional value between 0 and 1 (e.g., where 1 is the most severe), a value between 1 and 10, a percentage value (where 100% is the most severe), etc. Metrics 1030, a bounding box 1040, a frame counter 1050, a pause control 1060, a play control 1070, and step buttons 1080L and 1080R are also visible.

[0130] Figure 13 An example screen display 1300 of an ultrasound video feature classification system according to aspects of the present disclosure is shown. The screen display includes two annotated image frames 1010a and 1010b, each showing an image frame associated with a different pathology. Figure 13 In the example shown, frame 1010a and its associated metrics 1330a show a consolidation, while frame 1010b and its associated metrics show a B-line abnormality. However, other feature types or other numbers of feature types may be detected instead or in addition. In an example, an object detector and classifier may be loaded with a first set of parameters for detecting and classifying a particular pathology type and may be loaded with a second set of parameters for detecting and classifying a second pathology type, and then a diagnostic model or diagnostic step (e.g., a second classifier) may receive as input the video level metrics and / or classifications for both feature types in order to generate a diagnosis 1320. Figure 13 In the example shown, the presence and severity of consolidation and B-lines has led to a diagnosis of severe pneumonia, although other disease conditions may alternatively or additionally be diagnosed.

[0131] Figure 14 is a schematic diagrammatic representation in the form of a flow chart of an example diagnostic method 1400 according to aspects of the present disclosure. It should be understood that the steps of the method 1400 may be performed in the same manner as described above. Figure 14 The steps of method 1400 may be performed in a different order than shown, may include additional steps before, during, and after the steps, and / or may be replaced or eliminated in other embodiments. One or more of the steps of method 1400 may be performed by one or more devices and / or systems described herein, such as components of the ultrasound system 100 and / or the processor circuit 250.

[0132] At step 1410 , method 1400 includes receiving an image loop as described above.

[0133] At step 1420 , method 1400 includes detecting a first type of feature (eg, pathology) within each frame of the imaging loop using a first set of detection parameters, and generating a first set of frame-level metrics based on the detection.

[0134] At step 1430 , method 1400 includes computing a first set of video level metrics and determining a first video level classification for a first feature type using the techniques and apparatus described above.

[0135] At step 1430 , method 1400 includes detecting a second type of feature (eg, pathology) within each frame of the imaging loop using a second set of detection parameters, and generating a second set of frame-level metrics based on the detection.

[0136] At step 1450 , method 1400 includes computing a second set of video level metrics and determining a second video level classification for the second feature type using the second set of classification parameters using the techniques and apparatus described above.

[0137] At step 1460, method 1400 includes classifying the disease state depicted in the image loop based on the first video-level classification and / or the first set of frame-level metrics and the second video-level classification and / or the second set of frame-level metrics. This can be performed, for example, by the classifier using a third set of classification parameters, or it can be performed by the second classifier using a third set of classification parameters. In an example, the third set of classification parameters is derived by training the classifier or the second classifier using the video-level classification and / or the frame-level metrics of videos identified by an expert as exhibiting a particular disease state.

[0138] In step 1470, method 1400 includes reporting the classified disease state to the user as a diagnosis (e.g., Figure 13 ). The method is now complete.

[0139] As will be readily understood by those skilled in the art after becoming familiar with the teachings herein, the ultrasound video feature classification system advantageously allows accurate classification and diagnosis to be performed at the level of the entire video (e.g., ultrasound image loop) rather than at the level of individual image frames. This can achieve higher accuracy and greater confidence in the results for clinicians without significantly increasing the time required for classification and diagnosis.

[0140] The systems, methods, and devices described herein can be applicable to point-of-care and handheld ultrasound use cases, such as utilizing Philips Lumify. The ultrasound video feature classification system can be used for any automated ultrasound image loop classification, particularly in point-of-care settings and lung ultrasound, but also in diagnostic ultrasound and echocardiography. The ultrasound video feature classification system can be deployed on handheld mobile ultrasound devices, as well as portable or car-based ultrasound systems. The ultrasound video feature classification system can be used in a variety of settings, including emergency departments, intensive care units, and general inpatient settings. Applications can also be extended to out-of-hospital settings.

[0141] The display of individual detection results (e.g., bounding boxes on individual image loop frames) and the highlighting of those detections that contribute most to the metric(s) used for image loop classification are easily detectable. For example, if the final image loop level classification is based on the maximum confidence, maximum area, maximum product of confidence and area, etc. of all detections in the image loop, the individual detection that produced this can be highlighted directly in the frame. If the rules used to classify image loops are simple (e.g., based on one or two simple and understandable metrics), they can be clearly described in product user manuals and other product documentation, which will make them easy to detect. For example, a rule such as "if the largest detected feature exceeds 1 cm in area, then the image loop is classified as positive" is explainable and can be made transparent to the user. If the threshold(s) applied to the metric(s) are user-adjustable rather than fixed, this adds another layer of detectability because both the metric(s) and the threshold(s) are known to the user.

[0142] In the above examples and aspects, multiple variations are possible. For example, the systems, methods, and apparatus described herein are not limited to lung ultrasound applications. Instead, the same techniques can be applied to images of other organs or anatomical systems (such as the heart, brain, digestive system, vascular system, etc.). In addition, the techniques disclosed herein are also applicable to other medical imaging modalities where 3D data is available, such as other ultrasound applications, camera-based video, X-ray video, and 3D volumetric images, such as computer-assisted tomography (CT) scans, magnetic resonance imaging (MRI) scans, optical coherence tomography (OCT) scans, or intravenous ultrasound (IVUS) pullback sequences. The techniques described herein can be used in a variety of settings, including emergency room, intensive care, inpatient, and out-of-hospital settings.

[0143] Thus, the logical operations making up the various aspects of the technology described herein are referred to variously as operations, steps, objects, layers, elements, components, algorithms, or modules. Furthermore, it should be understood that these may occur or be executed or arranged in any order unless expressly required otherwise or a specific order is inherently required by the claim language.

[0144] All directional references (e.g., up, down, inside, outside, upward, downward, left, right, lateral, front, back, top, bottom, above, below, vertical, horizontal, clockwise, counterclockwise, proximal, and distal) are used only for identification purposes to help the reader understand the claimed subject matter and do not create limitations, particularly with respect to the position, orientation, or use of the ultrasound video feature classification system. Unless otherwise specified, connection references (e.g., attached, coupled, connected, combined, or "communicating") should be interpreted broadly and may include intermediate members between a collection of elements and relative movement between elements. Therefore, connection references do not necessarily imply that two elements are directly connected and in a fixed relationship to each other. The term "or" should be interpreted to mean "and / or" rather than "exclusive or". The word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. Unless otherwise specified in the claims, the values should be interpreted as merely illustrative and should not be considered restrictive.

[0145] The above specification, examples, and data provide a complete description of the structure and use of exemplary aspects of the ultrasound video feature classification system as defined in the claims. Although various aspects of the claimed subject matter have been described above with a certain degree of particularity or with reference to one or more individual aspects, those skilled in the art could make numerous changes to the disclosed aspects without departing from the spirit or scope of the claimed subject matter.

[0146] Other aspects are also contemplated. It is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative of particular aspects only and not limiting. Changes may be made in detail or structure without departing from the essential elements of the subject matter defined in the appended claims.

Claims

1. A system comprising: monitor; as well as a processor configured to communicate with the display, wherein the processor is configured to: receiving an ultrasound video of an anatomical structure acquired by an ultrasound probe, wherein the ultrasound video includes a plurality of frames; generating at least one frame level metric based on the ultrasound video for each frame of the plurality of frames detected as including pathology; generating at least one video level metric related to the pathology based on the at least one frame level metric for the frame of the plurality of frames detected as including the pathology; generating a video level classification of the pathology based on the at least one video level metric related to the pathology; and An on-screen display including the video level classification is provided to the display.

2. The system according to claim 1, wherein: The at least one frame level metric is computed by an object detection machine learning network.

3. The system according to claim 1, wherein: The screen display also includes the ultrasound video or a frame of the ultrasound video.

4. The system according to claim 1, wherein: The at least one frame level metric comprises: the number of detections of said pathology within said frame; or The area or confidence level of the bounding box, binary mask, or polygon representing the pathology.

5. The system according to claim 1, wherein: The at least one video level metric is selected from a list comprising: a maximum confidence of all detections of the pathology, a maximum area of all detections of the pathology, a number of detections of the pathology exceeding a minimum confidence level, a number of detections of the pathology exceeding a minimum area, a maximum product of the confidence and area of the pathology, an average of the highest confidence in each frame of the pathology, an average of the maximum area in each frame of the pathology, an average of the maximum product of the confidence and area in each frame of the pathology, an average number of detections of the pathology exceeding a minimum confidence in each frame, a number of frames or a percentage of frames containing a detection of the pathology exceeding a minimum confidence, a product of the confidences of the highest confidence detections of the pathology, a maximum product of the confidences of the highest confidence detections of the pathology, or any combination of the above.

6. The system according to claim 4, wherein: The pathology includes at least one of the following: B-lines, fused B-lines, pleural line changes, consolidation, or pleural effusion.

7. The system according to claim 1, wherein: Generating the video level classification of the pathology involves at least one of: a threshold, a regression, or a classification machine learning network.

8. The system according to claim 1, wherein: The video level classification includes at least one of the following: binary classification, discrete classification, or numerical classification.

9. The system according to claim 1, wherein: The processor is further configured to: generating at least one second frame level metric based on the ultrasound video for each frame of the plurality of frames detected as including a second pathology; generating at least one second video level metric related to the second pathology based on the at least one second frame level metric for the frames of the plurality of frames detected as including the second pathology; generating a second video level classification of the pathology based on the at least one second video level metric related to the second pathology; and An on-screen display including the second video level classification is provided to the display.

10. The system according to claim 9, wherein: The processor is further configured to: generating a classification of a disease state associated with the pathology and the second pathology based on the video level classification of the pathology and the second video level classification of the second pathology; and A screen display including the classification of the disease state is provided to the display.

11. A method comprising: The processor configured to communicate with the display performs the following operations: receiving an ultrasound video of an anatomical structure acquired by an ultrasound probe, wherein the ultrasound video includes a plurality of frames; generating at least one frame level metric based on the ultrasound video for each frame of the plurality of frames detected as including pathology; generating at least one video level metric related to the pathology based on the at least one frame level metric for the frame of the plurality of frames detected as including the pathology; generating a video level classification of the pathology based on the at least one video level metric related to the pathology; and An on-screen display including the video level classification is provided to the display.

12. The method according to claim 11, wherein Computing the at least one frame level metric involves an object detection machine learning network.

13. The method according to claim 11, wherein The screen display also includes the ultrasound video or a frame of the ultrasound video.

14. The method according to claim 11, wherein The at least one frame level metric comprises: the number of detections of said pathology within said frame; or The area or confidence level of the bounding box, binary mask, or polygon representing the pathology.

15. The method according to claim 11, wherein The at least one video level metric is selected from a list comprising: a maximum confidence of all detections of the pathology, a maximum area of all detections of the pathology, a number of detections of the pathology exceeding a minimum confidence level, a number of detections of the pathology exceeding a minimum area, a maximum product of the confidence and area of the pathology, an average of the highest confidence in each frame of the pathology, an average of the maximum area in each frame of the pathology, an average of the maximum product of the confidence and area in each frame of the pathology, an average number of detections of the pathology exceeding a minimum confidence in each frame, a number of frames or a percentage of frames containing a detection of the pathology exceeding a minimum confidence, a product of the confidences of the highest confidence detections of the pathology, a maximum product of the confidences of the highest confidence detections of the pathology, or any combination of the above.

16. The method according to claim 11, wherein The pathology includes at least one of the following: B-lines, fused B-lines, pleural line changes, consolidation, or pleural effusion.

17. The method according to claim 11, wherein Generating the video level classification of the pathology involves at least one of: a threshold, a regression, or a classification machine learning network.

18. The method according to claim 11, wherein The video level classification includes at least one of the following: binary classification, discrete classification, or numerical classification.

19. The method according to claim 11, further comprising: generating at least one second frame level metric based on the ultrasound video for each frame of the plurality of frames detected as including a second pathology; generating at least one second video level metric related to the second pathology based on the at least one second frame level metric for the frames of the plurality of frames detected as including the second pathology; generating a second video level classification of the pathology based on the at least one second video level metric related to the second pathology; and An on-screen display including the second video level classification is provided to the display.

20. The method according to claim 19, further comprising: generating a classification of a disease state associated with the pathology and the second pathology based on the video level classification of the pathology and the second video level classification of the second pathology; and A screen display including the classification of the disease state is provided to the display.

Images