Method and system for analyzing lung ultrasound

Abstract

A method (100) for analyzing ultrasound image data, comprising: (i) receiving (120) a time series of ultrasound image data for one or more of a plurality of different regions of one or two lungs of a patient; (ii) analyzing (130) the received ultrasound image data using a first trained clinical lung feature recognition algorithm to identify a first clinical feature in the patient's lungs, where identifying the first clinical feature comprises analyzing a plurality of frames in the time series, and where identifying the first clinical feature comprises analyzing the plurality of frames in the time series; identifying the first clinical feature includes identifying a location of the first clinical feature within the plurality of frames; (iii) analyzing (140) the identified first clinical feature using a trained clinical lung feature severity algorithm to characterize a severity of the identified first clinical feature; and (iv) providing (180) the identified first clinical feature and the severity of the characterized first clinical feature via a user interface.

Description

Technical Field

[0001] The present disclosure generally relates to methods and systems for analyzing lung ultrasound imaging to provide information about lung-related clinical characteristics. Background Art

[0002] Lung ultrasound imaging is an important tool for disease screening, monitoring, diagnostic support, and management. Important clinical features—such as B-lines, confluent B-lines, pleural line changes, consolidation, and pleural effusions—can be identified using lung ultrasound. The presence of these features is a predictor for a range of pulmonary and infectious diseases, including COVID-19 pneumonia. However, effective identification of clinical features using lung ultrasound may depend on operator experience, image quality, and choice of imaging settings, among other variables. Therefore, identifying clinical features is a challenging skill to learn, and success generally requires extensive specialized training and experience.

[0003] Automated quantification tools offer the potential to simplify and standardize image interpretation tasks, including ultrasound analysis. Studies have shown correlations between automated lung ultrasound features and expert scores, as well as prior correlations with gold standard measurements such as blood tests or chest CT. Automated analysis can even be used to diagnose conditions such as COVID-19 pneumonia. Automated tools that leverage traditional image processing techniques are well suited to extract interpretable image parameters that support human clinical interpretation. Image processing methods also benefit from potential advantages in simplicity, speed, and versatility. However, a significant drawback is that the performance of these techniques depends heavily on the discernment of manual parameters.

[0004] As an alternative to using handcrafted parameters derived from traditional image processing algorithms, techniques based on machine learning and artificial intelligence have become increasingly popular in the field of medical imaging, including for lung ultrasound applications.

[0005] While automated algorithms for detecting and classifying lung ultrasound features are beneficial, trained human operators are still required to review and report on the features. However, this human review can be time consuming. For example, many clinical lung ultrasound features (e.g., B-lines, lung sliding, dynamic bronchial inflation signs) require dynamic views to assess the type or quality of the feature. Entire video playback of the entire video loop is inefficient for reviewing or reporting individual features that are only present in a portion of the video, because the process requires repeated playback of the entire duration of the video (only a portion of which may be useful for assessing any particular feature), or repeated starting / stopping and replaying of manually selected portions of the video. In addition, most lung ultrasound examinations include multiple videos, each of which may require analysis. Therefore, the reviewer must manage information from multiple videos that are part of the examination in order to determine the overall status of the patient.

[0006] US2020 / 043602A1 describes a clinical condition detection system comprising a communication device that communicates with an ultrasound imaging device and is configured to receive a sequence of ultrasound image frames.

[0007] US2020 / 054306A1 describes an intelligent system comprising an electronic circuit configured to execute a neural network and to detect at least one feature in an image of a body part while executing the neural network. Summary of the invention

[0008] Therefore, there is a need for an automated lung ultrasound analysis tool that can analyze one or more lung ultrasound videos (also called cine loops) to identify clinical features related to the lungs.

[0009] The present disclosure relates to inventive methods and systems for analyzing ultrasonic lung imaging. Various embodiments and implementations herein relate to an ultrasonic analysis system, which optionally includes an ultrasound device configured to obtain an ultrasonic image of a patient's lungs. The system receives a time series of ultrasound image data of one or more of a plurality of different regions of one or both lungs of the patient. A first trained clinical lung feature recognition algorithm analyzes the received time series of ultrasound image data to identify a first clinical feature in the patient's lungs, wherein identifying the first clinical feature includes analyzing a plurality of frames in the time series, and wherein identifying the first clinical feature includes identifying a location of the first clinical feature within the plurality of frames. A trained clinical lung feature severity algorithm of the system analyzes the identified first clinical feature to characterize the severity of the identified first clinical feature. Optionally, a trained clinical feature prioritization algorithm analyzes one or more identified clinical features to prioritize reporting of the features. A user interface of the system provides the identified first clinical feature and the severity of the characterized first clinical feature, and optionally provides prioritization of one or more identified clinical features.

[0010] In general, in one aspect, a method for analyzing ultrasound image data is provided. The method includes: (i) receiving a time series of ultrasound image data of one or more regions of a plurality of different regions of one or both lungs of a patient; (ii) analyzing the received time series of ultrasound image data using a first trained clinical lung feature recognition algorithm to identify a first clinical feature in the lungs of the patient, wherein identifying the first clinical feature includes analyzing a plurality of frames in the time series, and wherein identifying the first clinical feature includes identifying a location of the first clinical feature within the plurality of frames; (iii) analyzing the identified first clinical feature using a trained clinical lung feature severity algorithm to characterize a severity of the identified first clinical feature; and (iv) providing the identified first clinical feature and the characterized severity of the first clinical feature via a user interface.

[0011] According to an embodiment, the method also includes analyzing the time series of the received ultrasound image data using the trained clinical lung feature recognition algorithm to identify a second clinical feature in the patient's lungs, wherein the second clinical feature is different from the first clinical feature; and analyzing the identified second clinical feature using the trained clinical lung feature severity algorithm to characterize the severity of the identified second clinical feature; wherein the providing step also includes providing the identified second clinical feature and the characterized severity of the second clinical feature via the user interface.

[0012] According to an embodiment, the method also includes analyzing the time series of the received ultrasound image data using a second trained clinical lung feature recognition algorithm to identify a second clinical feature in the patient's lungs, wherein the second clinical feature is different from the first clinical feature; and analyzing the identified second clinical feature using the trained clinical lung feature severity algorithm to characterize the severity of the identified second clinical feature; wherein the providing step also includes providing the identified second clinical feature and the characterized severity of the second clinical feature via the user interface.

[0013] According to an embodiment, the method also includes prioritizing the identified first clinical feature or the identified second clinical feature using a trained clinical feature prioritization algorithm, wherein the prioritization is based on one or more of: the type of identified clinical feature, the severity of the first clinical feature and the second clinical feature characterized, the timing of the first clinical feature and / or the second clinical feature in the time series of ultrasound image data, and / or the suspected or diagnosed clinical condition of the patient; wherein the providing step also includes providing the prioritization.

[0014] According to an embodiment, identifying the location of the first clinical feature within the plurality of frames comprises identifying a spatiotemporal location across the plurality of frames.

[0015] According to an embodiment, providing the identified first clinical feature and the characterized severity of the first clinical feature comprises providing a subset of a time series of the received ultrasound image data, the subset comprising the identified location of the identified first clinical feature. According to an embodiment, the subset is a time series. According to an embodiment, the subset is a static image.

[0016] According to an embodiment, the method further comprises receiving feedback from a user via the user interface regarding the provided identified first clinical feature and / or the severity of the characterized first clinical feature. According to an embodiment, the feedback comprises an adjustment of the severity of the characterized first clinical feature, a selection of one or more frames in the time series of ultrasound image data, an acceptance or rejection of the feature and / or a change in the type of feature.

[0017] According to an embodiment, the method further comprises generating a report comprising the identified first clinical feature and / or the characterized severity of the first clinical feature based on the received feedback.

[0018] According to another aspect, an ultrasound analysis system is configured to analyze ultrasound image data. The system includes: a time series of ultrasound image data of one or more regions of a plurality of different regions of one or both lungs of a patient; a trained clinical lung feature recognition algorithm configured to analyze the received time series of ultrasound image data to identify a first clinical feature in the lungs of the patient, wherein identifying the first clinical feature includes analyzing a plurality of frames in the time series, and wherein identifying the first clinical feature includes identifying a location of the first clinical feature within the plurality of frames; a trained clinical lung feature severity algorithm configured to analyze the identified first clinical feature to characterize a severity of the identified first clinical feature; a processor configured to: (i) analyze the received time series of ultrasound image data using the trained clinical lung feature recognition algorithm to identify the first clinical feature in the lungs of the patient; (ii) analyze the identified first clinical feature using the trained clinical lung feature severity algorithm to characterize a severity of the identified first clinical feature; and a user interface configured to provide the identified first clinical feature and the characterized severity of the first clinical feature.

[0019] According to an embodiment, the system also includes a trained clinical feature prioritization algorithm, which is configured to prioritize one or more identified clinical features; the processor is also configured to prioritize one or more identified clinical features using the trained clinical feature prioritization algorithm, wherein the prioritization is based on one or more of: the type of identified clinical feature, the severity of the first and second clinical features characterized, the timing of the first and / or second clinical features in the time series of ultrasound image data, and / or the suspected or diagnosed clinical condition of the patient; and the user interface is also configured to provide the prioritization.

[0020] According to an embodiment, the processor is further configured to receive feedback from a user via a user interface regarding the provided identified first clinical feature and / or the severity of the characterized first clinical feature. According to an embodiment, the processor is further configured to generate a report including the identified first clinical feature and / or the severity of the characterized first clinical feature based on the received feedback.

[0021] According to an embodiment, the user interface further comprises a summary display of the time series of ultrasound image data and the identified first clinical feature, wherein a user is able to select a region of the time series and / or the identified first clinical feature for review. According to an embodiment, after review by the user, the summary display of the time series of ultrasound image data and / or the identified first clinical feature is updated by the processor to show a status of the review.

[0022] It should be understood that all combinations of the aforementioned concepts and the additional concepts discussed in more detail below (as long as these concepts are not mutually inconsistent) are considered to be part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the inventive subject matter disclosed herein.

[0023] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In the drawings, like reference numbers generally refer to the same parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention.

[0025] Figure 1 is a flow chart of a method for analyzing ultrasound image data using an ultrasound analysis system according to an embodiment.

[0026] Figure 2 is a schematic diagram of an ultrasonic analysis system according to an embodiment.

[0027] Figure 3 is a flow chart of a method for analyzing ultrasound image data using an ultrasound analysis system according to an embodiment.

[0028] Figure 4 is a schematic representation of a graphical display of an ultrasound analysis system according to an embodiment.

[0029] Figure 5 is a schematic representation of a graphical display of an ultrasound analysis system according to an embodiment. DETAILED DESCRIPTION

[0030] The present disclosure describes various embodiments of ultrasound analysis systems and methods. More generally, applicants have recognized and appreciated that it would be beneficial to provide ultrasound analysis that automatically generates information about clinical features related to the lungs. For example, an ultrasound analysis system receives or obtains ultrasound image data that includes clinical features related to the lungs. The system extracts and provides information about multiple different clinical features related to the lungs from the ultrasound image data.

[0031] The present disclosure describes various embodiments of ultrasound analysis systems and methods. More generally, the applicant has recognized and appreciated that it would be beneficial to provide ultrasound analysis that automatically generates clinical features related to the lungs from a time series of ultrasound image data. The ultrasound analysis system receives or obtains a time series of ultrasound image data of one or more regions of multiple different regions of one or both lungs of a patient. A first trained clinical lung feature recognition algorithm analyzes the received time series of ultrasound image data to identify a first clinical feature in the patient's lungs, wherein identifying the first clinical feature includes analyzing a plurality of frames in the time series, and wherein identifying the first clinical feature includes identifying the location of the first clinical feature within the plurality of frames. The system's trained clinical lung feature severity algorithm analyzes the identified first clinical feature to characterize the severity of the identified first clinical feature. Optionally, a trained clinical feature prioritization algorithm analyzes one or more identified clinical features to prioritize reporting of these features. The system's user interface provides the identified first clinical feature and the severity of the characterized first clinical, and optionally provides prioritization of one or more identified clinical features. The analysis of multiple frames in a time series may be understood as analyzing the multiple frames simultaneously, jointly, holistically, concurrently, synchronously, together or collaboratively. The analysis of multiple frames in a time series may be understood as analyzing the multiple frames in both the spatial domain and the temporal domain. The analysis of multiple frames in a time series may be understood as analyzing both the spatial and temporal information contained in the multiple frames.

[0032] According to an embodiment, ultrasound analysis systems and methods disclosed herein or otherwise contemplated automatically identify candidate features of interest in a lung ultrasound movie playback, and provide a static or feature-focused dynamic review of the feature based on the feature type. According to an embodiment, a reviewer can easily navigate between candidate features without having to repeatedly replay the entire video, and without having to manually start / stop / restart playback. According to an embodiment, a report on a lung ultrasound examination can be efficiently generated by confirming or rejecting individual candidate features after review. Similarly, during and after the review, detected and / or confirmed features can be displayed in an efficient manner, which does not require replaying the entire ultrasound video.

[0033] Depending on the embodiment, the systems and methods disclosed herein or otherwise contemplated may be used in any environment and any system on which lung ultrasound is acquired and / or reviewed. In particular, the present invention may be used in point-of-care and handheld ultrasound devices, such as the Philips and many other devices and systems. These systems and methods can be used as part of clinical practice as well as as research and development tools to speed up annotation workflows, among many other uses.

[0034] Thus, the ultrasound analysis systems and methods disclosed herein or otherwise contemplated provide numerous advantages over the prior art. Providing an ultrasound analysis system and method that can automatically detect and analyze clinical features related to the lungs in an understandable and interpretable manner can prevent severe lung damage, improve lung diagnosis and patient prognosis, and thus potentially save lives.

[0035] refer to Figure 1 , in one embodiment, is a flow chart of a method 100 for analyzing ultrasound image data using an ultrasound analysis system. The methods described in conjunction with the figures are provided only as examples and should be understood not to limit the scope of the present disclosure. The ultrasound analysis system may be any system described herein or otherwise contemplated. The ultrasound analysis system may be a single system or a plurality of different systems.

[0036] In step 110 of the method, an ultrasonic analysis system 200 is provided. Figure 2 The embodiment of the ultrasound analysis system 200 shown, for example, includes one or more of a processor 220, a memory 230, a user interface 240, a communication interface 250, a storage device 260, and an ultrasound device 270 interconnected via one or more system buses 212. It should be understood that Figure 2 In some respects, the system 200 is abstracted, and the actual organization of the components of the system 200 may be different and more complex than shown. In addition, the ultrasound analysis system 200 may be any system described or otherwise contemplated herein. Other elements and components of the system 200 are disclosed and / or contemplated elsewhere herein.

[0037] At step 120 of the method, ultrasound image data is sent to the system, obtained by the system, or otherwise received by the system. The ultrasound image data includes a time series of ultrasound image data, such as a video including multiple frames. The ultrasound image data can be obtained for a single zone or region of the patient's lungs, or can be obtained for multiple different regions of one or more of the patient's lungs. For example, ultrasound image data can be obtained for one, two, or more regions. The ultrasound image data can be received by the system in real time, or can be stored in local and / or remote memory and received by the system at some point in the future.

[0038] The ultrasound image data may be obtained using any ultrasound device or system, which may be any device or system suitable for obtaining or otherwise receiving ultrasound image data of a patient. One or more parameters of the ultrasound device may be set, adjusted, pre-programmed, or otherwise determined by a healthcare professional. The ultrasound device or system may be remote from, local to, or a component of the ultrasound analysis system 200.

[0039] The ultrasound image data includes data or other information about one or more of a plurality of different clinical features associated with the lungs. According to an embodiment, a clinical feature is any identifiable aspect of the lungs. A clinical feature may be a normal aspect or an abnormal aspect of the lungs. A clinical feature may indicate a healthy lung or a diseased or injured lung. Thus, a clinical feature may be anything that can be identified, for example, within or from the ultrasound image data. Examples of clinical features include A lines, B lines, merged B lines, pleural line abnormalities, consolidation, pleural effusion, and the like.

[0040] At step 130 of the method, the ultrasound analysis system analyzes one or more time series of the received ultrasound image data to identify clinical features in the patient's lungs. According to an embodiment, the ultrasound analysis system includes a trained clinical lung feature recognition algorithm that is configured to identify clinical features in the patient's lungs. Identifying a first clinical feature may include, for example, comparing multiple frames in a time series. Identifying a first clinical feature may also include, for example, identifying the position of the first clinical feature within multiple frames of the time series. Many other methods for identifying clinical features are also possible. According to an embodiment, the trained clinical lung feature recognition algorithm is configured to identify one or more specific types of clinical lung features. For example, the clinical lung feature recognition algorithm can be trained to identify specific types of clinical lung features, such as A lines, B lines, merged B lines, pleural line abnormalities, consolidation or pleural effusion, etc.

[0041] As described above, according to one embodiment, the ultrasound analysis system includes a single trained clinical lung feature recognition algorithm that is configured to recognize two or more different types of clinical features in the patient's lungs. According to another embodiment, the ultrasound analysis system includes multiple trained clinical lung feature recognition algorithms, each trained clinical lung feature recognition algorithm is configured to recognize one or more different types of clinical features in the patient's lungs. For example, the first clinical lung feature recognition algorithm can be trained to recognize or otherwise characterize B-lines. The second clinical lung feature recognition algorithm can be trained to recognize or otherwise characterize pleural line abnormalities, etc.

[0042] The clinical lung feature recognition algorithm is trained, programmed, configured or otherwise designed to specifically analyze selected lung-related clinical features, which means that the trained algorithm will recognize and extract or identify information of the selected lung-related clinical features. Therefore, according to an embodiment, the ultrasound analysis system includes one or more trained clinical lung feature recognition algorithms, which are trained or configured to recognize and extract or identify one or more of a plurality of different possible lung-related clinical features. According to an embodiment, the clinical lung feature recognition algorithm or model can be a deep neural network, or can be another model, such as a random forest classifier, a support vector machine classifier, a boost classifier, or any other type of machine learning model or algorithm. The clinical lung feature recognition algorithm can be trained using any method for training an algorithm or model, and can be stored in local and / or remote memory.

[0043] Therefore, according to an embodiment, the ultrasound analysis system is configured to identify or otherwise characterize one or more lung ultrasound features of interest. The ultrasound analysis system is also configured with information about whether a static ("S-type feature") or dynamic ("D-type feature") display should be used to review or measure the feature. According to an embodiment, a preferred review mode - static or dynamic - is predefined for each clinical lung feature. For example, a static review mode can be preferably used for clinical lung features such as A-lines, pleural line abnormalities, consolidation, atelectasis and / or pleural effusion, etc. A dynamic review mode can be preferably used for clinical lung features such as B-lines, combined B-lines, dynamic bronchial inflation signs, and lung sliding, etc.

[0044] The clinical lung feature recognition algorithm of the ultrasound analysis system is configured or trained to identify spatiotemporal locations where features may exist in movie playback (i.e., "candidate features"). According to an embodiment, the clinical lung feature recognition algorithm may utilize conventional image processing techniques, including domain transforms such as filtering, thresholding, spatial transforms, and Fourier transforms. The algorithm may also utilize machine learning techniques, including deep learning, particularly convolutional neural networks (CNNs), which are trained to recognize and detect as well as locate features. According to an embodiment, the algorithm is configured or trained to distinguish different instances of the same feature type. For example, when selecting frames or short clips to be reviewed, the same consolidation should only be counted once, and separate consolidations in a given video loop should be counted separately.

[0045] At step 140 of the method, the ultrasound analysis system analyzes or characterizes the severity of the clinical feature identified by the analysis in step 130 of the method. According to an embodiment, the ultrasound analysis system includes a trained clinical lung feature severity algorithm that is configured to characterize the severity of the identified first clinical feature. For example, the trained clinical lung feature severity algorithm can be configured to identify the severity of a feature that may appear multiple times in a movie playback. According to an embodiment, the algorithm can utilize conventional image processing techniques or machine learning-based methods (including CNNs) to determine the severity of the feature using a multi-class classification or regression method. Therefore, the algorithm can utilize various methods to extract features and determine feature severity.

[0046] At optional step 150 of the method, the ultrasound analysis system analyzes one or more time series of the received ultrasound image data to identify another type of clinical feature in the patient's lungs. According to an embodiment, the ultrasound analysis system includes a second, third, or more trained clinical lung feature recognition algorithms, each of which is configured to recognize different types or variations of clinical features in the patient's (one or more) lungs. As with the first algorithm, identifying clinical features may include, for example, analyzing multiple frames in a time series. Identifying clinical features may also include, for example, identifying the location of clinical features within multiple frames of a time series. Many other methods for identifying clinical features are also possible. According to an embodiment, the second trained clinical lung feature recognition algorithm is configured to identify one or more specific types of clinical lung features that are different from the types of clinical lung features identified or otherwise analyzed by the first trained clinical lung feature recognition algorithm.

[0047] Therefore, at optional step 160 of the method, the ultrasound analysis system analyzes or characterizes the severity of the clinical features identified by the analysis in step 150 of the method. According to an embodiment, the ultrasound analysis system includes a trained clinical lung feature severity algorithm that is configured to characterize the severity of the identified clinical features. For example, the trained clinical lung feature severity algorithm can be configured to identify the severity of features that may appear multiple times in a movie playback. According to an embodiment, the algorithm can utilize conventional image processing techniques or machine learning-based methods (including CNN) to determine the severity of the features using multi-class classification or regression methods. Therefore, the algorithm can utilize various methods to extract features and determine feature severity. The trained clinical lung feature severity algorithm utilized in step 160 of the method can be the same algorithm utilized in step 140 of the method, or can be a separate or different trained clinical lung feature severity algorithm.

[0048] The order in which multiple trained clinical lung feature recognition algorithms are utilized can depend on various factors. According to an embodiment, the order of analysis of the system can be based on user selections or options, predetermined programming, aspects of the ultrasound examination itself (such as the purpose of the examination or the type of examination), demographic or clinical information about the patient (such as diagnosis), and / or other possible selection mechanisms. For example, a user can provide a list of one or more clinical features for analysis, or can select one or more clinical features from a menu of possible clinical features. As another option, the system can be configured, programmed, or otherwise designed to automatically analyze a given list of different clinical features in a specific order or configuration. However, this automatic order or configuration can be adjusted based on user input or other information (such as the purpose of the examination or the type of examination) and many other possible adjustment mechanisms. According to yet another embodiment, multiple trained clinical lung feature recognition algorithms can simultaneously analyze the received ultrasound image data.

[0049] At optional step 170 of the method, the ultrasound analysis system prioritizes the identified clinical features. According to an embodiment, the ultrasound analysis system includes a clinical feature prioritization algorithm that is configured or trained to rank, prioritize, or otherwise order or process clinical lung features identified by the analysis of the lung ultrasound image data. According to an embodiment, the clinical feature prioritization algorithm is configured or trained to determine which identified clinical lung features should be highlighted or provided to a user, and / or in what order the identified clinical lung features should be highlighted or provided to a user.

[0050] According to an embodiment, a clinical feature prioritization algorithm is trained or configured to determine an effective order for displaying one or more detected candidate features to a user. The prioritization can be based on one or more of the following: the type of clinical feature identified, the severity of the first clinical feature and the second clinical feature characterized, the timing of the first clinical feature and / or the second clinical feature in the time series of the ultrasound image data, and / or the suspected or diagnosed clinical condition of the patient, as well as many other options.

[0051] For example, the prioritization can be based on the temporal appearance of the identified features in the cine playback, with earlier appearances taking precedence over later appearances and vice versa. As another example, the prioritization can be based on a fixed sequence of feature types, which can be programmed or determined by the user. For example, the fixed sequence can be: (1) B-lines (if any); (2) lung consolidation (if any); (3) pleural effusion (if any); (4) pleural line abnormalities (if any); (4) lung sliding abnormalities (if any); and / or (5) other features (if any). Many other sequences are possible.

[0052] As another example, the prioritization can be based on the patient's suspected diagnosis, prognosis, and / or clinical condition. For example, for patients suspected of having COVID-19, the prioritization can be: 1. B line, 2. Abnormal chest line, 3. Consolidation, 4. Pleural effusion, and 5. Lung sliding, as well as other possible prioritizations. As another example, for patients suspected of having community-acquired pneumonia, the prioritization can be: 1. Consolidation, 2. B line, 3. Pleural line abnormality, 4. Pleural effusion, and 5. Lung sliding, as well as other possible prioritizations. As another example, for patients suspected of having cardiogenic pulmonary edema, the prioritization can be: 1. B line, 2. Consolidation, 3. Pleural line abnormality, 4. Pleural effusion, and 5. Other features, as well as other possible prioritizations. According to an embodiment, the prioritization can be based on the potential difficulty determined by the confidence score. For example, the prioritization can be given to easier candidates first and to challenging candidates last. Many other prioritizations are also possible.

[0053] At step 180 of the method, the ultrasound analysis system provides the identified clinical features and the severity of the characterized clinical features to the user via the user interface of the system. Other information is also possible, including but not limited to the identity of the patient, patient demographic information, diagnosis or treatment information, and various other possible information. The information can be provided via the user interface using any method for conveying or displaying information, and the user interface can be any device, interface, or mechanism for providing the conveyed or displayed information. The user can be any user who reviews lung ultrasound image data, including but not limited to technicians, medical professionals, clinicians, patients, and / or any other users.

[0054] In embodiments where several trained clinical pulmonary feature recognition algorithms analyze data and identify different clinical features of different types, the user interface provides the different clinical features and the severity of the different clinical features characterized. In embodiments where a trained clinical feature prioritization algorithm prioritizes clinical features, the user interface provides prioritization information.

[0055] According to an embodiment, providing the identified clinical feature to a user via a user interface of the system comprises providing a subset of the time series of the received ultrasound image data. The subset may comprise, for example, the identified location of the identified first clinical feature. According to an embodiment, the subset is a time series that is smaller than the complete time series received by the lung ultrasound system. According to an embodiment, the subset is one or more static images.

[0056] According to an embodiment, depending on the type of feature being displayed, the image review may be displayed dynamically with playback of the image sequence around the temporal center of feature detection, or statically with display of one or more frames where the feature was detected with the highest confidence or largest size. According to an embodiment, the user interface may include a UI option to switch between static / dynamic views, such as a play / stop option. Figure 4 , in one embodiment, is an example of a user interface for reviewing a lung ultrasound cine playback.

[0057] According to an embodiment, the user interface may also include a navigation pane to help the user move from one feature to the next during review and also provide a snapshot of one or more detected features and their location along the video playback. Figure 5 , in one embodiment, is an example of how a navigation plane may be displayed, where frames are shown as a series of slices stacked together to form a heat map. The heat map may be an axial projection or any other format capable of compressing the video data. For each feature, high confidence frames are highlighted and candidates from two or more detections are displayed. After reviewing each selected frame or short clip, the status of the review may be displayed. For example, as Figure 5 As shown, the display has been updated to show review states such as "Reviewed," "Skipped," and "Valid Frame." Although these labels are provided, they are non-limiting examples, and many other labels may be utilized - including text, graphics, and / or other indicators. Alternatively, rather than a separate graph for each feature, there may be a single graph showing all candidate features, with different feature types shown in different colors to easily separate them during review. The user may show / hide feature types based on priority.

[0058] At optional step 190 of the method, the user provides feedback about the provided information via a user interface of the lung ultrasound analysis system. The user interface allows the user to review the provided information and further enables the user to provide feedback to the system about the provided information. Feedback can be provided via the user interface using any method for conveying or providing information, and the user interface can be any device, interface, or mechanism for receiving the transmitted or provided information.

[0059] For example, a user can select or change a feature review priority order provided or determined by the system, a user can select a cine loop for review from all cine loops in the inspection, a user can select to display candidate feature types and / or severities, a user can select to display the portion of the cine loop where features are present in a dynamic or static view, a user can navigate to the next / previous candidate feature, a user can accept or reject or change the feature type and / or severity, a user can adjust the ROI around a detected feature (where, for example, a manual adjustment to any frame around a candidate can result in a corresponding automatic adjustment to the ROI in all surrounding frames), and / or a user can add annotations to be associated with the feature, as well as many other types of feedback.

[0060] At step 192 of the method, the lung ultrasound system generates a report including the identified first clinical feature and / or the severity of the first clinical feature characterized. Other information is also possible, including but not limited to the identity of the patient, patient demographic information, diagnosis or treatment information, and various other possible information. The report can be provided via a user interface using any method for conveying or displaying information, and the user interface can be any device, interface, or mechanism for providing the conveyed or displayed information.

[0061] According to embodiments, generation of the report may include selection of one or more cine loops from the examination (where, for example, names / numbers may have been selected to distinguish the selected cine loops, e.g., using lung region numbers or nomenclature such as "Left Anterior Inferior (LAI)", etc.). Generation of the report may also include displaying sample images for each reviewed / confirmed feature in each cine loop of the examination, the option to include or exclude any image or feature from the summary report, and / or a summary containing the total number of features or total number of frames for each feature type. Many other features of the report are also possible.

[0062] refer to Figure 3, in one embodiment, is a flow chart of a method 300 for generating a lung ultrasound feature report. At step 310, the system obtains, receives or otherwise acquires one or more lung ultrasound (LUS) movie playbacks of one or more zones or regions of the patient's lungs. At step 320, the system utilizes one or more trained clinical lung feature recognition algorithms to identify one or more clinical lung features, including the spatiotemporal location of the features. The system also utilizes one or more trained clinical lung feature severity algorithms to determine the severity of the one or more clinical lung features identified. At step 330, the system optionally prioritizes the one or more clinical lung features identified using a trained prioritization algorithm. At step 340, the system provides one or more clinical lung features, the severity of one or more clinical lung features, and / or the prioritization of one or more clinical lung features via a user interface using static and / or dynamic review. At step 350, the system generates a report including the determined information, which may optionally include feedback received from the user.

[0063] Thus, the methods and systems described herein or otherwise contemplated provide numerous advantages over the prior art. For example, the system provides improved interpretability of ultrasound imaging compared to prior art systems because clinicians are able to better assess more lung ultrasound imaging data, including more clinical lung features, in a shorter period of time. Detecting and visualizing relevant clinical lung features and providing intermediate results at the frame / video / exam level allows the user to interpret ultrasound findings together with other patient medical information and make a final, more informed clinical judgment, thereby improving patient outcomes.

[0064] Depending on the embodiment, the methods and systems described or otherwise contemplated herein include many applications. For example, the system can be used in a pre-hospital setting, as an initial assessment in the emergency room, for follow-up after treatment, and many other settings. The method is applicable to all ultrasound imaging systems, including point-of-care applications. The method and system can be used in a variety of settings, including an ambulance, ER, or critical care or surgical situations.

[0065] refer to Figure 2 , is a schematic diagram of an ultrasonic analysis system 200. The system 200 may be any system described or otherwise contemplated herein, and may include any components described or otherwise contemplated herein. It should be understood that Figure 2 Some aspects are abstracted, and the actual organization of components of system 200 may be different and more complex than shown.

[0066] According to an embodiment, the system 200 includes a processor 220 capable of executing instructions stored in the memory 230 or the storage device 260 or otherwise processing data to, for example, perform one or more steps of the method. The processor 220 may be formed by one or more modules. The processor 220 may take any suitable form, including but not limited to a microprocessor, a microcontroller, multiple microcontrollers, a circuit, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a single processor, or multiple processors.

[0067] Memory 230 may take any suitable form, including non-volatile memory and / or RAM. Memory 230 may include various memories, such as L1, L2 or L3 cache or system memory. Thus, memory 230 may include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read-only memory (ROM) or other similar memory devices. The memory may store an operating system, etc. The processor uses RAM to temporarily store data. According to an embodiment, the operating system may include code that controls the operation of one or more components of system 200 when executed by the processor. It is apparent that in an embodiment where the processor implements one or more functions described herein in hardware, software described as corresponding to such functions in other embodiments may be omitted.

[0068] The user interface 240 may include one or more devices for enabling communication with a user. The user interface may be any device or system that allows information to be communicated and / or received, and may include a display, mouse, and / or keyboard for receiving user commands. In some embodiments, the user interface 240 may include a command line interface or a graphical user interface that may be presented to a remote terminal via the communication interface 250. The user interface may be placed together with one or more other components of the system, or may be located remotely from the system and communicate via a wired and / or wireless communication network.

[0069] The communication interface 250 may include one or more devices for implementing communication with other hardware devices. For example, the communication interface 250 may include a network interface card (NIC) configured to communicate according to the Ethernet protocol. In addition, the communication interface 250 may implement a TCP / IP stack for communicating according to the TCP / IP protocol. Various alternative or additional hardware or configurations for the communication interface 250 will be apparent.

[0070] The storage device 260 may include one or more machine-readable storage media, such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium, an optical storage medium, a flash memory device, or a similar storage medium. In various embodiments, the storage device 260 may store instructions for execution by the processor 220 or data that the processor 220 may operate on. For example, the storage device 260 may store an operating system 261 for controlling various operations of the system 200.

[0071] It will be apparent that various information described as being stored in storage device 260 may additionally or alternatively be stored in memory 230. In this regard, memory 230 may also be considered to constitute a storage device, and storage device 260 may be considered to be a memory. Various other arrangements will be apparent. In addition, both memory 230 and storage device 260 may be considered to be non-transitory machine-readable media. As used herein, the term non-transitory will be understood to exclude transient signals but include all forms of storage devices, including both volatile and non-volatile memory.

[0072] Although system 200 is shown as including one of each of the described components, various components may be replicated in various embodiments. For example, processor 220 may include multiple microprocessors that are configured to independently perform the methods described herein or are configured to perform the steps or subroutines of the methods described herein, so that multiple processors collaborate to implement the functions described herein. In addition, in the case where one or more components of system 200 are implemented in a cloud computing system, various hardware components may belong to separate physical systems. For example, processor 220 may include a first processor in a first server and a second processor in a second server. Many other variations and configurations are possible.

[0073] According to an embodiment, the storage device 260 of the system 200 may store one or more algorithms, modules and / or instructions to perform one or more functions or steps of the methods described or otherwise contemplated herein. For example, the system may include ultrasound imaging data 262, a trained clinical lung feature recognition algorithm 263, a trained clinical lung feature severity algorithm 264, a trained clinical feature prioritization algorithm 265, and / or reporting instructions 266, among other possible instructions and / or data.

[0074] According to an embodiment, the ultrasound imaging data 262 is any ultrasound imaging data sent to the system, obtained by the system, or otherwise received by the system. The ultrasound image data includes a time series of ultrasound image data, such as a video including multiple frames. The ultrasound image data can be obtained for a single zone or region of the patient's lungs, or can be obtained for multiple different regions of one or more lungs of the patient's lungs. For example, ultrasound image data can be obtained for one, two or more regions. The ultrasound image data can be received by the system in real time, or can be stored in a local and / or remote memory and received by the system at a future point. Any ultrasound device or system 270 can be used to obtain ultrasound image data, which can be any device or system suitable for obtaining or otherwise receiving ultrasound image data of a patient. The ultrasound device or system can be remote from the ultrasound analysis system 200, local to the ultrasound analysis system 200, or a component thereof.

[0075] According to an embodiment, the trained clinical lung feature recognition algorithm 263 is any model or algorithm that is trained or configured to analyze the received ultrasound image data to identify clinical features in the patient's lungs. Identifying a first clinical feature may include, for example, analyzing multiple frames in a time series. Identifying a first clinical feature may also include, for example, identifying the position of a first clinical feature within multiple frames of a time series. According to an embodiment, the trained clinical lung feature recognition algorithm is configured to identify one or more specific types of clinical lung features. For example, the clinical lung feature recognition algorithm can be trained to identify specific types of clinical lung features, such as A lines, B lines, merged B lines, pleural line abnormalities, consolidation or pleural effusion, etc. According to an alternative embodiment, the ultrasound analysis system includes a plurality of trained clinical lung feature recognition algorithms, each of which is configured to identify one or more different types of clinical features in the patient's lungs. For example, a first clinical lung feature recognition algorithm can be trained to identify or otherwise characterize a B line. A second clinical lung feature recognition algorithm can be trained to identify or otherwise characterize a pleural line abnormality, etc. According to an embodiment, the clinical lung feature recognition algorithm or model can be a deep neural network, or can be another model, such as a random forest classifier, a support vector machine classifier, a boost classifier, or any other type of machine learning model or algorithm. The clinical lung feature recognition algorithm can be trained using any method for training an algorithm or model.

[0076] According to an embodiment, the trained clinical lung feature severity algorithm 264 is any model or algorithm that is trained or configured to analyze the received ultrasound image data to characterize the severity of the identified first clinical feature. For example, the trained clinical lung feature severity algorithm can be configured to identify the severity of a feature that may appear multiple times in a movie playback. According to an embodiment, the algorithm can utilize conventional image processing techniques or machine learning-based methods (including CNNs) to determine the severity of the feature using multi-class classification or regression methods. Therefore, the algorithm can utilize various methods to extract features and determine feature severity.

[0077] According to an embodiment, the trained clinical feature prioritization algorithm 265 is any model, algorithm or rule set that is utilized, trained or configured to analyze the identified clinical features in order to rank, prioritize or otherwise order or process the clinical lung features identified by the analysis of the lung ultrasound image data. According to an embodiment, the clinical feature prioritization algorithm is configured or trained to determine which identified clinical lung features should be highlighted or provided to the user, and / or in what order the identified clinical lung features should be highlighted or provided to the user. According to an embodiment, the clinical feature prioritization algorithm is trained or configured to determine an effective order for displaying one or more detected candidate features to the user. Prioritization can be based on the type of clinical feature identified, the severity of the characterized first clinical feature and the second clinical feature, the timing of the first clinical feature and / or the second clinical feature in the time series of the ultrasound image data, and / or the suspected or diagnosed clinical condition of the patient, as well as one or more of many other options.

[0078] According to an embodiment, the report instructions 266 guide the system to generate a report or visualization and provide the report or visualization to the user via the user interface 240 of the ultrasound analysis system 200. The report or visualization includes, for example, any information generated by the system during the analysis, including but not limited to the severity of the clinical features identified and characterized to the user via the user interface of the system. Other information is also possible, including but not limited to the identity of the patient, patient demographic information, diagnosis or treatment information, and various other possible information. Any method for conveying or displaying information can be used to provide information via the user interface, and the user interface can be any device, interface or mechanism for providing the conveyed or displayed information. The user can be any user who reviews lung ultrasound image data, including but not limited to technicians, medical professionals, clinicians, patients, and / or any other users.

[0079] According to an embodiment, the report instruction 266 instructs the system to generate and provide a final report, which may include, for example, feedback from a user. The final report may include the severity of the clinical features identified and / or the clinical features characterized. Other information is also possible, including but not limited to the patient's identity, patient demographic information, diagnosis or treatment information, and various other possible information. The report may be provided via a user interface using any method for conveying or displaying information, and the user interface may be any device, interface, or mechanism for providing the conveyed or displayed information. According to an embodiment, the generation of the report may include selecting one or more movie playbacks from the examination (where, for example, a name / number may have been selected to distinguish the selected movie playback, for example, using lung region numbering or naming, such as "left anterior lower (LAI)", etc.). The generation of the report may also include displaying sample images of each reviewed / confirmed feature in each movie playback of the examination, including or excluding any image or feature option from the summary report, and / or a summary report containing the total number of features or total number of frames of each feature type. Many other features of the report are possible.

[0080] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0081] Unless clearly indicated to the contrary, the articles "a" and "an" as used herein in the specification and claims should be understood to mean "at least one".

[0082] As used herein in the specification and claims, the phrase "and / or" should be understood to mean "either or both" of the elements so combined, i.e., elements present in combination in some cases and separately in other cases. Multiple elements listed with "and / or" should be interpreted in the same manner, i.e., "one or more" of the elements so combined. In addition to the elements specifically identified by the "and / or" clause, other elements may optionally be present, whether related or unrelated to those elements specifically identified.

[0083] As used herein in the specification and claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" should be interpreted as inclusive, that is, including at least one of multiple elements or lists, but also including more than one, and optionally, additional unlisted items. Only terms that clearly indicate the opposite, such as "only one" or "exactly one", or when used in the claims, "consisting of..." will refer to including exactly one element in a plurality of elements or lists of elements. Typically, the term "or" as used herein will only be interpreted as indicating an exclusive alternative (i.e., "one or the other but not both") when preceded by an exclusive term, such as "either", "one", "only one" or "exactly one".

[0084] As used herein in the specification and claims, the phrase "at least one" with respect to a list of one or more elements should be understood to mean at least one element selected from any one or more elements in the list of elements, but not necessarily including at least one of each element specifically listed within the list of elements, and not excluding any combination of elements in the list of elements. This definition also allows that elements may optionally be present other than those specifically identified within the list of elements referenced by the phrase "at least one," whether related or unrelated to those elements specifically identified.

[0085] It should also be understood that in any method claimed herein that includes more than one step or action, the order of the method steps or actions is not necessarily limited to the order in which the method steps or actions are described unless clearly indicated to the contrary.

[0086] In the claims, as well as in the foregoing description, all transitional phrases such as "comprising," "including," "carrying," "having," "containing," "involving," "maintaining," "consisting of," and the like shall be construed as open ended, i.e., meaning including, but not limited to, including. Only the transitional phrases "consisting of" and "consisting essentially of" shall be closed or semi-closed transitional phrases, respectively, as set forth in Section 2111.03 of the U.S. Patent Office Manual of Patent Examining Procedures.

[0087] Although several inventive embodiments have been described and illustrated herein, a person of ordinary skill in the art will readily appreciate various other means and / or structures for performing functions and / or obtaining results and / or one or more advantages described herein, and each of these changes and / or modifications is considered to be within the scope of the inventive embodiments described herein. More generally, it will be readily understood by those skilled in the art that all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and that actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which the present invention is taught. Those skilled in the art will recognize or be able to determine many equivalents of the specific inventive embodiments described herein using no more than routine experiments. Therefore, it should be understood that the foregoing embodiments are presented only by way of example, and within the scope of the appended claims and their equivalents, embodiments of the present invention may be practiced in a manner different from that specifically described and claimed. The inventive embodiments of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if two or more such features, systems, articles, materials, kits, and / or methods are not inconsistent with each other, any combination of these features, systems, articles, materials, kits, and / or methods is included within the scope of the invention disclosed herein.

Claims

1. A method (100) for analyzing ultrasound image data, comprising: receiving (120) a time series of ultrasound image data of one or more of a plurality of different regions of one or both lungs of a patient; analyzing (130) the received time series of ultrasound image data using a trained clinical lung feature recognition algorithm to identify a first clinical feature in a lung of the patient, wherein identifying the first clinical feature includes analyzing a plurality of frames in the time series, and wherein identifying the first clinical feature includes identifying a location of the first clinical feature within the plurality of frames; analyzing (140) the identified first clinical feature using a trained clinical-pulmonary feature severity algorithm to characterize a severity of the identified first clinical feature; and The identified first clinical feature and the characterized severity of the first clinical feature are provided (180) via a user interface.

2. The method according to claim 1, further comprising the steps of: analyzing (150) the received time series of ultrasound image data using the trained clinical lung feature recognition algorithm to identify a second clinical feature in the patient's lungs, wherein the second clinical feature is different from the first clinical feature; and analyzing (160) the identified second clinical feature using the trained clinical pulmonary feature severity algorithm to characterize the severity of the identified second clinical feature; Wherein, the providing step further includes providing the identified second clinical feature and the characterized severity of the second clinical feature via the user interface.

3. The method according to claim 1, further comprising the steps of: analyzing (150) the received time series of ultrasound image data using a second trained clinical lung feature recognition algorithm to identify a second clinical feature in the patient's lungs, wherein the second clinical feature is different from the first clinical feature; and analyzing (160) the identified second clinical feature using the trained clinical pulmonary feature severity algorithm to characterize the severity of the identified second clinical feature; Wherein, the providing step further includes providing the identified second clinical feature and the characterized severity of the second clinical feature via the user interface.

4. The method according to claim 1, further comprising: Prioritizing the identified first clinical feature or the identified second clinical feature using a trained clinical feature prioritization algorithm (170), wherein the prioritization is based on one or more of: a type of the identified clinical feature, a severity of the characterized first and second clinical features, a timing of the first and / or second clinical features in the time series of ultrasound image data, and / or a suspected or diagnosed clinical condition of the patient; Wherein, the providing step also includes providing the priority sorting.

5. The method according to claim 1, wherein: Identifying a location of the first clinical feature within the plurality of frames includes identifying a spatiotemporal location across the plurality of frames.

6. The method according to claim 1, wherein: Providing the identified first clinical feature and the characterized severity of the first clinical feature includes providing a subset of the time series of received ultrasound image data, the subset including the identified location of the identified first clinical feature.

7. The method according to claim 6, wherein: The subset is a time series.

8. The method according to claim 1, further comprising: Feedback is received (190) from a user via the user interface regarding the provided identified first clinical feature and / or the characterized severity of the first clinical feature.

9. The method according to claim 8, wherein: The feedback includes adjustment of the severity of the characterized first clinical feature, selection of one or more frames in the time series of ultrasound image data, acceptance or rejection of the feature, and / or change of the type of feature.

10. An ultrasound analysis system (200) configured to analyze ultrasound image data, comprising: a time series (262) of ultrasound image data of one or more of a plurality of different regions of one or both lungs of a patient; a trained clinical lung feature recognition algorithm (263) configured to analyze a time series of received ultrasound image data to identify a first clinical feature in a lung of the patient, wherein identifying the first clinical feature comprises analyzing a plurality of frames in the time series, and wherein identifying the first clinical feature comprises identifying a location of the first clinical feature within the plurality of frames; a trained clinical pulmonary feature severity algorithm (264) configured to analyze the identified first clinical feature to characterize a severity of the identified first clinical feature; A processor (220) configured to: (i) analyze the received time series of ultrasound image data using the trained clinical lung feature recognition algorithm to identify a first clinical feature in the patient's lungs; (ii) analyze the identified first clinical feature using the trained clinical lung feature severity algorithm to characterize the severity of the identified first clinical feature; and A user interface (240) is configured to provide the identified first clinical feature and the characterized severity of the first clinical feature.

11. The ultrasonic analysis system according to claim 10, wherein: The system also includes a trained clinical feature prioritization algorithm (265) configured to prioritize one or more identified clinical features; The processor is further configured to prioritize one or more identified clinical features using the trained clinical feature prioritization algorithm, wherein the prioritization is based on one or more of: a type of the identified clinical feature, a severity of the first and second clinical features characterized, a timing of the first and / or second clinical features in the time series of ultrasound image data, and / or a suspected or diagnosed clinical condition of the patient; and The user interface is further configured to provide the priority ranking.

12. The ultrasonic analysis system according to claim 10, wherein: Providing the identified first clinical feature and the characterized severity of the first clinical feature includes providing a subset of the time series of received ultrasound image data, the subset including the identified location of the identified first clinical feature.

13. The ultrasonic analysis system according to claim 10, wherein: The processor is further configured to: receiving feedback from a user via the user interface regarding the provided identified first clinical feature and / or the characterized severity of the first clinical feature; and Based on the received feedback, a report is generated including the identified first clinical feature and / or the characterized severity of the first clinical feature.

14. The ultrasonic analysis system according to claim 10, wherein: The user interface also includes a summary display of the time series of ultrasound image data and the identified first clinical feature, wherein a user is able to select a region of the time series and / or the identified first clinical feature for review.

15. The ultrasonic analysis system according to claim 14, wherein: After review by the user, the summary display of the time series of ultrasound image data and / or the identified first clinical feature is updated by the processor to show a status of the review.

Images