Automated system for ultrasound imaging analysis, reporting, and reviewing

Abstract

A system for analyzing ultrasound images includes an imaging mode of operation employed in a healthcare facility examination room to assist in analyzing ultrasound images from an ultrasound examination, a reporting mode of operation employed in a healthcare facility workstation for generating reports based on the system's analysis and examination findings obtained in electronic format directly from the ultrasound system, and a reading mode of operation employed in a healthcare reading room for allowing a healthcare professional to view the ultrasound examination findings and finalize the reports from the reporting mode to be sent to a healthcare facility data storage system, where the healthcare facility data storage system includes one or more of an Electronic Health Record (independent hospital information system or an electronic health / medical record ) system and a (Picture Archival and Communication System (independent healthcare facility data storage system ).

Description

AUTOMATED SYSTEM FOR ULTRASOUND IMAGING ANALYSIS, REPORTING, AND REVIEWINGFIELD

[0001] The disclosed exemplary embodiments are directed to an ultrasound assistance system, in particular, to a system for analyzing prenatal ultrasound images.BACKGROUND

[0002] Ultrasound imaging is very important during pregnancy to assess fetal development and plan for pregnancy management. However, problems such as high fetal mobility, excessive maternal abdominal wall thickness, and inter-observer variability limit the effectiveness of traditional ultrasound in clinical applications.

[0003] Fetal ultrasound imaging and interpretation relies heavily on the expertise and experience of ultrasound healthcare professionals (HCPs). The main limitation of manually performed fetal ultrasound examinations is the high intra-reader variability and low inter-reader similarity among ultrasound images captured and interpreted.

[0004] It would be advantageous to provide an ultrasound assistance system that addresses these shortcomings and others.SUMMARY

[0005] The disclosed embodiments are directed to a system for automated analyzation of ultrasound images, automated detection of diagnostic views / planes, automated identification of anatomical structures to confirm their presence / visualization or absence / non-visualization for identification of potential structural abnormalities, automated verification of quality / appropriateness criteria, automated caliper placement and measurements to assess growth and development and annotations of these views / planes for audit / archiving purposes, and automated reporting of ultrasound examination for reducing manual and repetitive tasks.

[0006] The application of artificial intelligence (Al) in ultrasound imaging optimizes the ultrasound examination by shortening the examination time, reducing the Health CareProfessional’s (HCP’s) workload, and improving diagnostic accuracy. The application of Al in ultrasound for standard plane detection, biometric parameter measurement, annotation, detection of abnormalities, and disease diagnosis facilitates conventional imaging approaches, in particular by providing decision support to clinicians in consideration of exam efficiency, organization, and display of examination findings.

[0007] The disclosed embodiments may have particular application to obstetric ultrasound including optimization of fetal ultrasound examination images, fetus pose estimation, and pre-term birth predictions. In addition, use of the disclosed Al and Machine Learning (ML) techniques are beneficial in analyzing ultrasound images, detecting fetal standard planes, automated detection of diagnostic views and planes, identification of anatomical structures, automated measurements, annotations of structures on these views and planes and performing automated measurements and annotations corresponding to specific trimesters of pregnancy.

[0008] In at least one aspect, the disclosed embodiments are directed to a browser-based Software As Medical Device (SaMD), that utilizes AI / ML technology to assist trained HCPs in analyzing ultrasound images in real-time during routine pregnancy diagnostic ultrasound examination or routine fetal / obstetric ultrasound examination that may be easily integrated with compatible ultrasound imaging systems.

[0009] The system is designed to support the examination workflow by providing a user with the ability to modify and customize the system’s analyses and, therefore aid an HCP in reviewing the system’s analyses and generating reports, in particular for fetal / obstetric ultrasound examination.

[0010] The device works by utilizing AI / ML algorithms to analyze ultrasound images following the quality criteria, i.e., according to ISUOG and Al UM clinical guidelines ensuring appropriate view / plane for structure detection, measurement, annotation, and, report generation, additionally having the option for the HCPs to override the above- mentioned findings of the software.

[0011] In at least another aspect, the disclosed embodiments are directed to a system for analyzing ultrasound images that includes an imaging mode of operation employed in an edge computing device in a healthcare facility ultrasound examinationroom to assist in analysis of ultrasound images and examination findings obtained in electronic format directly from an ultrasound system, a reporting mode of operation employed in a healthcare facility workstation for generating reports based on the imaging mode’s analysis of the ultrasound images and examination findings obtained in electronic format directly from the ultrasound system, and a reading mode of operation employed in a healthcare reading room workstation for providing a healthcare professional (HCP) with the ability to view the ultrasound images and examination findings obtained in electronic format directly from an ultrasound system, view the imaging mode’s analysis of the ultrasound images and examination findings, and using the ultrasound images and examination findings obtained in electronic format directly from an ultrasound system, and the imaging mode’s analysis of the ultrasound images and examination findings, to finalize the reports from the reporting mode to be sent to a healthcare facility data storage system, wherein the healthcare facility data storage system includes one or more of an independent hospital information system, an electronic health / medical record system, or an independent healthcare facility data storage system.In a further aspect, a system for analyzing ultrasound images includes an edge computing device having a display and a processor coupled to an ultrasound system and a local server coupled to the edge computing device, a reporting mode workstation, and a reading room workstation, the local server comprising an artificial intelligence engine, wherein the edge computing device is configured to send ultrasound examination images and examination findings obtained in electronic format directly from the ultrasound system to the local server for analysis by the artificial intelligence engine, retrieve analyses of the ultrasound examination images and examination findings generated by the artificial intelligence engine, wherein the analyses include diagnostic views, anatomical measurements, and annotations, and display the analyses for review and editing, wherein the local server is configured to provide the analyses to the reporting mode workstation, wherein the reporting mode workstation is configured to operate a reporting mode for generating reports based on the Al engine’s analyses, wherein the local server is configured to provide the ultrasound images and examination findings obtained in electronic format directly from the ultrasound system, the analyses of the ultrasound examination images and examination findings generated by theartificial intelligence engine, and the reports to the reading room workstation, wherein the reading room workstation is configured to operate a reading mode for allowing an HCP to view the ultrasound images and examination findings obtained in electronic format directly from an ultrasound system, the analyses of the ultrasound examination images and examination findings generated by the artificial intelligence engine, and the reports, and to finalize the reports, and wherein the local server is configured to send the ultrasound images and examination findings obtained in electronic format directly from the ultrasound system, the analyses of the ultrasound examination images and examination findings generated by the artificial intelligence engine, the reports, and the finalized reports, to a healthcare facility data storage system.

[0012] In a still further aspect, a method for analyzing ultrasound images includes requesting a patient’s medical information from an independent hospital information system or an electronic health / medical record, using an ultrasound system to capture ultrasound image frames of the patient during ultrasound examination, pre-processing the ultrasound image frames for analysis including one or more of resizing, noise reduction, and contrast enhancement, using an artificial intelligence engine to automatically analyze the ultrasound image frames to detect diagnostic views, provide anatomical measurements, and annotations; and automatically validate the diagnostic views, anatomical measurements, and annotations. The method further includes reading and editing the diagnostic views, anatomical measurements, and annotations by a HCP, and storing the patient’s ultrasound image frames, diagnostic views, anatomical measurements, annotations, and reviewed and edited diagnostic views, in one or more of the independent hospital information system or an electronic health / medical record system and independent healthcare facility data storage systemBRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 illustrates three modes of operation of a system for analyzing ultrasound images;

[0014] Figure 2 illustrates a detailed block diagram of various healthcare facility’s information system components integrated with the system components;

[0015] Figure 3 depicts an exemplary system software architecture of the disclosed embodiments;

[0016] Figure 4 shows a detailed diagram of the data flow of the imaging mode;

[0017] Figure 5 depicts a login feature that requires entry of a user’s user name and password;

[0018] Figures 6 and 7 show a setup feature having a user interface with options for choosing a workflow and a list of available fetal / obstetric views and structures for examination that the Al is capable of assessing for;

[0019] Figures 8 and 9 depict a dashboard feature that allows a user to select a patient from the patient modality worklist fetched from an independent hospital information system or an electronic health / medical record or an independent healthcare facility data storage system or any worklist server (e.g., an independent hospital information system, an electronic health / medical record system, an independent healthcare facility data storage system, a modality work list, or a remote installation services server), and where fetal growth and developmental parameters are calculated and displayed;

[0020] Figure 10 illustrates a quality criteria feature that contains the criteria the images / frames of the corresponding diagnostic view is assessed on, along with the indication of fulfillment of each criterion. The level of fulfillment of the criteria determines whether a frame can be considered an optimal quality diagnostic view for automated measurements and annotations;

[0021] Figure 11 shows an automated anatomical survey feature;

[0022] Figure 12 depicts an automated caliper placements and measurements feature;

[0023] Figures 13-14 portray a video player feature;

[0024] Figure 15 shows a summary of the imaging mode, providing an option to customize the formula and method for obtaining relevant measurements at the end of the examination .

[0025] Figures 16 and 17 illustrate an information tool feature for providing information on imaging techniques and landmarks;

[0026] Figure 18 shows a summarized findings feature;

[0027] Figure 19 depicts a user logout feature that logs a user out when activated;

[0028] Figure 20 portrays a detailed diagram of the data flow of the reporting mode;

[0029] Figures 21 -22 show a login feature of the reporting mode;

[0030] Figure 23 portrays the display of patient information that is fetched from the independent hospital information system or an electronic health / medical record system of the healthcare facility;

[0031] Figures 24-26 portray an editable pre-filled report template;

[0032] Figure 27 depicts a summary of the visualization states of all reported structures and anatomies;

[0033] Figure 28 depicts a window where a user can select pre-entered and customizable impressions for a scan or examination image;

[0034] Figure 29 shows an interface where a user may indicate that actions and steps have been completed to compile, format, and build a report;

[0035] Figure 30 shows an image view feature;

[0036] Figure 31 depicts a view where images may be selected for report generation;

[0037] Figure 32 shows a growth charts display feature;

[0038] Figures 34 and 35 illustrate a preview report feature;

[0039] Figures 36 and 37 illustrate a sample report generated by the software;

[0040] Figure 38 portrays a detailed diagram of the data flow of the reading mode; and

[0041] Figure 39 shows a network architecture typical for the system installed in a healthcare facility.DETAILED DESCRIPTION

[0042] The aspects and advantages of the exemplary embodiments will become apparent from the following detailed description considered in conjunction with the accompanying drawings. It is to be understood, however, that the drawings are designed solely for purposes of illustration and not as a definition of the limits of the invention, for which reference should be made to the appended claims. Additional aspects and advantages of the invention will be set forth in the description that follows, and, in part will be obvious from the description or may be learned by practice of the invention. Moreover, the aspects and advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out in the appended claims.

[0043] Abbreviations:Al: Artificial IntelligenceAIUM: American Institute of Ultrasound in MedicineAPI: Application Programming InterfaceDICOM: Digital Imaging and Communication in MedicineHCP: Healthcare ProfessionalHL: Health LevelISUOG: International Society of Ultrasound in Obstetrics and GynecologyML: Machine LearningMWL: Modality Work ListREST or RESTful: An API conforming to the REpresentational State Transfer architectural styleRPC: Remote Procedure Call

[0044] For purposes of the disclosed embodiments, ultrasound examination images include still images, cine, video frames, video loops, and video streams.

[0045] The terms fetal and prenatal are used synonymously throughout this application.

[0046] Figure 1 illustrates three modes of operation of the system 100 for analyzing ultrasound images: an imaging mode of operation 102 employed in a healthcare facility examination room to assist in analyzing ultrasound images during an ultrasound examination; a reporting mode of operation 104 employed in a healthcare facility workstation for generating reports based on the findings directly obtained from the ultrasound system and Origin Medical system’s analysis of the ultrasound images obtained from the ultrasound system; and a reading mode of operation 106 employed in a healthcare reading room for allowing an HOP to view the system’s findings from the ultrasound examination and finalize the reports from the reporting mode to be sent to a healthcare facility data storage system.

[0047] The system 100 may generally include an edge computing device with a display and a processor coupled to at least one ultrasound system and a local server. The local server may also be coupled to a workstation and a reading room workstation.

[0048] The edge computing device may have the capability to implement the imaging mode 102 including collecting ultrasound examination images from the ultrasound system and sending them to the local server for Al analysis. Once the Al analysis is complete, the edge computing device may retrieve an Al generated analysis of the ultrasound examination images, including diagnostic views, anatomical measurements, and annotations, and display the analysis for review and editing.

[0049] The local server 210 may operate to provide the reviewed and edited analysis of ultrasound images to the workstation, where the workstation may be configured to operate the reporting mode 104 for generating reports based on the Al engine’s analysis of the anatomical structures and examination findings obtained in electronic format directly from the ultrasound system .

[0050] The local server 210 may operate to provide the reports to the reading room workstation which is configured to operate the reading mode 106 and allow an HOP to view the ultrasound examination images and the interpretations of anatomical structures, and finalize the reports from the reporting mode 104.

[0051] The local server 210 may be further configured to send the ultrasound examination images, the interpretations of anatomical structures, the reviewed and edited interpretations of anatomical structures, the finalized reports to a healthcare facility data storage system.

[0052] The system 100 may generally be configured as a browser-based software as a medical device (SaMD) application intended to analyze fetal / obstetric structures using AI / ML techniques to automatically detect views, detect anatomical structures within the views, verify quality criteria of the views for performing automated measurements and annotations, and aid in reviewing and generation of reports. The system 100 is generally intended for use as a concurrent reading aid by HCPs, during the acquisition and interpretation of pregnancy ultrasound examinations or fetal / obstetric ultrasound examinations. The system 100 may be utilized during pregnancy or fetal / obstetric ultrasound examinations on first, second, and third-trimester expectant mothers with singleton pregnancies.

[0053] The system 100 may be implemented as a workflow optimization solution that assists in analyzing fetal / obstetric ultrasound examination images, by seamlessly integrating into an existing workflow, and may be effected as a browser-based application that runs Al algorithms on the local server 210, and acquires the data from the edge computing device placed inside the examination room. The findings of the Al algorithms may be displayed in the reporting mode 104, followed by subsequent examination image review in the reading mode 106.

[0054] Figure 2 illustrates a detailed block diagram of various healthcare facility’s information system components integrated with the system components. The components include a data acquisition unit 202, a data processing and display unit 204, and a data reception and storage unit 206.

[0055] The data acquisition unit 202 includes a hospital information system or an electronic health / medical record system 208, an independent healthcare facility data storage system 210, and at least one ultrasound system 212. In operation, a patient order is created in the independent hospital information system or an electronic health / medical record 208 and sent to the independent healthcare facility data storage system 210 as an HL7 message, shown as reference number 1. The independent healthcare facility data storage system 210 converts the HL7 message to a DICOM MWL and sends it to the ultrasound system 212. A HCP at the ultrasound system 212 then selects the patient order, shown as reference number 2. The independent healthcare facility data storage system 210 also sends the modality worklist (e.g., DICOM, MWL) to the data processing and display unit 204 which displays it on an edge computing device 214 in the imaging mode 102, a workstation 216 in the reporting mode 104, and on a reading room workstation 218 in the reading mode 106.

[0056] A user may then select the patient’s ultrasound examination images, shown as reference number 3, and send them from the ultrasound system 212 to the independent healthcare facility data storage system 210 in DICOM format, shown as reference number 8.

[0057] The data processing and display unit 204 includes an edge computing device 214 operating the imaging mode 102, the reporting mode 204 operating on the workstation 216, the reading mode operating on the reading room workstation 218, and a local server 220.

[0058] In operation, ultrasound examination images from the ultrasound system 212 are conveyed to the edge computing device 214, shown as reference number 4, which sends the ultrasound examination images to the local server 220 for Al analysis, shown as reference number 5. A backend system 222 of the local server 220 routes the ultrasound examination images to an Al engine 224 of the local server 220, which analyzes the images and gives back predictions, for example, diagnostic views, anatomical measurements, and annotations, which are displayed on the edge computing device 214 in the imaging mode 102, shown as reference number 6.

[0059] The local server 220 fetches the reviewed and modified examination findings from the imaging mode 102 of the edge computing device 214, and retrieves the patient data from the independent hospital information system or an electronic health / medical record 208 in HL7 format and displays the Al predictions, shown as reference number 9, and the patient data shown as reference number 10, on the workstation 216 in the reporting mode 104. The local server 220 also operates to send images, video loops, and reports prepared on the workstation 216 in the Reporting Mode 104 to the reading Mode 106 operating on the reading room workstation 218.

[0060] The data reception and storage unit 206 includes the independent hospital information system or an electronic health / medical record 208, and independent healthcare facility data storage system 210 which are present in the healthcare facility setting. In operation, ultrasound images and video loops of the HCP's choice are sent to the independent healthcare facility data storage system 210 in DICOM format, shown as reference number 12. The patient report is sent to the independent hospital information system or an electronic health / medical record system in HL7 format, shown as reference number 13, and is then sent to the independent healthcare facility data storage system 210 in HL7 format for final storage, shown as reference number 14.

[0061] Figure 3 depicts an exemplary system software architecture of the disclosed embodiments. The software architecture illustrates components and data flows for the imaging mode 102, the reporting mode 104, and the reading mode 106, as well as components and data flows for the local server 220. While proprietary names for the hardware and software components are mentioned, it should be understood that the proprietary names are exemplary, and that any software or hardware component may be utilized that is suitable for performing the functions described.Components and Data Flows for the Imaging Mode 102:

[0062] The edge computing devices 214 may be placed in the examination room proximate to the ultrasound system 212 and serve as a data source for the local server 220. Exemplary hardware for the edge computing devices 214 may include a processor, such as an Intel i5 10th generation or later, RAM of at least 8GB, storage of at least 256GB, a touch screen display, and a wired or wireless interface to an ultrasound system 212 and the local server 220. The edge computing devices 214 may utilize a Windows operating system and may operate a web browser, for example, Microsoft Edge, Google Chrome, Mozilla Firefox, or Apple Safari.

[0063] An HCP will utilize the imaging mode 102 on the edge computing devices 214 while performing the ultrasound examination . The edge computing devices 214 operate to fetch the imaging frames in electronic format directly from the ultrasound systems and send them to the local server 220. Along with the data fetching process, the edge devices also serve as the front-end for the imaging mode 102, where Al-assisted findings are displayed to the HCP performing the scan, and operate to capture interactions of the HCP in the imaging mode and send them back to the local server 220.

[0064] Figure 3 illustrates the following system components and data flows present in the healthcare facility examination room:[1.1] Patient data, including patient-specific data such as patient ID, history, and demographics;[1 .2] Communication protocols, including web-socket communication used by the edge computing devices 214 for fetching the ultrasound stream and real-time Al predictions from the local server 220. An exemplary API and RPC may include a REST or RESTful API used for fetching patient data, Al predictions, and other software-related information;[1 .3] A process that may be implemented to fetch data from the ultrasound systems 212, may be, for example, a Python executable (.exe) agent;[1 .4] Pre-stored data refers to data already stored in the healthcare facility independent healthcare facility data storage system, and may include previously acquired scan images, examination reports, patient data, etc., reviewed and sent by an HCP in DICOM format either through the reporting mode 104 or reading mode 106, or without the use of the system.Components and Data Flows for the Local Server 220:

[0065] Exemplary hardware for the local server 220 may include a processor, such as an Intel® Xeon® Silver 421 OR or better, RAM of at least 32GB, storage of at least 2TB, a graphics card, for example, an Nvidia RTX 2060 or better, and a wired or wireless interface providing network read write and DICOM read write access to the independent healthcare facility data storage system 210, and network read write access to the independent hospital information system or an electronic health / medical record . The local server 220 may also have a storage server for retaining audit and application logs for asset period, and a maintenance server with redundancy if the local server 220 fails. The local server 220 may operate a web server, for example, Nginx 1 .24 or later, a database, such as MySQL Enterprise Edition or Postgres16, or better, and a containerization function, such as Kubernetes 1 .28 or later.

[0066] The local server 220 is generally responsible for all communications between the healthcare facility and the system 100 for analyzing ultrasound images, including communication with the independent healthcare facility data storage system, independent hospital information system or an electronic health / medical record, and other healthcare facility systems. The local server 220 also processes the frames sent by the edge computing device 214, serves as a front end for the imaging mode 102, the reporting mode 104, and the reading mode 106, and maintains and stores access and audit logs that are necessary for updating and maintaining the various system software components. The local server 220 also operates a temporary database for storing the data for processing the ultrasound images and clinical findings. It should be understood that the temporary database operates to self-destroy all date after analysis of the data is complete.

[0067] Figure 3 also illustrates the following system components and data flows present in the local server 220:[2.1] A Kubernetes cluster enables running containerized applications, orchestrates containers, handles failures, scales services, and manages workload distributions, and is used for the optimum allocation of resources(scaling), automation, and management of processes among different healthcare facilities;[2.2] Client apps connect to the local server backend through APIs, enabling users to access, input, and interact with data;[2.3] A load balancer ensures that user requests are distributed efficiently to different server resources, prevents any single resource from being a bottleneck, and ensures high availability;[2.4] An ingress gateway manages external access to the various local server services, such as by routing incoming requests to the correct services inside the Kubernetes cluster;[2.5] A local container registry operates as a storage solution for DICOM images which are deployed as containers within the Kubernetes cluster;[2.6] A HELM tool that simplifies deployment of applications in the Kubernetes cluster by using “charts” packages;[2.7] An update and patch agent, which may be a healthcare facility admin, required to perform software and configuration updates;[2.8] A Keycloak identity server used to authenticate and authorize users into the system 100. It ensures that only authorized individuals can access specific resources or data.[2.9] The backend services are responsible for data processing, business logic, and interfacing with databases, and for handling requests from client apps and return responses. The backend server is used to host the frontend for the imaging 102, reporting 104, and reading 106 modes, used for orchestration and management of the Al engine, used for storing and retrieving data from the database, and operates as a point of communication between the system 100 and the healthcare facility.[2.10] The utility services include a collection of tools and platforms supporting various system tasks, for example, an ELK stack used for logging, monitoring, and storing software log data for debugging and cybersecurity audit purposes, aSpark engine for performing big data processing tasks, a Kafka event streaming platform acting as a messaging broker and for building real-time data pipelines, an Airflow platform for managing data workflows and batch processing tasks, and an HBase database management system for storing distributed data at a large scale;[2.11 ] The data stores house medical records, images, logs, application data, and other suitable data, and may include a MySQL relational database for storing ultrasound images and patient data only for the duration of examination, and a MinlO object storage for storing medical images only for the duration of examination[2.12] an Al Engine composed of a series of Al models used for the interpretation of ultrasound images.Components and Data Flows for the Reporting Mode 104 and Reading Mode 106:

[0068] The Al predictions in the imaging mode 104 and all the images saved on the ultrasound system 212 are made available at the workstation 216 via the reporting mode 104 which may operate to produce an editable pre-filled preliminary report based on the imaging mode findings from the local server 220. A user may make necessary modifications and prepare the preliminary report, and can then share the report with an HCP through the reading mode 106 for review and sign-off.

[0069] The preliminary report prepared by the HCP may be made available to the HCP through the reading mode 106 and then may be sent to the independent healthcare facility data storage system or independent hospital information system or an electronic health / medical record systems based on the healthcare facility system requirements.

[0070] Exemplary hardware for a workstation 216 running the reporting mode 104, and for a reading room workstation 218 running the reading mode 106 may include a processor, such as an Intel i5 10th generation or later, RAM of at least 8GB, storage of at least 256 GB, a display with a resolution of 1024x768 or better, and a wired or wireless interface to the local server 220. The workstation 216 and reading room workstation mayutilize a Windows operating system and may operate a web browser, for example, Microsoft Edge, Google Chrome, Mozilla Firefox, or Apple Safari.

[0071] A more detailed diagram of the data flow 400 of the imaging mode 102 is shown in Figure 4.

[0072] Referring to block 1 .0 of Figure 4, an administrator may define and configure protocols that will be used during the imaging mode 102, and therefore establishes the protocol and standards for the Al engine 224 to interpret ultrasound images during imaging. As shown in block 2.0, the imaging mode 102 requests the protocol specifications configured in step 1 .0. The backend system 222 provides the specifications to the imaging mode 102 which uses them to guide the imaging process, ensuring all necessary steps and standards are followed according to predefined protocols. As shown in block 3.0, the imaging mode 102 requests the patient's medical information relevant to the imaging process from the backend system 222, which may include a patient identification number, medical history, prior imaging results, and other pertinent clinical information.

[0073] As shown in block 3.1 , upon the imaging mode’s request, the backend system 222 fetches the patient information from the independent hospital information system or an electronic health / medical record 208 and as shown in block 3.2, stores this data into the local server database for the examination . As shown in block 3.3, the backend system 222 then sends the patient information, specific to the current examination, to the imaging mode for viewing by the HCP . As shown in block 4.0, once the HCP starts the examination in the imaging mode 102, the ultrasound system 212 starts capturing the imaging frames for analysis and review. The imaging mode 102 sends the captured frames to the backend system 22 as shown in block 5.0 where they are stored until completion of the examination .

[0074] As shown in block 5.1 , the backend system 222 preprocesses the frames received from the imaging mode 102, which may include resizing, noise reduction, contrast enhancement, or other image quality improvements to ensure that the Al algorithms can effectively analyze the frames. After the completion of preprocessing, the frames are sent to the Al engine for detection of diagnostic views / planes, identification ofanatomical structures, automated measurements and annotations of structures on these views / planes. As shown in block 5.2, the Al engine predictions are verified, and stored in the database by the backend system for use in reporting.

[0075] During the examination, the imaging mode 102 requests the predictions from the Al engine 224, as shown in block 6.0, and the backend system fetches the predictions from the database as shown in block 6.1 , and the annotations as shown in block 6.2, and sends them to the imaging mode 102. This allows the HCP to review Al- generated predictions and integrate them with their expertise. As shown in block 7.0, the HCP may edit or update the annotations based on their review, which ensures that the final output of the imaging process is a result of both Al capabilities and human expertise, enhancing accuracy and reliability.

[0076] A detailed description of the features and exemplary user interfaces for implementing the features of the imaging mode begins with Figure 5.

[0077] Figure 5 depicts a login feature that requires entry of a user’s user name and password, and based on the access provided in the administration settings, the user may view one or more of the imaging, reporting, and reading modes of operation.

[0078] Figures 6 and 7 show a setup feature having a user interface with a I clinical information panel showing options for choosing a workflow (i.e., the predefined set of anatomical views and structures to be analyzed by the device during the examination ), as shown in Figure 6 and a list of available fetal / obstetric views and structures for examination in Figure 7, where a user needing customization may choose the views and structures for analysis for that particular patient under consideration.

[0079] A dashboard feature is depicted in Figures 8 and 9 that allows a user to select a patient from the patient modality worklist fetched from independent hospital information system or an electronic health / medical record or independent healthcare facility data storage system, and select the patient ID and other associated information in the patient order, as shown in Figure 8, and upon selection of the patient with other associated information (for example, LMP, First name, Last name etc.), growth parameters like Estimated Delivery Date (EDD) and Gestational Age (GA) would be calculated by the software and displayed, as shown in Figure 9.

[0080] Figure 10 illustrates a quality criteria feature that contains the criteria the images / frames of the corresponding diagnostic view is assessed on. The quality criteria of the views and structures may be based on AIUM and ISIIOG guidelines. This feature is designed to automatically identify the fulfillment of quality criteria within a given image, as determined by clinical guidelines, and mark them off on a checklist. For example, based on clinical guidelines, a midsagittal plane of the fetus should include the presence of the thalami, brain stem, intracranial translucency, cisterna magna, and magnification to verify the quality criteria of the view or structure. This feature advantageously aids the Al engine in predicting and measuring a desired plane or view, according to the clinical guidelines (AIUM and ISUOG).

[0081] An automated anatomical survey feature is shown in Figure 11 which identifies and keeps track of the obstetric / fetal anatomies of whose, structures are seen in the views as per ISUOG and AIUM guidelines regarding key structures. Similar to the capture affirmation process, the user has the option to customize the anatomical structures of focus by deselecting / selecting structures from a full list of potential structures of that diagnostic view / plane.

[0082] Figure 12 depicts an automated caliper placements and measurements feature that automatically places caliper points in consideration of ISUOG and AIUM guidelines. The feature includes user options to override the automated caliper placements and measurements and to manually modify the caliper placements and measurements.

[0083] A video player feature is portrayed in Figure 13 that allows a user to view an entire scan video of a particular diagnostic view / plane and be able to select other frames of the same view / plane for Al or manual manipulations. The feature includes a ‘play / pause’ button for playing and pausing the video and a ‘loop’ button that plays the video in a loop as shown in Figure 14, and provides a summary view as shown in Figure 15.

[0084] Figures 16 and 17 illustrate an information tool feature that may provide information on imaging techniques and landmarks in the event of an indication that a structure is not visualized, an unavailability of optimal images, or a deviation inmeasurements. The information is entered and set in the administrator settings according to the clinical practices.

[0085] A summarized findings feature is shown in Figure 18, that provides a summary of the imaging mode, and allows a user to customize the formula and method for obtaining relevant measurements at the end of the examination .

[0086] Figure 19 depicts a user logout feature that provides an ‘End Exam’ button that logs a user out when activated.

[0087] A more detailed diagram of the data flow 2000 of the reporting mode 104 is portrayed in Figure 20. The data flow of the reporting mode 104 may begin with a HCP retrieving the frames along with their associated annotations that may include predictions of anatomies, measurements, or other relevant notes from the imaging mode, as shown in block 1 .0. At the workstation 216, the HCP may review the frames and their annotations as illustrated in block 2.0. This review process operates to validate the frames' accuracy and the annotations' correctness before a report is generated. As shown in block 2.1 , based on the reviewed frames and annotations, the predictions may need to be updated. These predictions could be diagnostic conclusions or other relevant findings that result from the analysis of the imaging data.

[0088] Block 2.2 depicts a notification to a user who is using the reading room workstation 218 upon completion of the prediction update, that the user may proceed with further clinical actions or decision-making processes. A preliminary report is prepared as portrayed in block 3, after a review and any required updates to the predictions. The report may compile all relevant information, including the original images, annotations, and any updated predictions post-review. Once the preliminary report is ready, the backend system 222 may notify an HCP at the reading room workstation 218, as shown in block 3.1 , indicating that the preliminary report is now available for further review or for implementing care pathways.

[0089] Figures 21 through 37 illustrate detailed descriptions of the features and exemplary user interfaces for implementing the features of the reporting mode 104.

[0090] A login feature is shown in Figures 21 through 23 that includes a login page, shown in Figure 21 , that allows a user to enter their login credentials. Figure 22 depicts a dashboard that allows the user to select a patient from the patient modality worklist fetched from independent hospital information system or an electronic health / medical record or independent healthcare facility data storage system, and select the patient ID and other associated information in the patient order, for which to prepare a report, and search for prior examination images stored in the system, for example, using an “Open Patient Record” button. The “Open Patient Record” button produces a Home Screen that includes the patient data and scan records, and allows the user to build reports, as depicted in Figure 23.

[0091] A pre-filled report template is portrayed in Figure 24, which is pre-filled from information gathered in the imaging mode 102. A user may modify the findings or record their observations in the summary, in addition to the software's findings and the findings obtained in electronic format directly from the ultrasound system, and may view and add the indication of the scan and history to the patient information. As illustrated in Figures 24 through 26, the user may also view, edit, and add information to the report.

[0092] The user also has an editable text window that has the summary of the visualization states of all the structures and anatomies in a text box, as depicted in Figure 27. As depicted in Figure 28, the user can select pre-entered and customizable impressions for the scan or examination . The user may also customize and add additional impressions when appropriate. The user may confirm their consent or agreement with the report details and findings by typing in their name. The user can indicate to the software that the necessary actions and steps have been completed to compile, format, and build the report, as portrayed in Figure 29.

[0093] An image view feature, shown in Figure 30, aids a user in viewing and selecting the necessary images from the image grids of the examination, captured for all the anatomy mentioned above, and categorized on the views and planes of that anatomy. Users have the option to modify the results of the automated anatomical survey and measurements. Users will also have the option to select the images that they wish to send to the clipboard for report generation, as depicted in Figure 31 .

[0094] Figure 32 shows a growth charts display feature, where a user may view and modify growth charts according to their clinical practices, while Figure 33 depicts a cine video viewing feature, where a user may view a video of an entire scan, along with videos that provide the user with an option to choose a section of anatomy in the whole video, when desired, and send the resulting chosen video section to archive.

[0095] A preview report feature is illustrated in Figures 34 and 35 that allows a user to preview a report based on sent images and graphs by placing the images and graphs in the preview report from the clipboard. The user may add the images and graphs to the report, as depicted in Figure 34 remove the images and graphs as desired, as depicted in Figure 35, and also modify the formatting of the report (for example, adding page breaks, modifying alignment, etc.).

[0096] Figures 36 and 37 illustrate a sample report generated by the software with summarized findings, examination images, and graphs. Visually, this is the final report copy that is transferred to the healthcare facility data storage system (independent healthcare facility data storage system or independent hospital information system or an electronic health / medical record ) or printed out as a hard copy.

[0097] A more detailed diagram of the data flow 3800 of the reading mode 106 is portrayed in Figure 38.

[0098] The reading mode 106 may initiate in an admin interface where protocols are configured, as illustrated in block 1 .0. Referring to block 2.0, the protocols along with the specifications are retrieved and the reading HCP is notified about the protocol specifications used by the HCP to assess the patient during the exam, ensuring that the review process is dynamic and informed by the most up-to-date standards or requirements. Once the preliminary report is prepared by the HCP on the workstation, the reading HCP in the reading room will be notified by the backend system, as shown in block 3.0. Upon notification, the reading HCP will request and fetch the preliminary report, the corresponding frames, and predictions / annotations from the database, as depicted in block 4.0. These frames represent the images or data that need to be reviewed, and annotated to provide additional context or preliminary assessments.

[0099] The reading HCP professional may then start preparing a final report based on the reviewed frames and annotations, as shown in block 5.0. As shown in block 5.1 , the preliminary report may be fetched for further review or comparison with the current findings. Any necessary edits may be made to the report, reflecting changes based on the live review process, additional insights from the reading HCP, or corrections and updates to the preliminary findings, as portrayed in block 5.2.

[0100] As shown in block 5.3, the workstation 216 is notified about the edited report, which triggers further processes, for example, finalizing and storing the report in a database. Frames and annotations that have been reviewed and potentially updated through the process are now acknowledged as 'Reviewed,' as illustrated in block 6.0, and the workstation is notified, indicating the review process for those specific frames and annotations is completed, as portrayed in block 6.1 .

[0101] The final step in the review process involves the signing and printing of the report, as depicted in block 7.0, which validates the review process and prepares the report for distribution and archiving, after which the reviewed frames and associated predictions or annotations are sent to the independent healthcare facility data storage system 210 in a DICOM format as depicted in block 8.0.

[0102] Figure 39 shows a network architecture typical for the system 100 installed in a healthcare facility. The system 100 runs locally and offline without internet access on dedicated healthcare facility-owned computing equipment within the healthcare facility’s network protected by its firewall. No data leaves the internal domain of the healthcare system.

[0103] The disclosed system for analyzing ultrasound images is advantageously capable of timely and secure access and interoperable use of electronic health data and may generally result in optimized health outcomes for individuals and populations. The system is configured to seamlessly integrate with existing healthcare systems and workflows, and to provide a harmonious and efficient environment for HCP’s .

[0104] The system is further configured to seamlessly integrate with independent hospital information system or an electronic health / medical record systems commonlyused in healthcare institutions, allowing for secure exchange of patient data and diagnostic results via HL7 standards.

[0105] The system operates to automatically update patient records, ensuring that all ultrasound diagnostic data is accurately and securely stored within the healthcare facility's independent hospital information system or an electronic health / medical record system.

[0106] The system is still further configured to seamlessly integrate with independent healthcare facility data storage system to store, manage, and retrieve ultrasound images and diagnostic reports, and ensures that all ultrasound images are centrally stored, making it easier for the healthcare system to access images from any location within the healthcare system network.

[0107] Reading HCPs in the reading room can access ultrasound images through the independent healthcare facility data storage system system, making it possible to review and provide immediate consultations to HCP s.

[0108] The system also offers RESTful APIs, allowing for open integration with other healthcare systems and third-party applications by providing a standardized communication and data exchange, and providing healthcare facilities with the ability to develop third-party custom solutions and applications that interface with the system, providing opportunities for innovation and tailored applications.

[0109] Various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings. However, all such and similar modifications of the teachings of the disclosed embodiments will still fall within the scope of the disclosed embodiments.

[0110] Various features of the different embodiments described herein are interchangeable, one with the other. The various described features, as well as any known equivalents can be mixed and matched to construct additional embodiments and techniques in accordance with the principles of this disclosure.

[0111] Furthermore, some of the features of the exemplary embodiments could be used to advantage without the corresponding use of other features. As such, the foregoing description should be considered as merely illustrative of the principles of the disclosed embodiments and not in limitation thereof.

Claims

Claims1 . A system for analyzing ultrasound images comprising: an imaging mode of operation employed in an edge computing device in a healthcare facility ultrasound examination room to assist in analysis of ultrasound images and examination findings obtained in electronic format directly from an ultrasound system; a reporting mode of operation employed in a healthcare facility workstation for generating reports based on the imaging mode’s analysis of the ultrasound images and examination findings obtained in electronic format directly from the ultrasound system; and a reading mode of operation employed in a healthcare reading room workstation for providing a healthcare professional (HCP) with the ability to view the ultrasound images and examination findings obtained in electronic format directly from an ultrasound system, view the imaging mode’s analysis of the ultrasound images and examination findings, and using the ultrasound images and examination findings obtained in electronic format directly from an ultrasound system, and the imaging mode’s analysis of the ultrasound images and examination findings, to finalize the reports from the reporting mode to be sent to a healthcare facility data storage system; wherein the healthcare facility data storage system includes one or more of an independent hospital information system, an electronic health / medical record system, or an independent healthcare facility data storage system.

2. A system for analyzing ultrasound images comprising: an edge computing device having a display and a processor coupled to an ultrasound system; anda local server coupled to the edge computing device, a reporting mode workstation, and a reading room workstation, the local server comprising an artificial intelligence engine; wherein the edge computing device is configured to: send ultrasound examination images and examination findings obtained in electronic format directly from the ultrasound system to the local server for analysis by the artificial intelligence engine; retrieve analyses of the ultrasound examination images and examination findings generated by the artificial intelligence engine, wherein the analyses include diagnostic views, anatomical measurements, and annotations; and display the analyses for review and editing, wherein the local server is configured to provide the analyses to the reporting mode workstation; wherein the reporting mode workstation is configured to operate a reporting mode for generating reports based on the Al engine’s analyses; wherein the local server is configured to provide the ultrasound images and examination findings obtained in electronic format directly from an ultrasound system, the analyses of the ultrasound examination images and examination findings generated by the artificial intelligence engine, and the reports to the reading room workstation; wherein the reading room workstation is configured to operate a reading mode for allowing an HCP to view the ultrasound images and examination findings obtained in electronic format directly from an ultrasound system, the analyses of the ultrasound examination images and examination findings generated by the artificial intelligence engine, and the reports, and to finalize the reports, and wherein the local server is configured to send the ultrasound images and examination findings obtained in electronic format directly from an ultrasound system, the analyses of the ultrasound examination images and examination findings generated by the artificial intelligence engine, the reports, and the finalized reports, to a healthcare facility data storage system.

3. A method for analyzing ultrasound images comprising: requesting a patient’s medical information from one or more of an independent Electronic Health Record, independent hospital information system, or an electronic health / medical record system for ultrasound imaging; using an ultrasound system to capture ultrasound image frames of the patient during an ultrasound examination; pre-processing the ultrasound image frames for analysis including one or more of resizing, noise reduction, and contrast enhancement; using an artificial intelligence engine to: automatically analyze the ultrasound image frames to detect diagnostic views, provide anatomical measurements, and annotations; and automatically validate the diagnostic views, anatomical measurements, and annotations; reviewing and editing the diagnostic views, anatomical measurements, and annotations by a HCP; and storing the patient’s ultrasound image frames, diagnostic views, anatomical measurements, annotations, and reviewed and edited diagnostic views, anatomical measurements, and annotations, in one or more of the independent hospital information system or an electronic health / medical record system, an independent Picture Archival and Communication System, or an independent healthcare facility data storage system.

Images