System and method for providing AI-assisted checklist for interventional medical procedures

Abstract

An imaging system (1300) and method includes a display (1324), a processor (1312), and a memory (1328) storing processor executable code that causes: to receive an image (1701) of a patient having a medical device (1705) disposed within the patient; detecting the medical device (1705) within the image (1701); detecting one or more reference markers (1709) within the image (1701); and generating a combined image (1702) to indicate a position (1707) of the medical device (1705) relative to the reference mark (1709) by superimposing a stack (1703) on the image (1701). The processor executable code also causes the image processing system (1300) to generate a checklist (1760) for a user to verify the location of the medical device (1705) for presentation on the display (1324) along with the combined image (1702) to assist the user in complying with clinical guidelines for placing the device (1705). Further, the processor executable code may pre-populate the checklist (1760) with checklist responses using an artificial intelligence model (220), the checklist responses for training or verifying the AI model (220) or for quality assurance purposes.

Description

Technical Field

[0001] The subject matter disclosed herein relates to medical image processing, and more particularly, to systems and methods for visualizing the placement of medical tubes or wires. Background Art

[0002] Medical imaging can be used to visualize medically placed tubes or lines (e.g., chest tubes, nasogastric tubes, endotracheal tubes, vascular lines, peripherally inserted central catheters (PICCs), catheters, etc.). However, it may be difficult for medical personnel (e.g., doctors, radiologists, technicians, etc.) to visualize these medically placed tubes or lines. In addition, medical personnel may be untrained or inexperienced, which may hinder their ability to identify medically placed tubes or lines and determine whether they are correctly placed. In addition, medical personnel may have to manually assess typical complications associated with tube / line placement and manually take measurements (which may be time consuming) to determine whether the medically placed tubes or lines are correctly placed. However, if the medically placed tubes or lines are misplaced, rapid intervention is required to move the tubes or lines to the appropriate position to obtain patient safety.

[0003] To assist with visualization and tube placement within a patient, certain systems and methods have been developed, such as those disclosed in U.S. Patent No. 11,410,341 (the '341 patent), entitled System And Method For Visualizing Placement of Medical Tube or Line, and U.S. Patent Application Serial No. 63 / 427,646, entitled Artificial Intelligence System and Method for Defining and Visualizing Placement of Catheter in a Patient Coordinate System Together with an Assessment of Typical Complications, the entire contents of each of which are expressly incorporated herein by reference for all purposes. In the '341 patent, an artificial intelligence (AI) is trained to function as part of an image processing system that includes a display, a processor, and a memory. The memory stores processor executable code for a trained AI, which, when executed by the processor, causes: receiving an image of a region of interest of a patient, wherein a medical tube (i.e., an endotracheal tube (ETT) or a nasogastric tube (NGT) or a peripherally inserted central catheter (PICC) line) is disposed within the region of interest; detecting the medical tube or line within the image; generating a combined image by superimposing a first graphic marker on the image indicating an end of the medical tube or line and a reference point of the patient's anatomical structure; and displaying the combined image on a display.

[0004] In the '646 application, in addition to displaying inserted medical devices and anatomical references on an image of a patient and reference calculations for device placement with respect to the anatomical references, the AI ​​systems and methods disclosed herein provide indications of certain types of devices and anatomical complications detected by the system within the displayed image. Thus, the '646 application not only provides images of the patient's anatomy and devices located within the anatomy, but also identifies the presence or absence of certain complications detected by the AI ​​systems and methods.

[0005] While the references mentioned above disclose systems and methods including AI for identifying and assessing various aspects of a patient's medical images, there remains the problem of enabling a user / reviewer to provide information regarding an initial assessment of an image for subsequent review of the image. The prior references mentioned above emphasize the device and associated landmarks, but the reviewer's assessment of correctness of placement may still be a nontrivial task. Furthermore, if the AI ​​system's determination as to whether the tube is correctly placed is typically a "black box" decision, where no information is provided to the reviewer / user to determine how the AI ​​reached this conclusion.

[0006] For example, in order to enable subsequent reviewers of the image to be quickly guided to the important aspects of the image, in many cases, the initial reviewer can provide an initial assessment in the form of annotations to the image, which identify the important aspects of the image for the subsequent reviewer. However, the process of reviewing and / or assessing the image, identifying the important aspects of the image (whether or not assisted by the AI ​​system and method), and then annotating the image to guide the subsequent reviewer to those important aspects is a time-consuming process. Specifically, it is often the case that the initial reviewer must browse a large number of project checklists to conduct a complete annotated review or assessment of a single image. In particular, when the initial reviewer has limited experience and / or when the initial reviewer is uncertain about the correct answer to the checklist item, it takes a lot of time to determine the response to each item on the checklist. In addition, before the reviewer concludes that the device is correctly placed, multiple aspects of the device must be verified in the checklist. In addition, the process of providing responses to the checklist items may be prone to errors caused by incorrect and / or negligent selection of responses to one or more items in the checklist items.

[0007] Thus, while checklists enable images to be annotated in a standardized manner to facilitate subsequent review of the images, the time required to complete the checklist and the potential for erroneous responses make the checklist process performed by the initial reviewer problematic.

[0008] Therefore, in order to enable the initial physician / user to more efficiently and accurately provide the necessary information regarding the initial assessment of medical images to subsequent reviewers, it is desirable to provide an AI-assisted image review system and method for completing an initial image review or assessment checklist. Summary of the invention

[0009] According to one aspect of an exemplary embodiment of the present disclosure, an image processing system includes a display, a processor, and a memory storing processor executable code, which, when executed by the processor, causes: receiving an image of a region of interest of a patient, wherein a medical tube or wire is disposed within the region of interest; detecting the medical tube or wire within the image; detecting one or more reference markers within the region of interest within the image, wherein the one or more reference markers are within the patient's body; generating a combined image by superimposing an overlay on the image to indicate a position of the medical tube or wire relative to the one or more reference markers; generating a checklist for verifying the position of the medical tube or wire relative to the one or more reference markers; and presenting the combined image and the checklist on the display.

[0010] According to another aspect of an exemplary embodiment of the present disclosure, a method for medical image processing includes the following steps: receiving, via a processor, an image of a region of interest of a patient, wherein a medical tube or wire is disposed within the region of interest; detecting, via the processor, the medical tube or wire within the image; detecting, via the processor, a plurality of reference markers within the region of interest within the image, wherein each of the plurality of reference markers is within the patient's body; generating, via the processor, a combined image by superimposing an overlay on the image to indicate a position of the medical tube or wire relative to one or more reference markers; generating, via the processor, a checklist for verifying the position of the medical tube or wire relative to the one or more reference markers; and causing, via the processor, the combined image and the checklist to be presented on a display.

[0011] According to another aspect of an exemplary embodiment of the present disclosure, an imaging system includes: a radiation source; a detector capable of receiving radiation emitted from the radiation source to form image data; a display for presenting information to a user; and a controller connected to the display and operable to control the operation of the radiation source and the detector to generate an image from the image data, the controller including an image processing system having a processor and a memory storing processor executable code, the processor executable code when executed by the processor causing: receiving an image of a region of interest of a patient, wherein a medical catheter, tube or line is disposed within the region of interest; detecting the medical tube or line within the image; detecting one or more reference markers within the region of interest within the image, wherein the one or more reference markers are within the patient; generating a combined image by superimposing an overlay on the image to indicate a position of the medical tube or line relative to the one or more reference markers; selecting multiple steps to form a checklist for verifying the position of the medical tube or line relative to the one or more reference markers; and displaying the combined image and the checklist on the display.

[0012] These and other exemplary aspects, features and advantages of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings illustrate the best mode presently contemplated of practicing the invention.

[0014] In the attached picture:

[0015] Figure 1 is a schematic diagram of a status comparator according to an exemplary embodiment of the present disclosure.

[0016] Figure 2 is a schematic diagram of an embodiment of a clinical progress analysis device according to an exemplary embodiment of the present disclosure.

[0017] Figure 3 is a schematic diagram of an embodiment of a learning neural network according to an exemplary embodiment of the present disclosure.

[0018] Figure 4 is a schematic diagram of an embodiment of an image processing system configured to execute example machine readable instructions to implement components disclosed and described herein, according to an exemplary embodiment of the present disclosure.

[0019] Figure 5 is a flow chart of an embodiment of a method for determining placement of a medically placed tube or wire within a region of interest according to an exemplary embodiment of the present disclosure.

[0020] Figure 6 is a first example of a combined image identifying a catheter within a patient according to an exemplary embodiment of the present disclosure.

[0021] Figure 7 is a second example of a combined image identifying a catheter within a patient according to an exemplary embodiment of the present disclosure.

[0022] Figure 8 is a first schematic diagram of a user interface according to an exemplary embodiment of the present disclosure, the user interface having a combined image identifying a tube or line within a patient and an AI system-generated checklist to be completed by a user to verify proper placement of the medically placed tube or line within a region of interest.

[0023] Fig. 9 is a flow chart of an embodiment of a method for providing an AI system-generated checklist to be completed by a user to verify correct placement of a medically placed tube or line within a region of interest in accordance with an exemplary embodiment of the present disclosure.

[0024] Fig.10 is a method for utilizing an exemplary embodiment of the present disclosure Fig. 9Schematic diagram of the federated learning process for training an AI system that generates checklists.

[0025] Fig.11 is an AI system for generating a checklist in combination with an exemplary embodiment of the present disclosure Fig. 9 Block diagram of an imaging system of the method. DETAILED DESCRIPTION

[0026] One or more specific embodiments will be described below. In order to provide a concise description of these embodiments, all features of an actual implementation may not be described in the specification. It should be understood that, as in any engineering or design project, in the development of any such actual implementation, numerous implementation-specific decisions must be made to achieve the developer's specific goals, such as complying with system-related and business-related constraints that may differ between implementations. In addition, it should be understood that such development efforts may be complex and time-consuming, but are still routine tasks of design, fabrication, and manufacturing for ordinary technicians who benefit from this disclosure.

[0027] When introducing elements of various embodiments of the present subject matter, the articles "a," "an," "the," and "said" are intended to indicate that there are one or more elements. The terms "comprising," "including," and "having" are intended to be inclusive, and mean that there may be additional elements in addition to the listed elements. Furthermore, any numerical examples in the following discussion are intended to be non-limiting, and thus the additional numerical values, ranges, and percentages are within the scope of the disclosed embodiments.

[0028] Imaging devices (e.g., gamma cameras, positron emission tomography (PET) scanners, computed tomography (CT) scanners, X-ray machines, fluoroscopy machines, magnetic resonance (MR) imaging machines, ultrasound scanners, etc.) generate medical images (e.g., raw Digital Imaging and Communications in Medicine (DICOM) images) representing body parts (e.g., organs, tissues, etc.) to diagnose and / or treat diseases. Medical images may include volume data, which includes voxels associated with body parts captured in the medical image. Medical image visualization software allows clinicians to segment, annotate, measure, and / or report functional or anatomical properties at various locations of the medical image. In some examples, clinicians may use medical image visualization software to identify regions of interest in medical images.

[0029] The acquisition, processing, quality control, analysis, and storage of medical image data play an important role in the diagnosis and treatment of patients in a healthcare setting. Medical imaging workflows and the devices involved in the workflows can be configured, monitored, and updated throughout the operation of the medical imaging workflows and devices. Machine and / or deep learning can be used to help configure, monitor, and update medical imaging workflows and devices.

[0030] Certain examples provide and / or facilitate improved imaging devices, thereby improving diagnostic accuracy and / or coverage. Certain examples facilitate improved image reconstruction and further processing, thereby improving diagnostic accuracy.

[0031] Certain examples provide an image processing device including an artificial intelligence system (AI system). For example, the AI ​​system can detect, segment, and quantify lesions. The AI ​​system can be a positive or negative discrete output for finding, segmenting, etc. For example, the AI ​​system can instantiate machine learning and / or other artificial intelligence to detect, segment, and analyze the presence of a medical device (e.g., a medically placed tube or wire). For example, the AI ​​system can instantiate machine learning and / or other artificial intelligence to detect the end of a medically placed tube or wire, detect a reference mark or anatomical mark, determine the position of the medically placed tube or wire relative to the reference mark or anatomical mark, measure the distance between the end of the medically placed tube or wire and the reference mark, and determine whether the tube or wire is correctly placed.

[0032] For example, machine learning techniques (whether deep learning networks or other experiential / observational learning systems) can be used to locate objects in images, understand speech and convert speech to text, and improve the relevance of search engine results. Deep learning is a subset of machine learning that uses a set of algorithms to model high-level abstractions in data using deep graphs with multiple processing layers (including linear and nonlinear transformations). Although many machine learning systems are implanted with initial features and / or network weights to be modified through learning and updating of the machine learning network, deep learning networks train themselves to identify "good" features for analysis. When using a multi-layer architecture, machines using deep learning techniques can process raw data better than machines using conventional machine learning techniques. The use of different layers of evaluation or abstraction facilitates data inspection of groups of highly relevant values ​​or distinctive topics.

[0033] Throughout this specification and claims, the following terms take the meanings explicitly associated with this document unless the context clearly indicates otherwise. The term "deep learning" is a machine learning technique that utilizes multiple data processing layers to identify various structures in data sets and classify these data sets with high accuracy. A deep learning network can be a training network (e.g., a training network model or device) that learns patterns based on multiple inputs and outputs. A deep learning network can be a deployed network (e.g., a deployed network model or device) that is generated by a training network and provides outputs in response to inputs.

[0034] The term "supervised learning" is a deep learning training method in which a machine is provided with classified data from a human source. The term "unsupervised learning" is a deep learning training method in which a machine is not given classified data but is made available for anomaly detection. The term "semi-supervised learning" is a deep learning training method in which a machine is provided with a small amount of classified data from a human source, while a larger amount of unclassified data is available to the machine.

[0035] The term "representation learning" is a field of methods that transforms raw data into representations or features that can be exploited in machine learning tasks. In supervised learning, features are learned via labeled inputs.

[0036] The term "convolutional neural network" or "CNN" is a biologically inspired network of interconnected data used in deep learning to detect, segment, and recognize related objects and regions in a dataset. CNN evaluates raw data in the form of multiple arrays, breaking the data into a series of levels, examining the learned features in the data.

[0037] The term "transfer learning" is the process by which a machine stores information used when it correctly or incorrectly solved one problem to solve another problem of the same or similar nature as the first problem. Transfer learning may also be called "inductive learning." For example, transfer learning may utilize data from a previous task.

[0038] The term "active learning" is a machine learning process in which a machine selects the set of examples to receive as training data, rather than passively receiving examples selected by an external entity. For example, when learning, the machine can be allowed to select examples that the machine determines will be most useful for learning, rather than relying solely on an external human expert or external system to identify and provide examples.

[0039] The terms "computer-aided detection" or "computer-aided diagnosis" refer to computers analyzing medical images for the purpose of suggesting a possible diagnosis.

[0040] Certain examples use neural networks and / or other machine learning to implement new workflows for image and associated patient analysis, including generating alerts based on radiological findings that can be generated and delivered at the point of care for radiological examinations. Certain examples use artificial intelligence (AI) algorithms to process one or more imaging examinations (e.g., images or sets of images) and provide alerts based on automatic examination analysis. Alerts (e.g., including notifications, recommendations, other actions, etc.) can be intended for technicians, clinical team providers (e.g., nurses, doctors, etc.), radiologists, administrative staff, surgical staff, and / or even patients who obtain the examination. For example, an alert can be used to indicate a specific or multiple quality control and / or radiological findings or their lack in the examination image data.

[0041] In some examples, the AI ​​algorithm may be (1) embedded within an imaging device, (2) running on a mobile device (e.g., a tablet, smartphone, laptop, other handheld or mobile computing device, etc.), and / or (3) running in the cloud (e.g., internally or externally), and the alert delivered via a web browser (e.g., which may appear on a radiology system, mobile device, computer, etc.). Such a configuration may be vendor neutral and compatible with conventional imaging systems. For example, if the AI ​​processor is running on a mobile device and / or in the "cloud", the configuration may receive images in the following ways: (A) directly from an X-ray and / or other imaging system (e.g., established as an auxiliary push destination, such as a Digital Imaging and Communications in Medicine (DICOM) node, etc.), (B) by keying a Picture Archiving and Communication System (PACS) destination for redundant image access, (C) by retrieving image data via a sniffer method (e.g., pulling a DICOM image from the system once it is generated), etc.

[0042] Certain examples provide apparatus, systems, methods, etc. for determining the progression of a disease and / or other condition based on the output of an algorithm instantiated using and / or driven by an artificial intelligence (AI) model (such as a deep learning network model, a machine learning network model, etc.). For example, the presence of a medically placed tube or line (e.g., a chest tube, a nasogastric tube, an endotracheal tube, a vascular line, a peripherally inserted central venous catheter, a central venous access catheter, a pulmonary artery catheter, etc.) can be determined based on the output of the AI ​​detection algorithm. In addition, the placement of a catheter, medical tube or line within a region of interest (e.g., lungs, stomach, vascular system, etc.) can be determined based on the output of the AI ​​detection algorithm (e.g., whether the catheter is properly placed).

[0043] Thus, certain examples provide systems and methods for detecting a medically placed catheter, tube, or wire within a patient's region of interest and whether the catheter, tube, or wire is properly placed within the region of interest based on an AI algorithm applied to patient data. Example methods include detecting the presence of a medically placed catheter, tube, or wire in an image; detecting an end of the medically placed catheter, tube, or wire in the image; detecting at least one reference marker or anatomical landmark in the image; determining an anatomical coordinate system or a patient coordinate system using the at least one reference marker or anatomical landmark, determining whether the end of the medically placed catheter, tube, or wire is properly placed relative to the reference marker or anatomical landmark, providing a visual representation of the patient coordinate system and the position of the end of the catheter, tube, or wire relative to the visual representation, and / or providing a physician with a notification as to whether the medically placed tube or wire is properly placed relative to the reference marker or anatomical landmark. In certain embodiments, the AI ​​system may detect one or more reference or anatomical landmarks, such as for defining a patient coordinate system used in determining proper placement of a catheter, tube, or wire relative to a patient coordinate system, detect the presence of a medically placed catheter, wire, or tube with reference to the patient coordinate system based on one or more anatomical landmarks, graphically mark the medically placed catheter, wire, or tube with a colored graphic overlay on the patient coordinate system, detect an end (e.g., a distal end) of a medically placed catheter, wire, or tube with reference to the patient coordinate system, graphically mark the end of a medically placed catheter, tube, or wire with reference to the patient coordinate system, graphically mark one or more reference or anatomical landmarks with reference to the patient coordinate system, calculate a distance between an end of a medically placed catheter, tube, or wire and an expected position of the end, and / or calculate and provide a confidence metric (e.g., for the calculated distance, for determination of the presence of a medically placed tube or wire, for accuracy of detecting the end of the tube or wire, for accuracy of detecting the reference or anatomical landmark, etc.). The AI ​​system is trained based on images with or without medically placed catheters, tubes or wires, images with correctly placed catheters, tubes or wires, images with misplaced catheters, tubes or wires, images with one or more reference or anatomical landmarks, and / or images without one or more of the reference or anatomical landmarks.

[0044] For example, a patient in an intensive care setting receives a chest x-ray (or other area) to monitor the placement of a medically placed catheter, tube, or line. If the catheter, tube, or line is misplaced, the medical team may need to intervene more quickly to correctly place the medical catheter, tube, or line. The artificial intelligence system may detect the presence of a medically placed catheter, tube, or line, detect the end of a medically placed catheter, tube, or line, detect one or more reference markers or anatomical markers, determine a patient coordinate system using the position of the detected markers, and assess whether the catheter, tube, or line is correctly placed. For example, an alarm may be generated and output at the point of care, on a device (e.g., an imaging device, an imaging workstation, etc.) to notify the clinical care team and / or otherwise provide instructions (e.g., a notification that the catheter is correctly placed or not, or an instruction to remove the catheter, tube, or line, an instruction to shift the catheter, tube, or line in a certain direction, etc.).

[0045] The technology described herein provides a faster means of determining whether a medically placed catheter, tube or line is not correctly placed and illustrating the location of the correct placement of the catheter, tube or line. This enables faster intervention to ensure that the catheter, tube or line is in the proper position for patient safety. In addition, this alleviates some of the burden on the medical team providing assistance to the patient, especially those who may be untrained or inexperienced.

[0046] Deep learning is a class of machine learning techniques that employs a representation learning approach, which allows a machine to be given raw data and determine the representations needed to classify the data. Deep learning uses a back-propagation algorithm that changes the internal parameters of the deep learning machine (e.g., node weights) to determine the structure in the data set. Deep learning machines can utilize a variety of multi-layer architectures and algorithms. For example, while machine learning involves identifying features to be used to train a network, deep learning processes raw data to identify features of interest without external identification.

[0047] Deep learning in a neural network environment consists of many interconnected nodes called neurons. Input neurons activated by external sources activate other neurons based on connections to those other neurons controlled by machine parameters. Neural networks function in a certain way based on their own parameters. Learning improves the machine parameters, and more broadly, the connections between neurons in the network, so that the neural network functions in the desired way.

[0048] Deep learning using convolutional neural networks uses convolutional filters to segment data to locate and identify learned observable features in the data. Each filter or layer of the CNN architecture transforms the input data to increase the selectivity and invariance of the data. This abstraction of the data allows the machine to focus on the features in the data that it is trying to classify and ignore irrelevant background information.

[0049] Deep learning operates on the understanding that many data sets consist of high-level features that in turn consist of lower-level features. For example, when examining an image, instead of looking for objects, it is more efficient to look for edges, which form motifs, which form parts, which form the objects you are looking for. These hierarchies of features can be seen in many different forms of data, such as speech and text.

[0050] The learned observable features include the objects and quantifiable regularities that the machine learned during supervised learning. A machine that is provided with a large set of effectively classified data is better positioned to distinguish and extract features that are relevant to the successful classification of new data.

[0051] A deep learning machine using transfer learning can correctly connect data features to certain classifications confirmed by a human expert. Conversely, the same machine can update parameters used for classification when a human expert tells it that the classification is wrong. For example, the settings and / or other configuration information can be guided by the use of learned settings and / or other configuration information, and the number of changes and / or other possibilities for settings and / or other configuration information for a given situation can be reduced when the system is used more times (e.g., repeatedly and / or by multiple users).

[0052] For example, an exemplary deep learning neural network can be trained using an expert classification dataset, which is classified and further annotated for object localization. This dataset builds the first parameters of the neural network, and this will become the supervised learning phase. During the supervised learning phase, it can be tested whether the neural network has achieved the desired behavior.

[0053] Once the desired neural network behavior has been achieved (e.g., the machine is trained to operate according to specified thresholds, etc.), the machine can be deployed for use (e.g., using "real" data to test the machine, etc.). During operation, the neural network classification can be confirmed or rejected (e.g., by an expert user, an expert system, a reference database, etc.) to continue to improve the neural network behavior. The exemplary neural network is then in a state of transfer learning because the classification parameters that determine the neural network behavior are updated based on ongoing interactions. In some examples, the neural network can provide direct feedback to another process. In some examples, the data output by the neural network is buffered (e.g., via the cloud, etc.) and verified before being provided to another process.

[0054] Deep learning machines using convolutional neural networks (CNN) can be used for image analysis. The stages of CNN analysis can be used for facial recognition in natural images, computer-aided diagnosis (CAD), etc.

[0055] High-quality medical image data may be acquired using one or more imaging modalities such as x-ray, computed tomography (CT), molecular imaging and computed tomography (MICT), magnetic resonance imaging (MRI), etc. Medical image quality is generally not affected by the machine that produces the image, but rather by the patient. For example, movement of the patient during an MRI may result in blurred or distorted images, which may prevent an accurate diagnosis.

[0056] Interpreting medical images without regard to quality is only a recent development. Medical images are mostly interpreted by physicians, but these interpretations can be subjective, influenced by the physician's experience and / or fatigue in the field. Image analysis via machine learning can support the workflow of healthcare practitioners.

[0057] For example, deep learning machines can provide computer-aided detection support to improve their image analysis in terms of image quality and classification. However, the problems faced by deep learning machines applied in the medical field often cause many misclassifications. For example, deep learning machines must overcome small training data sets and require repeated adjustments.

[0058] For example, deep learning machines can be used to determine the quality of medical images with minimal training. Semi-supervised and unsupervised deep learning machines can be used to quantitatively measure quality aspects of an image. For example, a deep learning machine can be used after an image has been acquired to determine whether the image is of sufficient quality for diagnosis. Supervised deep learning machines can also be used for computer-aided diagnosis. For example, supervised learning can help reduce susceptibility to misclassification.

[0059] Deep learning machines can leverage transfer learning when interacting with physicians to offset the small datasets available in supervised training. These deep learning machines can improve their computer-aided diagnosis over time through training and transfer learning.

[0060] Reference now Figure 1As also disclosed in U.S. Pat. No. 11,410,341 (the '341 patent), entitled System And Method For Visualizing Placement Of A Medical Tube Or Line, the entire contents of which are expressly incorporated herein by reference for all purposes, an example condition comparator apparatus 100 is shown to include a plurality of inputs 110, 115, an artificial intelligence (AI) system 120, and an output comparator 130. Each input 110, 115 is provided to the AI ​​system 120, which classifies images and / or other information in the respective input 110, 115 to identify a condition in the input 110, 115 and generates an indication of the identified condition based on the input 110, 115. In certain embodiments, the AI ​​system 120 can classify images and / or other information in the respective inputs 110, 115 to identify medically placed catheters, tubes, or wires (e.g., chest tubes, nasogastric tubes, endotracheal tubes, vascular lines, peripherally inserted central venous catheters, central venous access catheters, pulmonary artery catheters, etc.), and identify reference or anatomical landmarks associated with the catheters, tubes, or wires and their desired placement. Using the example comparator device 100, it can be determined whether the end of the catheter, tube, or wire is aligned relative to a patient coordinate system defined by one or more detected reference or anatomical landmarks ( Figure 6 ) is correctly placed in the correct position or area, or in the area of ​​interest of the patient. Specifically, both the end of the catheter, tube or line and the reference mark or anatomical mark can be located, and whether the end of the tube or line is correctly placed relative to the reference mark or anatomical mark can be determined by using the determined patient coordinate system. The distance between the end of the tube or line and the reference mark or anatomical mark can be measured via the patient coordinate system to determine whether the end of the catheter, tube or line is correctly placed. Confidence metrics (e.g., for the calculated distance, for the determination of the presence of the catheter, tube or line placed in a medical manner, for the accuracy of detecting the end of the catheter, tube or line, for the accuracy of detecting the reference mark or anatomical mark, etc.) can be calculated and / or provided via user-perceivable notifications, or stored for further reference. In addition, notifications or alarms about whether the catheter, tube or line placed in a medical manner can be provided. If the tube or line is not correctly placed, additional instructions can be provided via the patient coordinate system related to moving the catheter, tube or line in a certain direction.

[0061] Furthermore, using the example comparator device 100, it can be determined whether any complications are detected within the patient's anatomy, where the detected complications are in the form of non-biological complications (e.g., indwelling guidewires or other catheter failures from catheters, tubes, or wires) or biological complications (e.g., pneumothorax, hemothorax, etc.) If a complication is detected, a notification can be provided to the physician regarding the presence and location of the complication relative to the patient's coordinate system.

[0062] Figure 2 It shows that based on Figure 1 The exemplary clinical progress analysis device 200 is constructed with the exemplary condition comparator 100 of the embodiment of the present invention. The exemplary device 200 includes a data source 210, an artificial intelligence (AI) model or system 220, an electronic data storage device or database 230, a comparator 240, an output generator 250, and a trigger 260. The inputs 110, 115 may be provided to the AI ​​system 220 by the data source 210 (e.g., a storage device, an imaging device, etc. incorporated into the device 200 and / or otherwise connected to the device, etc.).

[0063] The example AI system 220 processes inputs over time to associate inputs from the data source 210 with classifications. Thus, the AI ​​system 220 processes input image data and / or other data to identify conditions in the input data and, based on one or more states (e.g., catheter, tube, or line present, catheter, tube, or line not present, reference landmark or anatomical landmark present, reference landmark or anatomical landmark not present, determination of a patient coordinate system using anatomical landmarks ( Figure 6 and Figure 7 ) to define correct placement of a catheter, tube or line, correct placement of a catheter, tube or line, misplacement of a catheter, tube or line, complication present, complication not present) to classify the condition as specified by an equation, threshold, and / or other criteria. In certain embodiments, the AI ​​system 220 processes input image data and / or other data to detect a medically placed catheter, tube or line, determine whether the end of a medically placed catheter, tube or line is correctly placed, identify the position of the end of the catheter, tube or line relative to a patient coordinate system based on the patient's anatomical landmarks, and determine the presence of any complications in the patient's anatomy related to the correct placement of the end of the catheter, tube or line. For example, the output of the AI ​​system 220 can be stored in a data storage device 230.

[0064] Over time, classifications made by the AI ​​system 220 relative to the same type of input 110, 115 from the data source 210 (e.g., lung MR images of the same patient acquired at time t0 and t1, etc.) may be generated and stored in the data storage device 230. The classifications are provided to a comparator 240, which compares the classifications at two or more different times (e.g., before the insertion of a catheter, tube, or line and after the insertion of the catheter, tube, or line) to identify a medically placed catheter, tube, or line and determine whether the end of the medically placed catheter, tube, or line is correctly placed. For example, at time t0, the catheter, tube, or line may not be present in the region of interest, and at time t1 or later, the end of the catheter, tube, or line may be placed in a location within the region of interest / acceptable area defined using the patient coordinate system (which may or may not be correctly placed).

[0065] The comparator 240 provides a result indicative of a trend / progress. In certain embodiments, the comparator 240 provides a result indicative of the placement of the end of a medically placed catheter, tube, or line. The output generator 250 converts the result into an output that can be displayed, stored, provided to another system for further processing, such as an alarm, command, adjustment of patient care (e.g., a point of care alarm system, an imaging / radiology workstation, a computer-aided diagnosis (CAD) processor, a scheduling system, a medical device, etc.), etc.

[0066] Trigger 260 coordinates actions between data source 210, AI system 220, data storage device 230, comparator 240, and output generator 250. Trigger 260 can initiate input of data from data source 210 to AI system 220, comparison of results from data storage device 230 by comparator 240, and output by output generator 250. Thus, trigger 260 acts as a coordinator between elements of device 200.

[0067] Figure 3 An example implementation of an AI system 220 is shown for processing image data for use by an AI model to quantify a condition (e.g., placement of a catheter, tube, or line). The example implementation of the AI ​​system 220 enables annotation of one or more images including organ regions and regions of interest within the organ regions. Figure 3 The example AI system 220 includes an image segmenter 1010 , a mask combiner 1020 , and a state comparator 1040 .

[0068] The exemplary image segmenter 1010 is used to identify a first mask and a second mask in an input image. For example, the image segmenter 1010 processes the image to segment a region of interest within an organ region identified in the image to obtain a first mask. The first mask is a segmentation mask, which is a filter that includes the region of interest in the image and excludes the rest of the image. For example, a mask can be applied to image data to exclude all regions except the region of interest. A convolutional neural network model (e.g., a generative adversarial network, etc.) can be used to obtain the mask. The image segmenter 1010 further processes the image to segment the organ region according to one or more criteria to obtain a second mask. For example, the second mask can represent the organ region, the area of ​​the organ region outside the region of interest, etc.

[0069] For example, if the organ region is a lung (and surrounding areas such as trachea), and the region of interest is a tube or line identified in the trachea, a first mask is generated to identify the tube or line placed medically, and a second mask is generated to identify the entire organ region. In another embodiment, if the organ region is a stomach, and the region of interest is a tube or line identified in the stomach, a first mask is generated to identify the tube or line placed medically, and a second mask is generated to identify the entire organ region. For example, if the organ region is a heart (and surrounding areas such as veins or other vascular systems), and the region of interest is a catheter or line identified in veins near the heart or other vascular systems, a first mask is generated to identify the catheter or line placed medically, and a second mask is generated to identify the entire organ region. Therefore, with regard to the catheter, tube or line placed medically, a first mask is generated for the catheter, tube or line, and a second mask is generated for the entire organ region (for example, vascular system, heart, lung, stomach, trachea, chest, pleural cavity, etc.) where the catheter, tube or line is placed.

[0070] The exemplary combiner 1020 combines the first and second masks and the associated areas with annotation terms in the image. For example, the annotation can be a relative qualification term used to generate quantification. For example, the mask area can be combined with descriptive terms (such as fuzzy, scattered, dense, etc.) to calculate the relative density value of the region of interest and organ area in the image. For example, the image area (e.g., the area of ​​the frontal image and the side image, etc.) can be combined to generate a volume metric.

[0071] The exemplary distance computer 1030 determines the distance between the end of the identified tube or wire and a reference or anatomical landmark (or determines the position of the tube or wire relative to the landmark). The distance may be calculated by identifying one or more attributes or landmarks within the image (such as the carina and / or spine).

[0072] In one exemplary embodiment for operating the exemplary distance computer 1030, the anatomical structures present in the image 1702 may be used in a variety of ways to form a patient coordinate system 1700 ( Figure 6 and Figure 7 In a first exemplary embodiment, the AI ​​system 220, the exemplary distance computer 1030 and / or the processor 1312 ( Figure 4 ) can define one or more local coordinate systems 1700 near the carina 1712 by detecting and displaying the midpoints of the patient's vertebral bodies / vertebrae 1704 and the "vertical axis" 1706 formed by these midpoints. The AI ​​system 220 and / or the processor 1312 can similarly define a "horizontal axis" 1708 formed by the midpoints of the transverse processes 1710 of the vertebral bodies / vertebrae 1704. These horizontal axes 1708 and vertical axes 1706 will be generated at each vertebral level. This embodiment selects the local vertical axis 1706 to be used to perform the measurement based on the position of one or more vertebrae 1704 near the carina 1712 and / or the end of the device. The detected horizontal axis 1708 and vertical axis 1706 can be fine-tuned to ensure that the two directions are precisely perpendicular to each other. This local coordinate system 1700 may be useful in the case of severe spinal disorders, where the shape of the spine is significantly different from a straight line, and therefore the local "vertical" axis may be significantly different at different locations within the patient's anatomy.

[0073] According to another exemplary embodiment of the present disclosure, to define a vertical axis 1706 for a chest image by employing the spinal axis, the AI ​​system 220 / processor 1312 may identify between the midpoint of the upper edge of the uppermost thoracic vertebra 1704 (which is typically the first thoracic vertebra) and the midpoint of the lower edge of the lowermost thoracic vertebra 1704 (which is typically the twelfth thoracic vertebra). In addition, to define a horizontal axis 1708, the AI ​​system 220 / processor 1312 may calculate a straight line orthogonal to the vertical axis 1706 that extends through the carina 1712. In a modified version of this process, the AI ​​system 220 / processor 1312 may detect the edges of multiple vertebrae 1704 and calculate the midpoint of each vertebra 1704, and then fit a straight line to the set of points. In this process, the AI ​​system 220 / processor 1312 may determine the total angle of rotation of the patient and adapt the coordinate system utilized to the patient position.

[0074] According to yet another exemplary embodiment of the present disclosure for determining a vertical axis 1706 for a chest image, the AI ​​system 220 / processor 1312 may use the position of the trachea detected by the AI ​​system. One potential embodiment defines a vertical axis between the carina 1712 and the uppermost midpoint of the trachea 1714. An alternative embodiment segments the entire trachea and fits a straight line to the detected tracheal points. To define a horizontal axis 1708, the AI ​​system 220 / processor 1312 calculates a straight line through the carina 1712 that is orthogonal to the vertical axis 1706.

[0075] In yet another exemplary embodiment of the present disclosure, the patient coordinate system 1700 may be determined by the AI ​​system 220 / processor 1312 using a deep learning regression model that predicts the patient rotation angle directly from the entire x-ray image.

[0076] The example status comparator 1040 uses the patient coordinate system to compare the distance or measured position of the catheter tip with a preset distance or expected position for the type of catheter, tube or line and / or the area of ​​interest where the catheter, tube or line is placed (e.g., according to a predetermined rule). Based on the comparison, the status comparator 1040 can determine whether the end of the catheter, tube or line is correctly placed relative to a reference marker or anatomical landmark. The determination and the patient coordinate system used to make the determination can be annotated onto the medical image to provide the physician with direct and clear information about the location of the catheter and the catheter tip and any position changes that need to be made to the catheter tip for correct placement.

[0077] Thus, the AI ​​system 220 may be configured to annotate medical images or sets of related medical images for AI / machine learning / deep learning / CAD algorithm training to quantify conditions. Such methods are consistent, repeatable methods that can replace today's common subjective methods to automatically and accurately detect the presence and placement of medically placed catheters, tubes, or wires.

[0078] Although example implementations are described in conjunction with those disclosed in U.S. Pat. No. 11,410,341, entitled System And Method For Visualizing Placement Of A Medical Tube Or Line (the '341 patent), the entire contents of which are expressly incorporated herein by reference for all purposes, the disclosed elements, processes, and / or devices described in the '341 patent may be combined, divided, rearranged, omitted, eliminated, and / or implemented in any other manner. In addition, the components disclosed and described herein may be implemented by hardware, machine-readable instructions, software, firmware, and / or any combination of hardware, machine-readable instructions, software, and / or firmware. Thus, for example, the components disclosed and described herein may be implemented by analog and / or digital circuits, logic circuits, programmable processors, application specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field programmable logic devices (FPLDs). When any of the claims in this patent are read to cover a purely software and / or firmware implemented apparatus or system, at least one of these components is expressly defined herein as including a tangible computer-readable storage device or storage disk, such as a memory, digital versatile disk (DVD), compact disk (CD), Blu-ray disk, etc., storing the software and / or firmware.

[0079] Figure 4 is a block diagram of an example image processing system or image processing system 1300 configured to perform at least the Figure 5 The image processing system 1300 may be, for example, a server, a personal computer, a mobile device (e.g., a cell phone, a smart phone, a tablet computer such as an iPad, etc.). TM ), a personal digital assistant (PDA), an Internet device, or any other type of computing device, such as a computing device that forms part of the medical radiography or digital imaging system 1301.

[0080] The image processing system 1300 of the illustrated example includes a processor 1312. The processor 1312 of the illustrated example is hardware. For example, the processor 1312 may be implemented by an integrated circuit, a logic circuit, a microprocessor, or a controller from any desired product line or manufacturer.

[0081] The processor 1312 of the illustrated example includes local memory 1313 (eg, a cache). Figure 4 The example processor 1312 performs at least Figure 5Instructions to implement training and implementation of the system, infrastructure, display and associated methods of the method 1600, such as the example data source 210, AI system 220, data storage device 230, comparator 240, output generator 250, trigger 260, etc. The processor 1312 of the illustrated example communicates with a main memory including a volatile memory 1314 and a non-volatile memory 1316 via a bus 1318. The volatile memory 1314 can be implemented by a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), a RAMBUS dynamic random access memory (RDRAM), and / or any other type of random access memory device. The non-volatile memory 1316 can be implemented by a flash memory and / or any other desired type of memory device. Access to the main memory 1314, 1316 is controlled by a clock controller.

[0082] The image processing system 1300 of the illustrated example also includes an interface circuit 1320. The interface circuit 1320 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), and / or a PCI express interface.

[0083] In the example shown, one or more input devices 1322 are connected to the interface circuit 1320. The input devices 1322 allow a user to enter data and commands into the processor 1312. The input devices may be implemented by, for example, sensors, microphones, cameras (still or video, RGB or depth, etc.), keyboards, buttons, mice, touch screens, trackpads, trackballs, isopoint, and / or voice recognition systems.

[0084] One or more output devices 1324 are also connected to the interface circuit 1320 of the illustrated example. The output device 1324 may be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display, a cathode ray tube display (CRT), a touch screen, a tactile output device, and / or a speaker). Thus, the interface circuit 1320 of the illustrated example typically includes a graphics driver card, a graphics driver chip, or a graphics driver processor.

[0085] The interface circuitry 1320 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, and / or network interface cards, to facilitate the exchange of data with an external machine (e.g., any kind of computing device) via a network 1326 (e.g., an Ethernet connection, a digital subscriber line (DSL), a telephone line, a coaxial cable, a cellular telephone system, etc.).

[0086] The image processing system 1300 of the illustrated example also includes one or more mass storage devices 1328 for storing software and / or data. Examples of such mass storage devices 1328 include floppy disk drives, hard drives, optical disk drives, Blu-ray disk drives, RAID systems, and digital versatile disk (DVD) drives.

[0087] Figure 4 The coded instructions 1332 may be stored in the mass storage device 1328, in the volatile memory 1314, in the non-volatile memory 1316, and / or on a removable tangible computer-readable storage medium (such as a CD or DVD).

[0088] A flowchart representing example machine readable instructions for implementing the components disclosed and described herein in the example method 1600 is combined with at least Figure 5 In an example, the machine-readable instructions include instructions for executing by a processor (such as a Figure 4 The program may be embodied in machine-readable instructions stored on a tangible computer-readable storage medium such as a CD-ROM, floppy disk, hard drive, digital versatile disk (DVD), Blu-ray disk, or memory associated with the processor 1312, but the entire program and / or portions thereof may alternatively be executed by a device other than the processor 1312 and / or embodied in firmware or dedicated hardware. In addition, although reference is made to at least Figure 5 The flowchart shown describes an example program, but many other methods of implementing the components disclosed and described herein may be alternatively used. For example, the execution order of the blocks may be changed, and / or some of the blocks described may be changed, eliminated, or combined. Although at least Figure 5 The flowcharts of the present invention depict example operations in the order shown, but these operations are not exhaustive and are not limited to the order shown. In addition, those skilled in the art may make various changes and modifications within the spirit and scope of the present disclosure. For example, the blocks shown in the flowcharts may be executed in an alternative order or may be executed in parallel.

[0089] As mentioned above, at least Figure 5The example processes of may be implemented using encoded instructions (e.g., computer and / or machine readable instructions) stored on a tangible computer readable storage medium, such as a hard drive, flash memory, read only memory (ROM), compact disk (CD), digital versatile disk (DVD), cache, random access memory (RAM), and / or any other storage device or storage disk where information is stored for any duration (e.g., for an extended period of time, permanently, transiently, for temporary buffering, and / or for caching of information). As used herein, the term tangible computer readable storage medium is expressly defined to include any type of computer readable storage device and / or storage disk and to exclude propagating signals and to exclude transmission media. As used herein, "tangible computer readable storage medium" and "tangible machine readable storage medium" may be used interchangeably. Additionally or alternatively, at least Figure 5 The example processes of may be implemented using coded instructions (e.g., computer and / or machine readable instructions) stored on a non-transitory computer and / or machine readable medium, such as a hard drive, flash memory, read-only memory, compact disk, digital versatile disk, cache, random access memory, and / or any other storage device or storage disk where information is stored for any duration (e.g., for an extended period of time, permanently, temporarily, for temporary buffering, and / or for caching of information). As used herein, the term non-transitory computer readable medium is expressly defined to include any type of computer readable storage device and / or storage disk, and excludes propagating signals and excludes transmission media. As used herein, the phrase "at least" when used as a transitional term in the preamble of a claim is open ended, just as the term "including" is open ended. In addition, just as the term "comprising" is open ended, the term "including" is also open ended.

[0090] As described above, these techniques can be used to identify a medically placed tube or line, and to determine whether the medically placed tube or line is correctly placed. For example, the medically placed tube or line can be an endotracheal tube, and the correct placement of the endotracheal tube in the trachea (e.g., relative to the carina) can be determined. In another example, the medically placed tube or line can be a nasogastric (NG) tube, and the correct placement of the nasogastric tube in the stomach can be determined. In another example, the medically placed tube or line can be a vascular line (e.g., a PICC line, a central venous access catheter (CVC), a pulmonary artery catheter (PAC), etc.), and the correct placement of the vascular line in a certain vascular system can be determined. In yet another example, the medically placed line can be a chest tube, and the correct placement of the chest tube in the chest (specifically, the inflation space) can be determined. These examples are intended to be non-limiting, and any other tube or line inserted into a region of interest in the body can be identified and its correct placement determined.

[0091] Figure 5 is a flow chart of an embodiment of a method 1600 for determining placement of a medically placed catheter, tube, or wire within a region of interest. One or more steps of the method 1600 may be performed by Figure 4 One or more steps may be performed simultaneously or in parallel. Figure 5The method 1600 includes receiving or obtaining an image (e.g., chest image) (frame 1602) of a patient including a region of interest (ROI). The image may include a catheter, tube or line placed medically inserted into the region of interest. The image may be provided when the patient is inserted with a catheter, tube or line. The method 1600 also includes receiving or obtaining an input (frame 1603) of a type (e.g., CVC or PAC) of a catheter, tube or line to be detected and / or a region of interest (e.g., central vein or artery) to be inserted thereinto of a catheter, tube or line. The input may be a user-defined distance or rule to define the correct placement of the end of a catheter, tube or line placed medically relative to a reference position or anatomical position (e.g., carina). In certain embodiments, the input may simply be a catheter, tube or line type and / or a desired region of interest for correctly placing a catheter, tube or line therein. Based on the input, certain defined distances or rules (e.g., to the left, right, above, and / or below a particular anatomical location) may be utilized that define the correct placement of the end of a particular catheter, tube, or line within a particular region of interest (e.g., a particular distance range above the carina for a CVC or PAC). The method 1600 also includes detecting the catheter, tube, or line within the image using the above-described techniques (box 1604). The method 1600 includes identifying the end (e.g., the distal end) of the catheter, tube, or line within the region of interest in the image (box 1606). The method 1600 also includes identifying a reference marker or anatomical landmark within the image (box 1608). The reference marker or anatomical landmark will vary based on the type of catheter, tube, or line utilized and the region of interest in which the catheter, tube, or line is disposed. For example, for an endotracheal tube, the reference marker or anatomical landmark may be the carina of the trachea. For a nasogastric (NG) tube, the reference marker or anatomical landmark may be a location within the stomach below the gastroesophageal junction. For a vascular line, a reference landmark or anatomical landmark may be the carina of the trachea or a location within the superior vena cava, the inferior vena cava, or proximal to the right atrium, etc. Determination of the location of the anatomical landmark also includes determination of the landmark utilized in forming the patient coordinate system 1700 (block 1609) according to one or more of the processes previously described.

[0092] After identifying the end of the catheter, tube or wire, the reference or anatomical landmark, and the patient coordinate system 1700, the method 1600 includes measuring the distance between the end of the catheter, tube or wire and an acceptable position or range of positions of the end relative to the reference or anatomical landmark (block 1610). The distance can be illustrated as a Euclidean length between the end and the desired position or range of positions, and / or as individual horizontal and vertical distance components relative to the patient coordinate system 1700. The method 1600 includes generating a combined image 1702 ( Figure 6 and Figure 7), the combined image having an overlay 1703 positioned thereon and including indications of various features of interest within the x-ray image 1701 of the anatomical structure, such as, but not limited to, the end of a catheter, tube, or wire, a reference marker or anatomical landmark, a patient coordinate system, and / or a measured distance identified in the combined image 1702 (block 1612). Generating the combined image 1702 including the x-ray image 1701 and the overlay 1703 includes overlaying various markers on the received x-ray image 1701 of the patient. For example, a color coding (e.g., a color-coded graphical overlay) may be overlaid on the detected catheter, tube, or wire 1705. In certain embodiments, the patient may include more than one catheter, tube, or wire, and the catheter, tube, or wire of interest is color-coded. A graphical marker may be overlaid on the image to indicate the end 1707 of the catheter, wire, or tube. Another graphical marker may be overlaid on the image to indicate a reference marker or anatomical landmark 1709. The graphical markers may include the same shape or different shapes. Non-limiting examples of these shapes can be hollow circles or other elliptical shapes, hollow straight line shapes, hollow triangle shapes, or other shapes. Different graphics and tubes can be color-coded using different colors. For example, the graphic markings of the tube or line, the graphic markings of the reference mark or anatomical mark, and the graphic markings of the tube or line can be green, blue, and yellow, respectively. The graphic markings can also be superimposed on the image to indicate the patient coordinate system 1700, the distance 1720 between the end of the tube or line and the reference mark or anatomical mark when calculating the distance, and the horizontal component 1722 and the vertical component 1724 of the distance 1720 as determined relative to the patient coordinate system 1700. The graphic markings of the distance can also include a measured value. Method 1600 also includes displaying the combined image on a display (frame 1614). The combined image can be displayed to medical personnel in real time so that they can adjust the placement of the tube or line as needed. In certain embodiments, the combined image can be displayed as a DICOM image.

[0093] In some embodiments, the method 1600 includes calculating one or more corresponding confidence metrics (box 1616). The confidence metric can be for the calculated distance, for the determination of the presence of a medically placed tube or line, for the accuracy of detecting the end of the tube or line, and / or for the accuracy of detecting a reference mark or anatomical landmark. The confidence metric can include a confidence level or a confidence interval. The confidence metric can be stored for future reference. In some embodiments, the method 1600 can include providing one or more confidence metrics in the confidence metric to the user (box 1618). For example, the confidence metric can be displayed on a combined image or provided on a separate device (e.g., a user's device). In some embodiments, the confidence metric can be written into a standard or private information tag (e.g., DICOM) and visible in a subsequent information system (e.g., PACS) to which the image is sent.

[0094] In certain embodiments, when determining whether the end of a medically placed tube or line is correctly placed (e.g., via a deep learning network model), the method 1600 includes comparing the measured distance between the end of the tube or line and a reference marker or anatomical landmark to a desired threshold (box 1620), and determining whether the distance is acceptable (box 1622). The desired threshold may represent an acceptable range of distances between the end of the tube or line and a reference marker or anatomical landmark of the tube or line to be correctly placed. For example, for an endotracheal tube, the desired threshold may be 2 centimeters (cm) to 3 centimeters above the carina (e.g., an anatomical landmark). For a nasogastric tube, the desired threshold may be a certain distance range below the gastroesophageal junction. For a central venous catheter (CVC), the desired threshold may be a distance range above or below a landmark (such as the carina or right atrium). If the measured distance is unacceptable, the method 1600 includes providing a user-perceivable indication of misplacement (box 1624). The indication may be provided on a display, where the combined image is displayed or provided on another device (e.g., a user's device). The indication may be text stating that the tube or line is misplaced. In certain embodiments, the text may be more specific and state that the tube or line is too high (e.g., greater than the desired 2cm to 3cm for endotracheal tube placement) or too low (e.g., less than 2cm for endotracheal tube placement). In certain embodiments, the text may provide additional instructions (e.g., for raising or lowering the end of the tube or line by a certain distance). In some embodiments, the text may be color-coded (e.g., coded in orange or red) to further indicate misplacement. In some embodiments, the indication may be provided via one or more graphic marks or color coding of the tube or line displayed on the combined image. For example, one or more of the graphic marks (e.g., for the end of the tube or line, for reference marks or anatomical marks, and / or an indication of the measured distance therebetween) and / or the tube or line may be color-coded to a specific color (e.g., red or orange) to indicate misplacement. Alternatively or in addition thereto, if the tube or line is misplaced, one or more of the graphic marks may flash. If the measured distance is acceptable, the method 1600 includes providing a user-perceivable indication of the correct placement of the tube or line (box 1626). The indication may be provided on a display where the combined image is displayed or provided on another device (e.g., a user's device). The indication of correct placement may be text stating that the tube or line is correctly placed. In some embodiments, the indication for correct placement may be provided via color coding of one or more graphic marks of the tube or line displayed on the combined image (e.g., all graphic marks and / or tubes or lines may be green color coded). In some embodiments, the indication of correct placement or misplacement may be written into a standard or private information tag (e.g., DICOM) and visible in a subsequent information system (e.g., PACS) to which the image is sent.In certain embodiments, determination of the ends of medically placed tubes or wires may be made manually by medical personnel viewing the displayed combined image.

[0095] In yet another exemplary embodiment of method 1600, in block 1625, the AI ​​system 220 / processor 1312 may also be configured to detect any complications, e.g., non-biological complications, such as contraindications, including indwelling guidewires, and biological complications, including but not limited to pneumothorax, hemothorax, etc. If detected by the AI ​​system 220 / processor 1312, in block 1627, an alert may be provided to the user / physician regarding the presence, location, and type of the detected complication.

[0096] Figure 6 and Figure 7 1 is an example of a combined image 1702 (e.g., a DICOM image) that may be displayed on a display identifying a catheter, tube, or line 1705 (e.g., a CVC) within a patient. As depicted, the combined image 1702 includes an x-ray image 1701 (e.g., an image of an anatomical structure, such as a chest image) of the patient showing the CVC 1705 disposed within a central vein and includes an overlay 1703. The overlay 1703 includes a first graphical marker 1734 (e.g., a circle) superimposed on the x-ray image 1701 that indicates the location of an end 1707 of the CVC 1705. A second graphical marker 1736 (e.g., a solid circle with a chevron shape) superimposed on the chest image indicates a desired placement location 1737 of the tip 1707, which may be determined relative to or identical to a reference or anatomical location 1709 (e.g., vertebra 1704, carina 1712, trachea 1714, which may be the same or different for each anatomical location 1709). A third graphical marker 1738 indicates a distance 1720 (e.g., a Euclidean distance) between the tip 1707 of the CVC 1705 and the reference or anatomical location 1736. A numerical value 1740 of the measured distance accompanies the graphical marker 1738. Optionally as part of the third marking 1738 , a patient coordinate system 1700 is also shown in the overlay 1703 , comprising a vertical axis 1706 , a horizontal axis 1708 , and a horizontal component 1722 and a vertical component 1724 of a distance 1720 determined relative to the patient coordinate system 1700 and accompanied by their independent numerical values ​​1740 .

[0097] In some embodiments, a confidence metric in the measured distance generated by the artificial intelligence is also displayed (e.g., as indicated by a confidence level). In some embodiments, the tube 1705, the graphical marker 1734, and / or the graphical marker 1736 may be color-coded (e.g., yellow, green, and red). The combined image 1702 may include a title (not shown) that includes information related to the image 1702. For example, the title includes the type of tube or line 1705 (e.g., NG tube, CVC), whether the placement of the tube is correct, and the calculated distance 1720 between the end 1707 of the tube 1705 and the desired position reference marker 1736 and / or anatomical marker 1709. In some embodiments, the title may include an indication of whether the tube or line is detected. In some embodiments, one or more confidence metrics (e.g., for the calculated distance, for the determination of the presence of a medically placed tube or line, for the accuracy of detecting the end of the tube or line, and / or for the accuracy of detecting a reference marker or anatomical marker) may be displayed on the combined image 1702.

[0098] Reference now Figure 8 The illustrated exemplary embodiment of , presents a schematic diagram of an output device 1324 having a combined image 1702 identifying a catheter, tube or line 1705 within a patient and an accompanying user checklist 1760 that can be displayed on a display. Figure 8 , a combined image 1702 includes an x-ray image 1701 (e.g., a chest image) of a patient with an overlay 1703 showing an NG tube 1705 disposed within the esophagus and extending into the stomach. A first graphical marker 1734 (e.g., a circle) forming part of the overlay 1703 on the x-ray image 1701 indicates a detected location of an end or tip 1707 of the NG tube 1705. A second graphical marker 1736 (e.g., a circle) forming part of the overlay 1703 on the x-ray image 1701 indicates a detected placement location of a side orifice 1709 of the NG tube 1705. The overlay 1703 on the combined image 1702 additionally includes a first anatomical reference or location 1711 (e.g., diaphragm 1715) and a second anatomical reference or location 1713 (e.g., carina 1712) superimposed on the x-ray image 1701 and showing the locations of the first anatomical reference 1711 and the second anatomical reference 1713.

[0099] About AI system 220 / processor 1312 Figure 8 The operation of combining images 1702 will now be referred to Fig. 9 , perform the previously described Figure 5A modified embodiment 1600' of method 1600 is provided so that a combined image 1702 is generated by step 1612 and then displayed in step 1614. As mentioned above, in certain embodiments, the tube 1705, the first graphic mark 1734, the second graphic mark 1736, and / or the anatomical references 1711, 1713 may be color coded (e.g., yellow, green, and red) to enable easier viewing within the combined image 1702. Fig. 9 1701, method 1600' also includes step 1630, in which the AI ​​system 220 / processor 1312 generates a checklist 1760 for presentation to a user on output device 1324 along with combined image 1702 and stack 1703 in step 1614. To generate the checklist 1760, the AI ​​system 220 / processor 1312 receives input regarding the anatomical structures present within the x-ray image 1701, the types of tubes 1705 present within the anatomical structure image 1701, and the locations of the tubes 1705 and their features, including the distal end 1707 and the side port 1709, within the x-ray image 1701 as determined by the AI ​​system 220 / 1312. In step 1632, input to the checklist generation step 1630 may additionally be provided from the mass storage 1328 or another electronic storage device in which information regarding accepted clinical guidelines and / or parameters for properly placing the wire or tube 1705 within the anatomical structure for various interventional procedures is stored. Figure 8 In the illustrated example of , since tube 1705 is an NG tube and the anatomical structure in x-ray image 1701 is a chest, information from mass storage device 1328 related to the proper placement of tube 1705 in the chest may be used to form a checklist 1760 that includes, but is not limited to:

[0100] 1. On the chest (AP / PA) x-ray image 1701, the tube 1705 begins descending near the midline;

[0101] 2. Tube 1705 bisects the airway at carina 1712 without deviation;

[0102] 3. Tube 1705 passes through the diaphragm 1715 near the midline and then deflects toward the stomach;

[0103] 4. The tip 1707 of the NG tube 1705 should be visible at least ten (10) cm beyond the gastroesophageal junction (GEJ) and the side port 1709 should be securely placed in the stomach.

[0104] Using this information, the AI ​​system 220 / processor 1312 may generate a checklist 1760 in step 1630 , in order to present the checklist 1760 on the output device 1324 in conjunction with the combined image 1702 in step 1614 .

[0105] In AI Systems 220 and Figure 8 In the exemplary embodiment of the results of the operation of the method 1600', in step 1632, the checklist 1760 is presented together with the combined image 1702, including a plurality of steps 1762. Each step 1762 corresponds to a certain aspect of the combined image 1702 to be evaluated by the user, such as but not limited to clinical guidelines associated with the evaluation of the correct placement of the tube or line 1705, which are in the Figure 8 1702 by steps regarding the proper placement of the NG tube shown in the combined image 1702. Steps 1762 may be provided in the checklist 1760 as separate items to be verified by the initial reviewer within the combined image 1702, such as in the form of fixed text instructions based on relevant clinical guidelines to assist less experienced users in verifying the placement of the tube 1705 in the combined image 1702 and to standardize the information provided with respect to the combined image 1702 for later review. Steps 1762 may optionally each be accompanied by user-selectable responses 1763, such as check boxes 1764, that enable the initial reviewer to provide input regarding each step 1762. Figure 8 In the exemplary embodiment shown, the check boxes 1764 include a "yes" box 1766, a "no" box 1768, and an "uncertain" box 1770 associated with each step 1762, which can be selected by the reviewer in step 1633, such as by using the input device 1322 ( Figure 4 ) Click one of the boxes 1766, 1768, 1770 to select. The selection made by the user on the check box 1764 can be stored in association with the combined image 1702 for later review by a subsequent user.

[0106] In another exemplary embodiment of the present disclosure, the check boxes 1764 may each be associated with a confidence or classification score, e.g., for each step 1762, the "yes" box 1766 may have a score or value of +1, the "no" box 1768 may have a value of -1, and the "uncertain" box 1770 or step 1762 with no answer may have a value of 0. After the user has selected the appropriate box 1766, 1768, 1770 for each step 1762, the total score for all steps 1762 for the combined image 1702 may be determined by reviewing the selected boxes 1766, 1768, 1770 for each step 1762 by the AI ​​system 220 / processor 1312 in step 1634. The AI ​​system 220 / processor 1312 may then calculate the total score for the combined image 1702 and output the total score to the user, and / or may save the total score in association with the combined image 1702. If the total score of the combined image 1702 is below a predetermined threshold, the combined image 1702 may be annotated or identified for quick review by a subsequent user (e.g., a radiologist). This becomes particularly advantageous where the AI ​​system 220 / processor 1312 is operably connected to or contained within the imaging system 1301, such that the x-ray image 1701 is provided directly from the imaging system 1301 to the AI ​​system 220 / processor 1312 for use in generating the combined image 1702 and the checklist 1760. Upon receiving the total score of the combined image 1702 (such as a combined image 1702 generated in real time at the patient's bedside), the total score may be used to categorize the combined images 1702 of different patients so that a subsequent reviewer can review the various combined images 1702 based on the total score of each of the combined images 1702.

[0107] As an alternative to determining an overall score for the combined image 1702, or in addition to determining an overall score, the AI ​​system 220 / processor 1312 may provide a checklist 1760 for the combined image 1702 that includes a user-selectable response 1763 in the form of a prioritized review step 1772, which may have a single checkbox, or may have multiple checkboxes, such as Figure 8 1702 is shown with a "yes" box 1774 and a "no" box 1776. Regardless of or in addition to the total score determined for the combined image 1702, by selecting boxes 1774, 1776 in the priority review step 1772 with the input device 1322, the user input in step 1633 can mark the combined image 1702 for quick review by a subsequent reviewer when it is reviewed in step 1634.

[0108] In yet another exemplary embodiment of the present disclosure, when the AI ​​system 220 / processor 1312 performs an assessment of the x-ray image 1701 and the position of the tube 1705 and associated tube features (e.g., the tip 1707 and the side opening 1709), and the position and / or length of the tube 1705 relative to anatomical features (e.g., the carina 1712 and the diaphragm 1715) in steps 1606, 1608, and 1610 according to method 1600', the AI ​​system 220 / processor 1312 may change the checklist 1760 in step 1630 based on the analysis results of the AI ​​system 220 / processor 1312. For example, where the AI ​​system 220 / processor 1312 determines that the tube 1705 in the x-ray image 1701 satisfies the first two steps 1762 of the checklist 1760 associated with the clinical guidelines for placing the tube 1705 (e.g., the position of the tube 1705 relative to the carina 1712), but no clear determination can be made regarding other steps 1762 (e.g., the position of the tube 1705 relative to the diaphragm 1715), in step 1630, the AI ​​system 220 / processor 1312 may remove the first two steps 1762 from the checklist 1760 provided on the output device 1324. In an alternative embodiment, all steps 1762 may be presented to the checklist 1760, but certain steps 1762 (such as, as previously discussed, the first two steps 1762) are pre-populated by the AI ​​system 220 / processor 1312 and are marked or otherwise identified in a manner different from the unpopulated steps 1762, e.g., the pre-populated steps 1762 have a different color than the unpre-populated steps 1762, such as graying out the pre-populated steps 1762 while highlighting the unpopulated steps 1762. In this way, the AI ​​system 220 / processor 1312 may reduce the time required to review the combined image 1702 by focusing the reviewer's attention on the aspects of the combined image 1702 that need to be assessed by the reviewer. For example, if the AI ​​system 220 / processor 1312 has already pre-populated some or all of the steps 1762 in the checklist 1760, the user / reviewer need only verify the pre-populated steps 1762, thereby reducing the number of necessary clicks for the correct pre-populated steps 1762.

[0109] According to another exemplary embodiment of the present disclosure, in order to train the AI ​​system 220 / processor 1312 to perform the various analysis steps 1606, 1608, 1610, 1630 in the method 1600', the AI ​​system 220 / processor 1312 may be applied to an x-ray image 1701 including a tube 1705 represented therein to perform the method 1600'. The output from the steps 1606, 1608, 1610, 1630 of the method 1600' may be hidden from the user, while a checklist 1760 is provided to the user in step 1614 to obtain user input regarding an assessment of the placement of the tube 1705 within the x-ray image 1701 in step 1633. The output from the AI ​​system 220 / processor 1312 obtained in the method 1600' may then be reviewed and / or compared to the user input in step 1634 to verify and / or correct the operation of the AI ​​system 220 / processor 1312 when performing the method 1600'. Additionally, the results of the comparison of the output from the AI ​​system 220 / processor 1312 with the user input on the checklist 1760 may be a combination of demographic and other characteristics of the patient who is the subject of the x-ray image 1701 in order to assess the performance of the AI ​​system 220 / processor 1312 within a relevant subgroup within the method 1600'. Once the AI ​​system 220 / processor 1312 is trained to provide sufficiently accurate results consistent with the user input used for the training and validation of the AI ​​system 220 / processor 1312, the AI ​​system 220 / processor 1312 may be instantiated on the imaging system 1301.

[0110] In yet another exemplary embodiment of the present disclosure, the output of method 1600' generated by the AI ​​system 220 / processor 1312 may be provided to the user in a non-hidden manner along with the checklist 1760 in the form of pre-populated answers to the various steps 1762 in the checklist 1760 in step 1614. When presented with the combined image 1702 and the checklist 1760, the user may then review the answers to the steps 1762 inserted by the AI ​​system 220 / processor 1312 to verify and / or correct the answers in step 1633, thereby providing feedback to the AI ​​system 220 / processor 1312 when reviewing the user input in step 1634 for use in verifying and / or correcting the operation of the AI ​​system 220 / processor 1312 when executing method 1600'. In certain embodiments, the validation may be accomplished with a single input from the user that is used to validate all answers provided by the AI ​​system 220 / processor 1312 in the checklist 1760, or may be accomplished by independently verifying the answers provided for each step 1762 in the checklist 1760. By performing such validation on the output of the AI ​​system 220 / processor 1312 from the method 1600', the information provided regarding the combined image 1702 (i.e., the responses on the checklist 1760) will provide increased reliability to further reviewers (e.g., radiologists).

[0111] In either manner of training and validating the operation of the AI ​​system 220 / processor 1312 when executing method 1600', Fig.10 In the exemplary embodiment of training shown, in the exemplary joint evaluation or training method 1200, replicas 1230 of the global AI system 1220 may be copied from a central server 1250 and distributed to a plurality of different clinical sites 1240. The replicas 1230 may be trained at each site 1240 in a hidden, non-hidden, or combined manner to validate the individual replicas 1230 against x-ray images 1701 obtained at the clinical site 1240. Once validated at the individual sites 1240 against the various demographic characteristics and medical problems of the patients treated at each site 1240, the trained replicas 1230 or AI models 1260 may be returned to the central server 1250 for later instantiation on imaging systems at other clinical sites for use in association with subgroups of patients with demographic characteristics and / or medical problems similar to those used to train the replicas 1230, without any transmission of the x-ray images 1701 and associated patient data used to train the replicas 1230 outside of the clinical site 1240.

[0112] In addition to being able to be used by the AI ​​system 220 / processor 1312 to train and validate the operation of the method 1600', the responses provided to step 1762 of the checklist 1760 can be stored for purposes other than being associated with the combined image 1702. More specifically, referring to Fig. 9 , in step 1634, the responses in the checklist 1760 may be compiled and reviewed by the AI ​​system 220 / processor 1312 or other separate AI model or computing device (not shown), for example, connected via network 1326, to generate various feedback and summary tables (not shown) and statistics regarding the responses provided in the checklist 1760 during the review of the combined image 1702. The information provided in the feedback, summary tables, and / or statistics may be used to assess the effectiveness of user responses in review of past and future combined images 1702 for quality of care purposes, monitor the incidence of misplaced tubes or other imaging devices 1705 and enable the AI ​​system 220 / processor 1312 to monitor and evaluate the accuracy of future responses to the checklist 1760 by providing review or feedback on the user responses 1763, and / or provide recommendations regarding responses 1763 from the user, including but not limited to identifying unusual and / or inaccurate responses 1763 on the checklist 1760 and prompting the user to review the accuracy of the responses 1763 before completing the checklist 1760 when the combined image 1702 is presented to the user (such as at the patient's bedside).

[0113] In yet another exemplary embodiment of the present disclosure, Fig.11 is a block diagram of an embodiment of an imaging system 1301 that is designed to acquire raw image data and process the image data for display and review / verification in accordance with embodiments of the AI ​​system 220 / processor 1312 and method 1600' of the present invention. Those skilled in the art will appreciate that various embodiments of the present invention are applicable to many medical imaging systems that implement x-ray tubes, such as x-ray imaging systems or fluoroscopic imaging systems. Other imaging systems, such as computed tomography (CT) imaging systems and digital radiography (RAD) imaging systems that acquire volumetric image three-dimensional data, also benefit from the present invention. The following discussion of the x-ray imaging system 1301 is merely an example of one such specific implementation and is not intended to be limiting in terms of modality.

[0114] like Fig.11As shown, imaging system 1301 includes an X-ray tube or X-ray source 1340 configured to project an X-ray beam 1342 through a human subject or patient 1344. X-ray source 1340 may be a conventional X-ray tube that produces X-rays 1342 having an energy spectrum that typically ranges from thirty (30) keV to two hundred (200) keV. X-rays 1342 pass through patient 1344 and, after being attenuated, impinge on detector assembly 1346. Each detector module in detector assembly 1346 generates an electrical signal that represents the intensity of the impinging X-ray beam as it passes through patient 1344, and, therefore, represents the attenuated beam. In one embodiment, detector assembly 1346 is a scintillator-based detector assembly, however, it is also contemplated that direct conversion type detectors (e.g., CZT detectors, photon counting detectors, etc.) may also be implemented.

[0115] The processor 1312 receives signals from the detector 1346 and generates an image corresponding to the patient 1344 being scanned. The image processing system 1300 communicates with the processor 1312 to enable an operator to control scan parameters and view the generated images using an input device 1322. That is, the input device 1322 includes some form of operator interface, such as a keyboard, a mouse, a voice-activated controller, or any other suitable input device that allows an operator to control the imaging system 1301 and view reconstructed images or other data from the image processing system 1300 on an output device 1324 (such as a display). In addition, the input device 1322 allows the operator to store the generated images in a mass storage device 1328, which may include a hard drive, a floppy disk, an optical disk, etc. The operator may also use the input device 1322 to provide commands and instructions to the image processing system 1300 for controlling an X-ray source controller 1348 that provides power and timing signals to the X-ray source 1340.

[0116] Based on the foregoing, it should be appreciated that the above disclosed methods, apparatus, and articles of manufacture have been disclosed to monitor, process, and improve the operation of imaging and / or other medical systems using a variety of deep learning and / or other machine learning techniques.

[0117] Thus, certain examples facilitate image acquisition and analysis at the point of care, such as at the patient imaging site, via a portable imaging device. If an image should be retaken, further analysis should be performed immediately, and / or other criticalities explored sooner rather than later, the exemplary systems, devices, and methods disclosed and described herein can facilitate such actions to automate analysis, streamline workflow, and improve patient care.

[0118] Certain examples provide specially configured imaging devices that can acquire images and operate as a decision support tool at the point of care for a critical care team. Certain examples provide imaging devices that are used as medical devices to provide and / or facilitate diagnosis at the point of care to detect radiological findings, etc. The device can trigger a critical alarm for a radiologist and / or critical care team to immediately attract the patient's attention. The device enables triage of patients after a patient examination (such as in a screening setting), where a negative test allows the patient to go home, while a positive test will require the patient to be seen by a physician before going home.

[0119] In some examples, the mobile device and / or cloud offering enables a vendor-neutral solution to demonstrate point-of-care alerts on any digital x-ray system (e.g., fully integrated, upgrade kit, etc.). In some examples, embedded AI algorithms executed on a mobile imaging system (such as a mobile x-ray machine, etc.) provide point-of-care alerts in real-time during and / or after image acquisition, etc.

[0120] By hosting the AI ​​on the imaging device, the mobile x-ray system can be used in rural areas, for example, without a hospital information technology network, or even on a mobile truck that brings imaging to the patient community. In addition, if there is a long delay in sending the image to the server or cloud, the AI ​​on the imaging device can be executed instead and an output generated back to the imaging device for further action. Rather than having the x-ray technologist move to the next patient and the x-ray device no longer at the patient's bedside with the clinical care team, image processing, analysis, and output can occur in real time (or substantially in real time given a certain data transfer / retrieval, processing, and output delay) to provide relevant notifications to the clinical care team while the clinical care team and the device are still at or near the patient. For example, for trauma situations, treatment decisions need to be made quickly, and some examples mitigate the delays caused by other clinical decision support tools.

[0121] Mobile X-ray systems travel throughout the hospital to the patient bedside (e.g., emergency room, operating room, intensive care unit, etc.). Within the hospital, network communications may be unreliable in "dead" areas of the hospital (e.g., basements, rooms with electrical signal interference or blockages, etc.). For example, if the X-ray equipment relies on establishing Wi-Fi to push images to a server or cloud that is hosting the AI ​​model and then wait to receive the AI ​​output back to the X-ray equipment, the patient is at risk of unreliable critical alerts when needed. Additionally, if a network or power outage affects communications, the AI ​​operating on the imaging device can continue to act as an independent mobile processing unit.

[0122] Examples of alerts generated for general radiology may include critical alerts (e.g., for mobile x-rays, etc.), such as tube and wire placement, pleural effusion, lobar collapse, indwelling CVC guidewire, pneumoperitoneum, pneumoperitoneum, pneumonia, etc.; screening alerts (e.g., for fixed x-rays, etc.), such as tuberculosis, lung nodules, etc.; quality alerts (e.g., for mobile and / or fixed x-rays, etc.), such as patient positioning, cropped anatomy, inadequate technique, image artifacts, etc.

[0123] Thus, certain examples improve the accuracy of artificial intelligence algorithms. Certain examples take into account patient medical information as well as image data to more accurately predict the presence of critical findings, urgent findings, and / or other problems.

[0124] Although certain example methods, apparatus, and articles of manufacture have been described herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus, and articles of manufacture that reasonably fall within the scope of the claims of this patent.

[0125] Technical effects of the disclosed subject matter include providing systems and methods that utilize AI (e.g., a deep learning network) to determine whether a medically placed tube or line is correctly placed within a region of interest (e.g., relative to a reference marker or anatomical landmark). The systems and methods may provide real-time feedback that determines in a more accurate and faster manner whether a medically placed tube or line is misplaced. Thus, allowing for rapid intervention to move the tube or line to the proper position for patient safety, if necessary.

[0126] This written description uses examples to disclose the subject matter, including the best mode, and also to enable those skilled in the art to practice the subject matter, including making and using any devices or systems and performing any included methods. The patentable scope of the disclosed subject matter is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to fall within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insignificant differences from the literal language of the claims.

Claims

1. An image processing system (1300), the image processing system comprising: a. Display (1324); b. Processor (1312); and c. a memory (1328) storing processor executable code which, when executed by the processor (1312), causes: i. receiving an image (1701) of a region of interest of a patient, wherein a medical tube or line (1705) is disposed within the region of interest; ii. detecting the medical tube or wire (1705) within the image (1701); iii. detecting one or more reference markers (1709) within the region of interest within the image (1701), wherein the one or more reference markers (1709) are within the patient; iv. generating a combined image (1702) by superimposing an overlay (1703) on the image (1701) to indicate the position of the medical tube or line (1705) relative to the one or more reference markers (1709); v. generating a checklist (1760) for verifying the position of the medical tube or line (1705) relative to the one or more reference marks (1709); and vi. Presenting the combined image (1702) and the checklist (1760) on the display (1324).

2. The image processing system (1300) of claim 1, wherein generating the inspection checklist (1760) comprises: a. accessing a memory (1328) storing clinical guidelines for properly placing the medical tube or line (1705) relative to the one or more reference markers (1709); and b. Selecting a plurality of steps (1762) from the clinical guideline for use in the checklist (1760).

3. The image processing system (1300) of claim 2, wherein selecting the plurality of steps from the clinical guideline comprises selecting less than all of the steps (1762) in the checklist (1760).

4. The image processing system (1300) of claim 2, wherein selecting the plurality of steps from the clinical guideline comprises providing a user-selectable response (1764) associated with each of the steps (1762).

5. The image processing system (1300) of claim 4, wherein the user-selectable responses (1764) include a confidence score associated with each user-selectable response (1764).

6. An image processing system (1300) according to claim 5, wherein the processor executable code, when executed by the processor (1312), causes a confidence score for the combined image (1702) to be calculated based on the confidence scores of the user-selectable responses (1764) from the checklist (1760).

7. The image processing system (1300) of claim 4, wherein the user-selectable response (1764) comprises a priority review response (1772).

8. The image processing system (1300) of claim 4, wherein the processor executable code, when executed by the processor (1312), causes a determination to be made as to whether the medical tube or line (1705) is correctly positioned relative to the one or more reference markers (1709).

9. The image processing system (1300) of claim 8, wherein determining whether the medical tube or wire (1705) is correctly positioned within the image (1701) comprises employing artificial intelligence (220) to determine whether the medical tube or wire (1705) is correctly positioned within the image (1701).

10. An image processing system (1300) according to claim 9, wherein the processor executable code, when executed by the processor (1312), causes the determination made by the artificial intelligence (220) as to whether the medical tube or line (1705) is properly positioned within the image (1701) to be compared with the user-selectable response (1764) in the checklist (1760).

11. An image processing system (1300) according to claim 10, wherein comparing the determination made by the artificial intelligence (220) as to whether the medical tube or wire (1705) is properly positioned within the image (1701) with the user-selectable responses (1764) includes pre-selecting by the artificial intelligence (220) one or more user-selectable responses from the user-selectable responses (1764) in the checklist (1760).

12. A method for medical image processing, the method comprising: a. Receiving, via a processor (1312), an image (1701) of a region of interest of a patient, wherein a medical tube or line (1705) is disposed within the region of interest; b. detecting, via the processor (1312), the medical tube or wire (1705) within the image (1701); c. detecting, via the processor (1312), a plurality of reference markers (1709) within the region of interest within the image (1701), wherein each of the plurality of reference markers (1709) is within the patient; d. generating, via the processor (1312), a combined image (1702) by superimposing an overlay (1703) on the image (1701) to indicate the position (1707) of the medical tube or line (1705) relative to the one or more reference markers (1709); e. generating, via the processor (1312), a checklist (1760) for verifying the position (1707) of the medical tube or line (1705) relative to the one or more reference markers (1709); and f. causing, via the processor (1312), presentation of the combined image (1702) and the checklist (1760) on a display (1324).

13. The method of claim 12, wherein the step of generating the checklist (1760) comprises, via the processor: a. accessing a memory (1328) storing clinical guidelines for properly placing the medical tube (1705) relative to the one or more reference markers (1709); and b. Selecting the plurality of steps includes selecting the plurality of steps from the clinical guideline.

14. The method of claim 13, wherein the step of generating the checklist (1760) includes providing, via the processor (1312), user-selectable responses (1764) associated with each of the steps of the checklist (1760).

15. The method of claim 12, comprising determining, via the processor (1312), whether the medical tube or line (1705) is correctly positioned relative to the one or more reference markers (1709).

Images