Methods and systems for outputting pathological locations
By receiving patient data, applying machine learning models, and mapping them to a clinical knowledge database, the efficiency and accuracy issues of pathological location detection in existing technologies have been resolved, achieving efficient and accurate automated detection of pathological locations.
Patent Information
- Application Number
- CN202610672029.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-05-16
- Filing Date
- 2026-05-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies struggle to efficiently and accurately detect pathological locations in medical datasets, especially in complex medical images, and also raise concerns about patient privacy.
By receiving patient data, a trained machine learning model is applied to identify findings, which are then mapped to nodes in a clinical knowledge database to determine the pathological location, and the pathological location is output.
It enables more efficient and precise location of pathological sites, reduces data requirements, and improves the accuracy and efficiency of automated medical diagnosis.
Smart Images

Figure CN123118435A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to systems and methods for outputting discovered pathological locations. Specifically, embodiments of the present invention relate to using trained machine learning models for medical data processing, and particularly for medical image data processing. Background Technology
[0002] Detecting pathology in medical datasets is a widely used, critical, yet complex and therefore error-prone task within the medical data processing diagnostic workflow. In particular, automatically locating approximate pathological regions will be valuable for downstream medical diagnostic workflows.
[0003] Machine learning methods and algorithms are applied to medical data to generate insights and / or anatomical / pathological findings. Typically, such methods and algorithms can be run and / or trained on computing units. Machine learning methods have generally proven to be very versatile across a wide range of application areas. For example, machine learning methods are used to support decision-making in autonomous driving. Similarly, automated systems that provide medical diagnoses by processing medical data (such as medical images, patient data, and / or reports) rely on machine learning methods.
[0004] However, due to the complexity and sheer number of pathologies occurring in medical data, it is often impossible to efficiently extract the sought-after pathological locations when applying machine learning methods to medical data. Furthermore, it is difficult to accumulate sufficient pathological data for training machine learning models. In many cases, the inherent sensitivity of patient-related data raises significant patient privacy concerns.
[0005] One way to address this problem is to implement machine learning methods to detect organs and / or organ parts in medical datasets. Here, the overall appearance of organs is usually consistent enough to be detected even in the presence of pathology.
[0006] However, one problem with this is that detecting the location of an organ or part of an organ is insufficient to distinguish clinical findings located in pathological areas. For example, highlighting a patient's kidney to illustrate signs of swelling and inflammation in the area near the ureter is not precise enough for medical diagnostic management.
[0007] Known methods enable coarse estimations of the location of pathological findings. These methods can be adapted to navigate to the vicinity of a given description of the finding, accelerating human interaction within the workflow. However, these methods still lack the precision required for a fully automated workflow to output the pathological location of a finding. Summary of the Invention
[0008] Therefore, the object of embodiments of the present invention is to provide an improved method and system for detecting pathological locations to support medical diagnostic workflows. In particular, the object of embodiments of the present invention is to provide systems and methods that enable more efficient output of pathological locations discovered through anatomical examination.
[0009] This objective is achieved by a computer-implemented method according to embodiments of this application, an apparatus for implementing the computer-implemented method, a corresponding computer program product, and a computer-readable storage medium.
[0010] The technical solution according to the invention is described below with respect to the claimed device and the claimed method. The features, advantages, or alternative embodiments described herein can be similarly assigned to other claimed purposes, and vice versa. In other words, the claims concerning the inventive method can be improved by the description of the device or the claimed features. In this case, for example, the functional features of the method are implemented by the target units or elements of the device.
[0011] According to one aspect, a computer-implemented method for outputting pathological locations is provided. The method includes several steps. One step involves receiving and / or acquiring patient data. Another step involves applying a trained machine learning model to the patient data to determine one or more findings. Another step involves mapping one or more findings to nodes in a clinical knowledge database to determine one or more mapped findings. Nodes in the clinical knowledge database include reference findings and / or reference data. Another step involves determining the pathological location associated with one or more mapped findings, particularly based on the reference findings and / or reference data. A third step involves outputting the pathological location.
[0012] Preferably, a trained machine learning model can be provided for determining one or more findings in a patient dataset. The trained machine learning model can be received, particularly at a processing unit, and especially from a remote unit such as a server. Providing the trained machine learning model may include creating a local version and / or updating the machine learning model, particularly at the processing unit. Providing the trained machine learning model may include providing a log file, wherein the log file includes parameterizations of the trained machine learning model.
[0013] Typically, machine learning models are configured to provide a desired or predetermined type of output by processing a certain type of input data. Thus, machine learning models mimic human cognitive functions and other aspects of human thought. Specifically, by being trained on training data, machine learning models are able to adapt to new environments and detect and infer patterns. Other expressions for machine learning models can be machine-learned models, trained functions, trained machine learning models, trained mapping specifications, mapping specifications with trained parameters, functions with trained parameters, artificial intelligence-based algorithms, or machine learning algorithms.
[0014] Typically, the parameters of a machine learning model can be adapted through training to facilitate model updates. Specifically, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Furthermore, representation learning can be employed. In particular, the parameters of a machine learning model can be iteratively adapted through several training steps.
[0015] Specifically, the machine learning model may include neural networks, support vector machines, decision trees, and / or Bayesian networks, and / or the trained function may be based on k-means clustering, Q-learning, genetic algorithms, transformer networks, and / or association rules. Specifically, the neural network may be a deep neural network, a convolutional neural network, or a convolutional deep neural network. Furthermore, the neural network may be an adversarial network, a deep adversarial network, and / or a generative adversarial network. Additionally, the neural network may include a transformer network.
[0016] Machine learning models can be, in particular, any suitable language model. In one implementation, the language model can be a custom and / or specially trained language model for natural language processing, such as BERT (Bidirectional Encoder Representation from Transformer). In another implementation, the language model can be a pre-trained deep learning-based LLM (Large Language Model). The LLM can be based on a transformer architecture that uses a self-attention mechanism to capture long-range dependencies in the text. An example of a transformer-based architecture is GPT (Generative Pre-trained Transformer) with a multi-layer transformer decoder architecture, which can be pre-trained to optimize the next token prediction task and then fine-tuned for various downstream tasks using labeled data. GPT-based LLMs can be trained using reinforcement learning based on human feedback to perform various natural language processing tasks.
[0017] A pathological location can refer to a specific area within an anatomical structure in the human body. Anatomical structures can include one or more organs of the patient (e.g., the patient's lungs or liver) or areas between organs of the patient (e.g., muscles). In other words, a pathological location can be an anatomical structure, such as an organ or part of an organ. A pathological location can also refer to an area where a pathology or abnormality is detected in medical images. For example, a pathological location could be a kidney, a portion of a ureter, and / or a specific part of the kidney, showing signs of abnormal tissue swelling and inflammation.
[0018] Patient data (also known as patient datasets) can be associated with a specific patient. In other words, patient data may vary from patient to patient. Patient data may include medical information and / or data about the patient. Patient data may include medical image data. Medical image data may be stored in standard image formats, such as the Digital Imaging and Communications in Medicine (DICOM) format. Medical image data may be stored on a local site's memory or computer storage system, such as a Picture Archiving and Communications System (PACS). Whenever DICOM is mentioned herein, it should be understood that this refers to the "Digital Imaging and Communications in Medicine" (DICOM) standard, such as according to the DICOM PS3.1 2020c standard (or any later or earlier version of said standard).
[0019] According to some examples, patient data may include at least one medical image dataset, preferably multiple medical image datasets, depicting different parts of the patient's body. Therefore, the depicted body parts typically include multiple anatomical bodies and / or organs (including compartments or anatomical structures). As an example, a patient dataset may be or may include chest images. Chest images may show lung tissue, bones (e.g., the pleural cavity), the heart and aorta, lymph nodes, etc. However, patient-specific medical image data may also be received separately from the patient data.
[0020] Based on some examples, patient data may include at least one medical report, particularly a radiological report. For instance, a medical report may include a textual description of information related to the patient. The description may include information about body parts, findings, diagnoses, treatments, and / or any other medical information about the patient.
[0021] Medical reports can be in structured or unstructured form. Structured medical reports facilitate the exchange of patient information. However, the proposed method can also handle unstructured reports. Medical reports may include at least one of the following components:
[0022] Patient Information Section: This section includes the patient's demographic details, such as name, age, gender, and contact information.
[0023] Chief complaint section: This section records the main reasons why the patient sought medical help.
[0024] Past medical history section: This includes information about the patient's previous illnesses, surgeries, medications, and allergies.
[0025] The physical examination results section should include a comprehensive description of the patient's physical appearance, vital signs, and examination results.
[0026] Laboratory and Diagnostic Test Results: This section shows the results of any relevant tests or investigations performed on the patient.
[0027] Assessment and Diagnosis (sometimes also called Impression): This section outlines how healthcare professionals assess and diagnose a patient's condition.
[0028] Treatment Plan Section: This section details the recommended treatment options, including medications, procedures, and lifestyle changes.
[0029] Prognosis and Follow-up Section: Healthcare professionals outline the expected outcomes of the patient's condition and provide guidance on follow-up care.
[0030] Alternatively or additionally, patient data may include a dataset of two-dimensional pathological images depicting tissue portions of the patient, known as whole-slide images.
[0031] According to some examples, patient data includes image processing results associated with at least one medical image. In particular, the image processing results can be determined based on the medical image. Specifically, the image processing results can be or can include (validated) detection results of medical findings in the corresponding medical image, (validated) classification results of medical findings in the corresponding medical image, and / or segmentation results of the corresponding medical image.
[0032] According to some examples, patient data includes information about the patient, such as the patient's height, weight, age, and / or medical history data, such as appendectomy.
[0033] A (medical) discovery can indicate a patient's condition or pathology. The condition or pathology may be relevant to the patient's diagnosis. A discovery can include a description of the anatomical location and / or other aspects of the medical finding. A discovery can be a data object that includes multiple datasets and / or information associated with one or more medical discoveries. A (medical) discovery can include associated lateralization and / or additional contextual site descriptions. A discovery may also be referred to as the clinical discovery description phase. A discovery can include contextual lateralization and / or site descriptions describing the pathological location of the discovery. A discovery can include a pathological location.
[0034] Medical findings and / or descriptions of anatomical locations can be associated with anatomical structures that distinguish the patient from other patients. Medical findings can be located within different organs of the patient (e.g., within the patient's lungs or liver) or between the patient's organs. In particular, medical findings can also be associated with foreign bodies.
[0035] In particular, medical findings can be associated with vegetations (also known as “tumors”), especially benign vegetations, in situ vegetations, malignant vegetations, and / or vegetations with indeterminate / unknown behavior. In particular, medical findings can be associated with nodules, especially pulmonary nodules. In particular, medical findings can be associated with lesions, especially lung lesions.
[0036] (Medical) findings can include classification. Classification can involve identifying the type of finding and / or providing a classification based on multiple predefined categories such as benign or malignant.
[0037] Based on some examples, a mapped discovery refers to a discovery associated with one or more nodes in a clinical knowledge database. For this purpose, reference discoveries and / or reference data linked to the nodes can be mapped to the discovery. Mapping reference discoveries and / or reference data (or typically data) to discoveries can include adding information and / or data to discoveries and / or correcting information and / or data in discoveries. Mapped discoveries can include reference discoveries and / or reference data and discoveries. Mapped discoveries can be related to or based on reference discoveries and / or reference data and discoveries. Mapped discoveries can be stored in temporary storage. Discoveries, particularly the information defining the discoveries, can be adapted to the reference discoveries and / or reference data that generated the mapped discoveries. In other words, by mapping discoveries to nodes in the clinical knowledge database, the structured information in the database provides context and centralized data for the discoveries.
[0038] In one example, the clinical knowledge database (also known as the clinical database) is a large-scale AI-embedded database of clinical knowledge. The clinical knowledge database may include a clinical knowledge ontology. Medical features and / or information are mapped to nodes in the clinical knowledge ontology. For this purpose, graph models and / or language models can be used to provide the clinical knowledge database. The clinical knowledge database may include nodes that model clinical nomenclature and / or ontology.
[0039] Clinical knowledge databases can be based on standard ontology graphs, such as RadLex, UMLS (Unified Medical Language System), or any other suitable imaging ontology.
[0040] Optionally, a graph model can be created to store the relationships between nodes and edges in the clinical knowledge database to achieve a more indirect descriptive association. To generate embeddings or vector representations of medical features, an embedding model (e.g., text-ada-embedding-002 or even a fine-tuned BERT) can be used. In the graph model, links between ontology concepts can be stored as embeddings (e.g., the MR concept is a sub-concept of diagnostic imaging).
[0041] Optionally, in the language model, the medical features of the image and DICOM (Digital Imaging and Communication in Medicine) tags (e.g., modality, study date, view location, pixel array, study description) and their links to ontology concepts (e.g., RadLex identifiers) are converted into paragraph descriptions and stored as embeddings.
[0042] These two types of embeddings (from the language model and the graph model) can then be combined to find the closest match to the request.
[0043] In one implementation, the extracted features are also associated with corresponding nodes based on metadata, which can be non-standardized and potentially site-specific. For example, the extracted features may include image acquisition using an MR (magnetic resonance) protocol with contrast agent. The image acquisition tags and associated MR acquisition parameters are then associated with the corresponding nodes for subsequent retrieval.
[0044] Medical features can be associated with one or more corresponding nodes in a clinical knowledge database. In particular, nodes can be represented as linguistic embedding vectors of medical features and / or discovered descriptions and aliases.
[0045] Based on some examples, reference findings and / or reference data in a clinical knowledge database refer to standardized and / or structured medical information. Reference findings and / or data can be structured within nodes of a clinical knowledge database, specifically representing anatomical objects of interest, pathological locations, and / or particular medical conditions.
[0046] Specifically, reference findings can be predefined medical conditions and / or pathologies categorized and / or classified within a database. In other words, reference findings can serve as a benchmark or standard by which patient-specific findings can be compared. For example, reference findings may include detailed descriptions of various types of tumors, their typical locations, and associated symptoms.
[0047] In particular, reference data may include additional contextual information that can support the determination of the pathological location of the discovery. Reference data can encompass a broad range of medical knowledge, such as anatomical descriptions, typical imaging characteristics, and known correlations between different medical conditions. In other words, reference data can provide background information and context for the discovery. For example, reference data may include lateral and / or site-specific descriptions of the discovery and / or condition.
[0048] Acquiring and / or receiving patient data may include receiving stored data from a memory or computer storage system (e.g., a Picture Archiving and Communication System (PACS)). For example, additional patient data, such as recorded medical images, may be received directly from the image acquisition device. Patient data may be received by loading patient-specific medical data and / or information from the storage device or memory of a computer system, or by receiving patient datasets from a remote computer system.
[0049] Applying a trained machine learning model to patient data to determine one or more findings may include applying a pre-trained machine learning model to a patient-specific dataset for analysis. Analyzing the dataset may include extracting features from medical images, reports, or other relevant medical information included in the dataset. For example, patient data may include radiology reports. The trained machine learning model may be applied to radiology reports containing textual descriptions and / or phrases that include medical findings and / or diagnostic information. The machine learning model may process the reports to identify and / or extract the textual descriptions and / or phrases as relevant findings. Relevant findings may be prioritized. Relevant findings may be output, for example, as a (prioritized) list.
[0050] As another example, a trained machine learning model can analyze radiology reports and extract phrases indicating the presence of abnormalities, such as “pulmonary nodules,” “kidney swelling,” or “hepatitis.” These extracted findings are then prioritized based on their clinical significance and presented in a list format for further review by healthcare professionals.
[0051] Mapping one or more findings to nodes in a clinical knowledge database to determine one or more mapped findings may include identifying one or more nodes in the clinical knowledge database that match findings determined from patient data. Matching nodes can be identified by inputting the findings as requests into the database; these matching nodes are associated with the same or similar reference findings. Optionally, the matching nodes may be sorted. Nodes may present anatomical objects of interest that can be mapped to findings. If a matching node is found, the associated information and / or pathological location are mapped to the findings, resulting in the mapped findings. In this context, mapping can generally refer to adding information from the database to a finding and / or refining a finding based on information from the database.
[0052] Determining the pathological location associated with one or more mapped findings may include extracting and / or determining the pathological location from data included in and / or associated with the mapped findings. For this purpose, reference findings and / or reference data extracted from a clinical knowledge database may be used. For each finding, the pathological location may be determined.
[0053] For example, mapped findings may include multiple (optionally ordered) possible and / or related pathological locations. Determining pathological locations may include identifying the pathological location most likely associated with a finding. For example, a score may be used to determine the most likely pathological location.
[0054] Outputting pathological locations may include displaying the pathological location on a device such as a computer screen, and / or providing the pathological location for use in methods and / or data processing devices. For example, a pathological location may be visualized in a medical image by highlighting a specific area within an anatomical structure associated with the discovery. Alternatively or additionally, the pathological location may be output by including (particularly formatting) the pathological location in a reporting and / or data format for use in other medical systems, such as for further comprehensive analysis and / or recording methods.
[0055] Alternatively, outputting the pathological location may include providing the pathological location to downstream processing, such as for downstream image analysis processing. Specifically, image processing algorithms may be selected based on the pathological location. For example, different algorithms may exist specifically for analyzing different regions of the body, and the pathological location can be used as a decision aid for selecting the most suitable algorithm. Alternatively, images from multiple images may be selected and displayed based on the determined pathological location. Alternatively, rendering algorithms may be executed based on the determined pathological location. For example, the field of view and / or viewing angle of the rendering algorithm may be set based on the determined pathological location.
[0056] This method offers significant advantages by accurately locating pathological sites found in patient data, particularly unstructured patient data. It enables the synergistic combination of available medical data to provide an automated workflow for outputting pathological sites. The context of medical knowledge databases can be leveraged to include and base relevant medical information associated with the discovery, including contextual details such as laterality and site description. This method advantageously improves the automated medical diagnostic workflow for outputting relevant pathological sites. This allows for a more efficient diagnostic workflow in the healthcare setting. Furthermore, this method reduces the amount of data required to determine pathological sites.
[0057] There are various options for training the model, which will be further elaborated below. For example, supervised learning can be applied. This method involves training the model on a labeled dataset, where each input is paired with the correct output. The model learns to map inputs to outputs by minimizing the difference between its predictions and the actual labels. The advantage is high accuracy and reliability when large, high-quality labeled datasets are available.
[0058] Alternatively, or entirely, unsupervised learning can be used. In this approach, a model is trained on an unlabeled dataset. The model learns to recognize patterns and structures in the data without being explicitly instructed to look for anything. This model can leverage large amounts of unlabeled data, which are more readily available than labeled data. The training data can include large amounts of unlabeled text data from sources such as books, articles, and / or websites. The data can be collected from the internet without specific labels.
[0059] Based on some examples, a machine learning model can be trained by applying a training dataset to the model in order to determine the parameters of the machine learning model and / or to parameterize the machine learning model.
[0060] Based on some examples, a machine learning model can be updated by updating one or more parameters. Also, updates can be performed by applying an additional training dataset to the model and / or when, in particular, user-determined evaluation results indicate an increase in the model's error rate. In this way, the machine learning model can be continuously optimized.
[0061] Machine learning models (also known as "trained functions") learn by adjusting parameters that determine the mapping from input data to output data. In neural networks, these parameters are, for example, the weights or weighted parameters (e.g., edge weights) of individual layers and nodes. In supervised learning processing of trainable functions, backpropagation can be used. During training, the trained function is applied to the training input data to generate corresponding output values, the target values of which are known in the form of the training output data. The difference between the output values and the training output data can be used to introduce a cost or loss function as a measure of how well the trained function performs a given task. The training objective is to find the (local) minimum of the cost function by iteratively adjusting the parameters (e.g., edge weights) of the trained function. This optimization problem can be performed using stochastic gradient descent or other methods known in the art.
[0062] According to the implementation, the training dataset will include a medical training dataset and a training comparison dataset, and, depending on the configuration of the trained function, an associated empirically validated reference dataset and / or a validated (particularly relative) similarity metric. The validated reference dataset may be based on user annotations made from analysis or diagnoses based on the medical training dataset. The same applies to the validated similarity metric.
[0063] Therefore, according to some implementations, training a trained function may include applying the trained function to a medical training dataset or a training comparison dataset to generate output values, and comparing the output values with a validated reference dataset and / or a validated similarity metric. Based on this comparison, one or more parameters of the trained function may then be adjusted.
[0064] For example, a medical training dataset may include patient data. A validated reference dataset may include findings. In other words, a medical training dataset can be used as input data for a trained function. A validated reference dataset can be used as comparative data for the output data of a trained function. In other words, training data may include input data, particularly including, for example, reported patient data, and associated expected findings.
[0065] Based on some examples, a clinical knowledge database is generated by: receiving one or more medical images, extracting features from one or more medical images using one or more machine learning models, associating the extracted features with corresponding nodes in the clinical knowledge database, and outputting a clinical knowledge database with the extracted features associated with the corresponding nodes.
[0066] Nodes in a clinical knowledge database can be output along with the extracted features associated with the corresponding nodes. For example, a clinical knowledge ontology database can be output by storing it on a computer system's memory or storage device, or by transmitting it to a remote computer system and / or server.
[0067] According to one implementation, the computer-implemented method includes several additional steps. One step involves receiving one or more patient-specific images associated with and / or included in the patient data. Another step involves identifying one or more markers within the patient-specific images. A further step involves matching one or more markers with one or more mapped findings, such that the mapped findings include one or more matched markers.
[0068] Patient data may include at least medical / radiological reports. Patient data may include the patient's history and / or previous medical images. Patient-specific images may specifically refer to the most recent or current medical images associated with the patient. Mapped findings may include one or more matching tags.
[0069] According to some examples, one or more patient-specific images (also referred to simply as "images") include multiple medical images, medical image data, and / or medical image datasets.
[0070] Patient-specific images can include visual descriptions of anatomical objects of interest within the patient's body. Typically, medical images can depict any captured anatomical object, such as organs, bones, blood vessels, abnormalities, etc. Medical images can be recorded using any suitable modality and / or image acquisition device, such as CT (computed tomography), MRI (magnetic resonance imaging), US (ultrasound), X-ray, or any other medical imaging modality or combination of medical imaging modalities. Medical images can be 2D (two-dimensional) images, 3D (three-dimensional) volumes, or 4D (four-dimensional) volumes (3D plus time).
[0071] Medical image datasets can be used for medical image research. They can include datasets providing three dimensions in space, or three dimensions providing two dimensions in space and one dimension in time; two dimensions providing two dimensions in space; and / or four dimensions providing three dimensions in space and one dimension in time.
[0072] Medical image datasets can depict a patient's body parts in the sense that they contain three-dimensional image data of the patient's body parts. Medical image datasets can represent image volumes. A patient's body parts can be included within these image volumes.
[0073] Medical image datasets include image data, for example, in the form of two-dimensional or three-dimensional arrays of pixels or voxels. Such arrays of pixels or voxels can represent intensity, absorption, or other parameters as a function of three-dimensional position, and can be obtained, for example, through appropriate processing of measurement signals obtained from medical imaging modalities.
[0074] Medical imaging modalities correspond to systems used to generate or produce medical image data. For example, medical imaging modalities can be computed tomography (CT) systems, magnetic resonance (MR) systems, angiography (or C-arm X-ray) systems, positron emission tomography (PET) systems, ultrasound imaging systems, etc. Specifically, computed tomography is a widely used imaging method that uses “hard” X-rays generated and detected by a special rotating instrument. The resulting attenuated data (also called raw data) is presented by computational analysis software that produces detailed images of the internal structures of parts of the patient's body. The resulting set of images is called a CT scan, which can form multiple series of consecutive images presenting internal anatomical structures in cross-sections perpendicular to the body's axis. Magnetic resonance imaging (MRI), used to provide another example, is an advanced medical imaging technique that utilizes the effect of a magnetic field on the movement of protons. In an MRI machine, the detector is an antenna, and the signal is analyzed by a computer to create detailed images of the internal structures in any part of the human body.
[0075] Medical image datasets can include multiple images or image slices. Slices can each show a cross-sectional view of an image volume. Slices can comprise a two-dimensional array of pixels or voxels, which are image data. The arrangement of slices in a medical image dataset can be determined by the imaging modality or by any post-processing scheme used.
[0076] In addition, medical image datasets can be two-dimensional pathological image datasets depicting the tissue portions of a patient, i.e., so-called full-view slice images.
[0077] Based on some examples, medical image datasets may have already been acquired at a local site. The local site can be a medical institution and / or hospital, specifically including at least one medical imaging modality.
[0078] Medical image datasets can be stored in standard image formats such as the Medical Digital Imaging and Communications (DICOM) format, and stored on a local site's memory or computer storage system, such as a Picture Archiving and Communications System (PACS). Whenever DICOM is mentioned herein, it should be understood that this refers to the "Medical Digital Imaging and Communications" (DICOM) standard, such as according to the DICOM PS3.1 2020c standard (or any later or earlier version of said standard).
[0079] According to some examples, one or more patient-specific images include (in particular, local datasets and / or parameterized data associated with patient data).
[0080] Based on some examples, for each medical image data, the local dataset includes validated image processing results, in particular empirical detection results of medical findings in the corresponding medical image data, empirical classification results of medical findings in the corresponding medical image data, and / or segmentation of the corresponding medical image data.
[0081] According to some examples, parameterized data includes validated image processing results or their parameterization. Depending on the case, parameterization may include indications of medical imaging modalities (or modalities) using the acquired medical image data, imaging parameters used when acquiring the medical image data, medical findings included in the medical image data, and segmentation of objects included in the medical image data.
[0082] A marker can refer to a specific anatomical point and / or region within a patient-specific image. Markers can include various segmented organs, anatomical structures, and body parts. Markers can include points (particularly points described by coordinates relative to a patient-specific image) and / or regions (particularly regions described by edges and / or boundaries of areas within a patient-specific image). Specifically, markers can include image coordinate nodes.
[0083] One or more patient-specific images can be received and / or acquired, for example, directly from the image acquisition device, since the medical images are recorded as one or more patient-specific images at the image acquisition device. One or more patient-specific images can be received by loading previously acquired medical images from the storage device or memory of a computer system or by receiving medical images of a specific patient from a remote computer system.
[0084] Landmarks can be detected by applying segmentation algorithms to patient-specific images. Segmentation algorithms process patient-specific images to identify one or more landmarks. Segmentation algorithms can analyze patient-specific images against a predefined set of detectable landmarks to consistently identify anatomical structures across different medical images. Landmarks may include coordinates indicating and / or relating to anatomical structures in the medical image. In particular, landmarks may indicate pathological locations.
[0085] Matching one or more flags with one or more mapped findings can include comparing the location information (e.g., coordinates) of the mapped findings with the flags.
[0086] Within the mapping, the coordinates of a marker can be associated with a discovery. The highest-ranking matching marker associated with a patient-specific image can be determined. The determined discovery can be matched against markers representing anatomical structures detected in the patient-specific image. If a match is found, the location within the patient-specific image is connected to the discovery using the associated coordinates (location information within the patient-specific image). The coordinates can include spatial information (e.g., coordinates, paths, models, etc.). In one implementation, the coordinates can include 3D coordinates transformed to a specific view, such as reformatted, clipping plane, or surface views from multiple coordinate systems.
[0087] By correlating the location information of markers detected in patient-specific medical images with findings, pathological locations can be accurately determined. Furthermore, the spatial coordinates of the pathological locations in the images are associated with the findings for future analytical steps and / or methods.
[0088] Based on some examples, one or more flags that match one or more mapped findings can be output. Output flags may include outputting patient-specific images. Output flags may also be adapted to include only the flags, for example, by highlighting the flags in the image.
[0089] According to some examples, markers are determined by applying a trained machine learning model (specifically a feature encoder) to patient-specific images. According to some examples, markers consist of encoded features identified by the machine learning model based on a predefined set of markers. According to some examples, the machine learning model differs from and / or is independent of the trained machine learning model used to determine the discovery. This could mean that the machine learning model is trained independently of the trained machine learning model used to determine the discovery and / or includes a different architecture.
[0090] According to some examples, segmentation can be targeted at organs, sites, or other compartments of a patient's body. According to some examples, the segmentation steps can refer to obtaining the outline of the corresponding organ or compartment and / or delineating the organ or compartment from the remainder of the image data to determine one or more markers in the image. In particular, markers may include the delineated organ or compartment and / or may be located on the outline of the corresponding organ or compartment.
[0091] This review of the use of machine learning models, particularly transformer networks with feature encoders, references Vaswani et al.’s “Attention Is All You Need” published on June 12, 2017, at arXiv:1706.03762, the contents of which are incorporated herein by reference in their entirety.
[0092] As some examples, off-the-shelf generative AI functions, such as DALL-E or Midjourney, can be used. As other examples, custom generative AI functions based on a transformer architecture can be used, trained on pairs of parameterizations and corresponding (real) data.
[0093] According to one implementation, the computer-implemented method includes several additional steps. One step involves receiving one or more maps comprising a plurality of reference markers. Another step involves matching one or more mapped findings with the reference markers, such that the mapped findings include one or more matched reference markers.
[0094] Mapped discoveries may include one or more matching reference flags. Matching reference flags with mapped discoveries may include adding information and / or data of the reference flags to the mapped discoveries and / or modifying information and / or data of the mapped discoveries to include links to the reference flags.
[0095] Atlases can be canonical human atlases and / or graphical models that include spatial ontological relationships between a target pathological location and nearby detected and / or potential locations, marked with anatomical landmarks. Atlases may include human models. In other words, atlases represent reference patients and / or human models. Atlases can be generated and / or received from databases and / or sources. An example of an atlas is a human reference atlas available from HubMAP. Each atlas is typically pre-annotated with a densely sampled set of spatial (anatomical) landmarks that match landmarks in an established ontological concept.
[0096] Anatomical landmarks (also known as reference landmarks) can be landmarks included in the atlas. Each anatomical landmark can consist of 3D coordinates relative to the human model in the atlas. Anatomical landmarks can refer to specific anatomical points and / or regions within the human model in the atlas. Anatomical landmarks can include various segmented organs, anatomical structures, and body parts. Anatomical landmarks can include points (particularly points described by nodes in the graphical model of the atlas) and / or regions (particularly regions described by coordinates relative to the human model in the atlas). The graphical model can represent connections between nodes. In particular, landmarks can include a densely sampled set of spatial anatomical landmarks that match landmarks in the ontological concept.
[0097] Maps can be received from multiple sources, such as databases and / or servers. One or more maps can be received by loading particularly previously generated maps from storage devices or memory on a computer system, or by receiving maps from remote storage devices such as cloud storage. For example, a human reference map can be received from HubMAP. HubMAP, or the Human Biomolecular Mapping Project, is a project that provides a comprehensive map of the human body at the molecular level. This publicly available map can be used to map various biological structures and their functions to the human body.
[0098] Specifically, matching one or more mapped findings with anatomical landmarks in the received atlas is performed only if the pathological location cannot be determined from one of the detectable landmarks in the patient-specific image. This can be a special case because detectable landmarks in patient-specific images typically only cover sparse sampling of general anatomical features (e.g., organ parts) and not pathology. In other words, the atlas is used to find the nearest matching landmark within the human atlas.
[0099] To match one or more mapped findings with anatomical landmarks, more than one atlas can be used, such as atlases specifically for males and females or pediatrics. As a sub-step, matching one or more mapped findings with anatomical landmarks in the atlas can include a graph-based search, where the edges of the graph model are connections within a medical ontology representing anatomical space-ontology relationships. As another sub-step, the nearest anatomical landmark is determined based on the relevant structure given by the nodes within the graph model.
[0100] For example, the right kidney anatomical region is a node in the graph, where the upper and lower parts or parenchyma of the kidney are related nodes within the graph model. Therefore, by performing a graph search, the nearest anatomical landmarks are identified within related nodes (specifically body structures and / or regions) at different levels to match the findings of the mapping.
[0101] Atlases can be used to precisely locate pathological findings and provide contextual information for determining pathological locations. Furthermore, atlases support combinations of various medical data sources and information.
[0102] Based on some examples, atlases are received and / or selected from multiple atlases based on patient data.
[0103] According to an implementation, the computer-implemented method includes several additional steps. One step involves receiving one or more patient-specific images associated with patient data. Another step may involve receiving one or more atlases including multiple reference markers. Another step may involve determining anchor markers in the patient-specific images based on the reference markers. Another step may involve associating the mapped findings with anchor markers within the patient-specific images, such that the mapped findings include one or more matching anchor markers.
[0104] Mapped discoveries may include one or more matched anchor tags. Matching anchor tags with mapped discoveries may include adding anchor tag information and / or data to the mapped discoveries and / or modifying the information and / or data of the mapped discoveries to include links to the anchor tags.
[0105] Specifically, anchor marks refer to marks detected and / or identified in both atlases and patient-specific images. In other words, anchor marks are common marks derived from both atlases and patient-specific images.
[0106] One or more patient-specific images can be received, for example, directly from the image acquisition device, because the medical images are recorded as one or more patient-specific images at the image acquisition device. One or more patient-specific images can also be received by loading previously acquired medical images from the storage device or memory of a computer system, or by receiving specific patient medical images from a remote computer system.
[0107] Maps can be received from multiple sources, such as databases and / or servers. One or more maps can be received by loading (especially previously generated) maps from storage devices or memory of a computer system or by receiving maps from remote storage devices such as cloud storage devices.
[0108] Determining anchor marks in patient-specific images based on reference marks may include comparing reference marks in the atlas with detectable marks in the patient-specific image. Matching reference marks and / or anatomical marks in the atlas with images may include graph-based search, where edges of a graph model are mapped to unstructured marks in the patient-specific image.
[0109] Additionally, in another step, the mapped findings can be correlated with anchor marks within patient-specific images. This step may include estimating the pathological location of the findings based on (shared) anchor marks present in the atlas and patient-specific images that represent the spatial relationship between the findings and the anchor marks.
[0110] Advantageously, this can provide insights into relationships with anchor marks from atlases. These relationships can be used to estimate the location of target pathologies based on anchor marks in patient-specific images.
[0111] According to the implementation method, landmarks and / or anchoring landmarks in a patient-specific image are determined based on at least one of imaging modality, acquisition parameters, acquisition orientation, image appearance, anatomical field of view, or classification and detection.
[0112] By identifying landmarks and / or anchoring landmarks in patient-specific images based on at least one factor such as imaging modality, acquisition parameters, acquisition orientation, image appearance, anatomical field of view, or classification and detection, adaptive and accurate landmark identification becomes possible. By taking these various factors into account, the detected landmarks are aligned with specific features of the medical image. This improves the adaptability and reliability of landmark detection processing, leading to more accurate localization of pathological locations.
[0113] According to one implementation, mapping one or more findings to nodes in a clinical knowledge database includes several steps. One step involves resolving patient data and / or findings into one or more vectors. Another step involves performing a vector search between vectors representing patient data and vectors representing nodes in the clinical knowledge database to determine the similarity between one or more vectors and nodes in the clinical knowledge database.
[0114] Vectors, in particular, refer to the numerical representation of patient data. Various attributes, such as medical history and / or reports, can be converted into numerical form and organized, for example, into arrays or lists of numbers. Nodes in a clinical knowledge database can include data representing the node or vectors associated with the node. A vector representing a node in a clinical knowledge database can include the data and information of a node, or be associated with a node in numerical form.
[0115] Parsing patient data and / or findings into one or more vectors can include converting the patient data into numerical form and / or organizing it into vectors. Parsing patient data into one or more vectors can also include transforming the information into arrays and / or lists of numbers. For example, a transformer or encoder can be used to encode lateralization and / or site information into numerical values.
[0116] Vector search can include comparing vectors representing patient data with vectors representing nodes in a clinical knowledge database to find similarities. In other words, it measures the proximity and / or similarity of vectors in a multidimensional space. Performing a vector search can include determining cosine similarity, Euclidean distance, or other metrics that quantify the similarity between vectors. By performing a vector search, nodes in the clinical knowledge database that match the patient data are identified, particularly considering comparison or similarity metrics such as scores, priority schemes, and / or thresholds.
[0117] Vector representations enable comparisons of patient data with other vectors representing nodes in a clinical knowledge database. By performing vector similarity searches, the similarity between patient data vectors and clinical knowledge vectors and / or nodes can be determined to identify and discover relevant medical conditions and / or data associated with them.
[0118] According to the implementation method, identifying one or more mapped pathological locations includes at least one of the following:
[0119] - Based on body structure lateralization and / or site filtering, location pointer data is associated with one or more mapped findings.
[0120] - Prioritize the position pointer data to determine the first position pointer.
[0121] - Match the mapped discovery of at least a first location pointer with the pathological location associated with the node in the clinical knowledge database to determine the approximate target location as the pathological location.
[0122] Location pointer data refers to specific data points used to identify and / or locate anatomical structures and / or pathological findings within the human body. In other words, location pointer data is data used to accurately map findings to specific pathological locations. Location pointer data may include reference points within the human body. Reference points represent points that are well-defined and generally accepted in medical practice, such as lateralization or location information of body structures.
[0123] Body structure and / or laterality are information indicating whether the discovery is on the left or right side of the body. Site information is specific details about the exact anatomical location, such as organs or tissues.
[0124] One step involves filtering location pointer data associated with one or more mapped findings based on body structure laterality and / or location. This step may include examining the location pointer data to determine if the finding is on the left and / or right side of the body and / or to identify a specific anatomical location. For example, if a lesion is found in the left hippocampus, the location pointer data is filtered to ensure that only information about the left side of the brain and hippocampal region is considered in the matching step. In other words, by filtering the location pointer data, it is narrowed down to the most relevant information.
[0125] Another step is to prioritize the filtered position pointer data to determine the first position pointer. Prioritization can include ranking the filtered position pointer data based on its relevance and importance to a specific anatomical location. For example, if multiple data points are available for the left hippocampus, the most relevant and accurate data point is selected as the first position pointer by estimating the distance to the found data.
[0126] Another step involves matching the mapped, discovered first location pointers with pathological locations associated with nodes in the clinical knowledge database to determine approximate target locations as pathological locations. Filtered and / or priority-ranked location pointers are compared with reference data in the clinical knowledge database. For example, the first location pointer of a lesion in the left hippocampus is matched with similar pathological locations in the database to determine the most accurate target location.
[0127] Matching location pointer data with mapped findings can improve the localization of pathological locations by ensuring that data is filtered and prioritized based on body structure laterality and specific anatomical sites.
[0128] According to the implementation method, mapping one or more findings to nodes in the clinical knowledge database includes validating the findings and / or ranking the findings based on the clinical knowledge database.
[0129] Validating findings by mapping one or more findings to nodes in a clinical knowledge database can include comparing the identified findings with reference data in the database to identify potentially erroneous and / or irrelevant findings. Multiple findings can be identified by applying a trained machine learning model to patient data. Validating findings can include narrowing down multiple findings to exclude others. The trained machine learning model can, for example, identify erroneous, irrelevant, and / or numerous findings. Validating findings can include discarding reference data from the database and / or confirming findings.
[0130] Ranking findings based on a clinical knowledge database can include prioritizing, particularly validated, findings to determine a relevance score. For example, findings can be ranked based on their similarity to reference data in the clinical knowledge database. For instance, if multiple data points are available for a finding, that finding is selected as the primary finding. Another prioritization can be determined based on the scores of data points in the databases associated with the finding.
[0131] By validating and / or sorting findings and mapping one or more findings to nodes in a clinical knowledge database, the data on the findings can be enhanced or incorrect or irrelevant information can be excluded.
[0132] According to the implementation method, applying a machine learning model to patient data to determine one or more findings includes identifying a portion of the patient data to which a trained neural network is applied.
[0133] According to the implementation method, patient data includes radiological reports. Determining a portion of the patient data specifically includes determining a portion of the radiological reports.
[0134] Determining a portion of patient data may depend on a local site. A local site may, for example, refer to an organization that will deploy computer-implemented methods. In particular, a local site may refer to a healthcare environment or organization, or facility, such as a hospital, laboratory, clinic, university, or association of one or more of the foregoing.
[0135] A portion of patient data can refer to various specific sections or segments of patient data. These sections can include medical image data, such as two-dimensional or three-dimensional arrays of pixels or voxels representing different body parts, like chest images showing lung tissue and the pleural cavity. Sections can include radiological reports or portions of radiological reports, such as textual descriptions of body parts, findings, diagnoses, and treatments. Additionally, sections can include pathological image data, such as full-field slices depicting tissue portions. By focusing on these specific sections, healthcare professionals can more effectively apply trained neural networks to analyze and interpret data, leading to more accurate and efficient diagnostic and treatment plans.
[0136] In other words, advantageously, by focusing on specific aspects, trained machine learning models can be effectively applied to analyze patient data.
[0137] According to an implementation, the output pathological location includes the pathological location in a patient-specific image output on a display device.
[0138] According to the implementation method, outputting pathological locations includes providing pathological locations for use in data processing methods.
[0139] Outputting pathological locations on a display device may include displaying medical images and associated pathological locations on an output device such as a monitor or computer display. Outputting pathological locations on a display device may also include rendering medical images and pathological locations based on calculated appearances, and then transmitting the rendered data as a graphic signal to the display device.
[0140] Providing pathological locations for data processing includes providing them via files, particularly log files. Additionally, pathological locations may be formatted as report and / or data formats. Providing pathological locations may include generating files or reports that include the coordinates and / or description of the pathological locations. These files or reports may be integrated into and / or stored in electronic health records (EHRs).
[0141] Outputting pathological locations in patient-specific images on a display device and / or outputting pathological locations for data processing methods can improve diagnostic workflows.
[0142] Based on some examples, log files may include pathological locations. Based on some examples, log files are formatted according to the JSON standard. JSON stands for JavaScript Object Notation and is an open standard file and data exchange format that uses human-readable text to store and transmit data objects consisting of property-value pairs and arrays. This improves method interoperability.
[0143] According to the implementation method, outputting the pathological location includes several steps:
[0144] - Viewing preferences for receiving medical images
[0145] - Generate a display layout for medical images enhanced with pathological locations based on viewing preferences.
[0146] - Output display layout.
[0147] For example, viewing preferences for medical images can be defined based on at least one of anatomical display orientation, windowing techniques, rendering modes, and / or any other suitable viewing preferences.
[0148] Display layouts can specifically include information and / or settings for output devices, particularly for displaying medical images. Display layouts for one or more medical images are automatically generated based on viewing preferences. A language model can translate viewing preferences into internal viewing implementations and settings for output devices such as displays. The display layout system can provide translation of user-input instructions in clinical nomenclature into relevant medical images.
[0149] Viewing preferences can be received specifically via user input. User input may include a description of the medical image and / or viewing preferences for the medical image. Viewing preferences can be extracted from text / voice commands by a language model parser. User input may include language prompts for the viewing preferences. Viewing preferences can be derived from language prompts. Viewing preferences can be sent as language prompts to output devices and / or interface units to translate the viewing preferences into rendering parameters. Language prompts can be prompts or inputs to a language model that defines a particular rendering input scheme. For example, a language prompt may indicate to the language model that the renderer can support viewing the patient in anterior, left, and right positions, and instruct the language model to translate the user-input viewing preferences into one of these positions.
[0150] Display layouts for one or more medical images can be automatically generated based on viewing preferences. These layouts can include rendering parameters. In one example, viewing preferences extracted from text / speech commands by a language model parser are sent as language prompts to the language model to translate the viewing preferences into rendering parameters. The renderer can then generate the display layout based on these parameters.
[0151] A renderer included in the output device can generate a display layout based on rendering parameters. For example, the display layout can be output by displaying it on a computer system's display device, by storing it in the computer system's memory or storage device, or by transmitting it to a remote computer system. The implementations described herein can be implemented, for example, in PACS (Picture Archiving and Communication System) viewer applications. For instance, when a user loads patient data in a PACS viewer application, the user can provide input in a chat window, and the system can automatically select medical images and generate a display layout for displaying the medical images to the user.
[0152] By receiving viewing preferences from the user and outputting image data according to the display layout, an efficient system can be provided for displaying pathological locations associated with the findings.
[0153] Based on some examples, patient-specific images can be enhanced to show pathological locations. Enhancing patient-specific images may include overlaying and / or highlighting specific areas where abnormalities or pathological findings are identified. By enhancing patient-specific images, clinicians can improve the visualization of pathological localization.
[0154] According to one aspect, an apparatus is provided. The apparatus includes means for receiving patient data. The apparatus includes means for applying a trained machine learning model to the patient data to determine one or more findings. The apparatus includes means for mapping one or more findings to nodes in a clinical knowledge database to determine one or more mapped findings. The apparatus includes means for determining pathological locations associated with one or more mapped findings. The apparatus includes means for outputting the pathological locations.
[0155] Devices for receiving patient data may include temporary storage units. A temporary storage unit is typically a designated space and / or container within a storage device and / or computing unit, configured to store data items or information for a defined period of time. After the defined period, the data items or information may be moved to a permanent storage location, further processed, and / or discarded. For example, a temporary storage unit may store received patient data for up to 48 hours before deleting the data.
[0156] A device for applying a trained machine learning model to patient data, a device for mapping one or more findings to nodes in a clinical knowledge database, and / or a device for determining pathological locations associated with one or more mapped findings may include a computing unit and / or be implemented by the computing unit.
[0157] The computing unit and / or processing unit may be a data processing system or part of a data processing system. Such a data processing system may include, for example, a cloud computing system, a computer network, a computer, a tablet computer, a smartphone, etc. The computing unit may include hardware and / or software. The hardware may include, for example, one or more processors, one or more memories, and combinations thereof. One or more memories may store instructions for performing the method steps according to the invention. The computing unit may include temporary storage units. The hardware may be configurable by software and / or operable by software. Typically, all units, subunits, or modules may exchange data with each other, at least temporarily, for example, via a network connection or a corresponding interface. Therefore, the individual units may be positioned separately from each other. Furthermore, the computing unit may be configured as an edge device.
[0158] The device for outputting pathological locations may include an interface unit. The interface unit may include an interface for exchanging data with one or more local clients, for example, via the Internet. The interface unit may also be adapted to interface with one or more users of the system, for example, by displaying processing results to a user (e.g., in a graphical user interface).
[0159] The device may be specifically configured to perform one or more steps of the methods described in its various aspects and / or examples.
[0160] The advantages of combining the methodological aspects and the example descriptions can also be achieved through the corresponding configured device components.
[0161] According to some examples, the device may also include means for providing clinical databases and / or machine learning models. For example, means for providing clinical databases may include database units and / or cloud storage devices. Means for providing machine learning models may include storage units, server units, and / or cloud storage devices.
[0162] According to another aspect, the present invention relates to a computer program product comprising instructions that, when executed by a computer, cause the computer to perform at least one step of a method according to one or more of the above-described method aspects and examples. In other words, the present invention relates to a computer program product comprising program elements that, when loaded into the memory of a computing unit, cause the computing unit to perform steps according to one or more of the above-described method aspects and examples.
[0163] According to another aspect, the present invention relates to a computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform at least one step of a method according to one or more of the above-described method aspects and examples. In other words, the present invention relates to a computer-readable medium storing program elements that can be read and executed by a computing unit to perform steps according to one or more method aspects and examples when the program elements are executed by the computing unit.
[0164] Implementing the present invention through a computer program product and / or a computer-readable medium has the following advantages: existing providing systems can be easily adapted through software updates to operate as proposed in the present invention.
[0165] A computer program product can be, for example, a computer program, or include another element adjacent to the computer program itself. This other element can be: hardware, such as a storage device on which the computer program is stored, a hardware key for using the computer program, etc.; and / or software, such as documentation or software keys for using the computer program. A computer program product may also include development materials, a runtime system, and / or a database or library. A computer program product can be distributed across several computer instances. Attached Figure Description
[0166] The features, characteristics, and advantages of the invention described above, as well as the ways in which they are implemented, will become clearer and more readily understood from the following description of the embodiments, which will be described in detail with reference to the accompanying drawings. The following description does not limit the invention to the included embodiments. The same components, parts, or steps may be labeled with the same reference numerals in different drawings. Generally, the drawings are not drawn to scale.
[0167] In the following figures:
[0168] Figure 1 A computer-implemented method for outputting pathological locations, according to an embodiment, is schematically depicted.
[0169] Figure 2 A computer-implemented method for outputting pathological locations, according to an embodiment, is schematically depicted.
[0170] Figure 3 A computer-implemented method for outputting pathological locations, according to an embodiment, is schematically depicted.
[0171] Figure 4 An exemplary data flow diagram related to a computer-implemented method for outputting pathological locations, according to an embodiment, is schematically depicted.
[0172] Figure 5An exemplary data flow diagram related to a computer-implemented method for outputting pathological locations, according to an embodiment, is schematically depicted.
[0173] Figure 6 An embodiment of the device according to the embodiment is illustrated schematically. Detailed Implementation
[0174] Figure 1 , Figure 2 and Figure 3 A computer-implemented method for outputting pathological locations is described. Figure 4 and Figure 5 The corresponding data flow is shown in the diagram. The method comprises several steps. The order of the steps does not necessarily correspond to the step numbers, but can vary between different embodiments of the invention. Furthermore, a single step or a series of steps can be repeated. According to the method of the invention, steps can be performed individually.
[0175] In step S10, specifically at device 1, patient data PD is received. According to an embodiment, patient data PD is received from a patient database PDB. Specifically, the patient database PDB can be configured to store and / or process any medical data of the patient, particularly patient data PD. According to an embodiment, patient data PD is at least partially recorded at medical system 10. Patient data PD can be received directly from medical system 10. Specifically, medical system 10 can be configured to record the patient's medical image data.
[0176] According to some examples, patient data PD includes medical image data, particularly patient-specific images (PIs). According to some examples, patient data PD includes at least one medical report, particularly a radiology report.
[0177] In step S20, a machine learning model NN can be applied to the patient data PD. The machine learning model NN can be received at device 1. The machine learning model NN can be provided to device 1, specifically to and managed by the computing unit BM of device 1.
[0178] The machine learning model NN can be easily trained, meaning it can be deployed at device 1 according to the learned task. According to one implementation, the learned task may include automating the processing of patient data PD to derive (medical) findings AD. Specifically, the machine learning model may be configured to process the patient's medical report to determine the medical findings. In particular, the machine learning model NN may be a language processing model for extracting relevant findings information from the text and / or portions of the medical report.
[0179] Specifically, machine learning models (NNs) can generate text summaries of patient data (PDs) as findings. For example, if the patient data PD includes image data, a visual transformer that has been trained to analyze medical image data and incorporate findings into textual impressions (like those found in medical reports) can be used.
[0180] Based on some examples, a medical report may include information about parts of the patient's torso, as well as findings including lesions in the patient's lungs or liver. By applying a machine learning model (NN) to the medical report, the NN can identify atypical atrophic lesions (AD), specifically lesions in the patient's lungs or liver.
[0181] The machine learning model NN can also be trained on local data. This causes the machine learning model NN to be updated (locally). According to some examples, such training can occur during runtime (e.g., when a user reviews the processing results of the machine learning model NN). For example, a radiologist can reject or accept an AD (Advanced Disorder) found by the machine learning model NN. Furthermore, a radiologist can add ADs found that the machine learning model NN does not determine. User input U can be used as ground values for further optimization (i.e., training the machine learning model NN). The ground values, together with the underlying local data, can form the training data for the machine learning model NN.
[0182] Optionally, the machine learning model NN can operate based on natural language prompts. These prompts can be viewed as instructions or control commands for the machine learning model NN. Such prompts can be generated based on patient data in optional sub-steps. To some extent, step S20 can be viewed as a translation step that translates patient data PD into a set of AD findings, upon which other steps of the method can take action, and / or other functional and / or machine learning model actions, based on these AD findings.
[0183] At step S30, one or more discovery ADs are mapped to nodes in the Clinical Knowledge Database (CDB) to determine one or more mapped discoveries. Nodes include reference discoveries and / or reference data. In other words, the discovery AD is compared with reference discoveries and / or reference data represented as nodes in the CDB. Mapping the discovery AD to nodes in the database may include determining reference discoveries and / or reference data corresponding to the discovery AD. In other words, medically relevant reference discoveries and / or reference data are determined for the determined discovery AD. For example, a discovery AD including lesions in a patient's liver may be compared and / or mapped with reference discoveries and / or reference data including liver lesions and / or liver cancer.
[0184] A node that maps a discovery discovery (AD) to a database (CDB) may include adding at least a portion of the identified reference discovery and / or reference data to the discovery AD to receive the mapped discovery. The reference discovery and / or reference data may specifically contain pathological information, particularly pathological location (FS).
[0185] Specifically, reference findings and / or reference data can be used to validate the discovery alerts (ADs) determined by the machine learning model (NN). The discovery ADs can be corrected, modified, and / or edited based on the reference findings. Information from the reference findings can be added to the information of the discovery ADs.
[0186] At step S40, a pathological location FS associated with one or more mapped findings is determined. Determining the pathological location FS associated with one or more mapped findings may include extracting pathological location information or a description from information and / or data associated with one or more mapped findings. In other words, data and / or information about mapped findings may be searched for pathological location information. If pathological location information cannot be determined, the pathological location FS may include determining the next nearest pathological location. Determining the next nearest pathological location may include determining a body region and / or portion adjacent to one of the findings AD. Steps S20 and S30 may be repeated for at least one body region and / or portion adjacent to one of the findings AD.
[0187] Finally, at step S50, the pathological location FS can be output. Specifically, the pathological location FS can be output to one or more other (medical) devices. Specifically, the pathological location FS can be output to the user U, particularly via the display device AE. Specifically, the pathological location FS can be output to provide (training) data to an updated machine learning model NN.
[0188] Go to Figure 2 At step S60, one or more patient-specific image PIs associated with or included in the patient data may be received. The patient data received in step S10 may be, in particular, radiological reports and / or contain historical / previous medical images, in which case the (current and / or latest) medical image PIs associated with the reports may be received and / or retrieved from the patient data PD.
[0189] For example, patient data (PD) can include reports on lesions in a patient's lungs or liver. Associated patient-specific images (PIs) can show portions of the patient's torso, including the lungs or liver.
[0190] Based on other examples, patient-specific images (PIs) can include digital pathology images of the patient. Detecting atopic dermatitis (AD) can involve segmenting digital pathology images based on one or more tissue types.
[0191] At step S61, one or more landmarks LM within the patient-specific image PI are determined. For example, a segmentation model can be used to determine the landmarks LM. Determining the landmarks LM may include identifying specific anatomical points and / or regions within the patient-specific image PI. In other words, the determined landmarks LM may include various segmented organs, anatomical structures, and body parts and / or regions.
[0192] At step S62, one or more flags LM are matched with one or more mapped findings. Mapped findings may include one or more matched flags LM. Mapped findings may be corrected, modified, and / or edited based on the flags LM. Information from the flags LM may be added to the information of the mapped findings.
[0193] The marker LM identified in a patient-specific image PI and / or the location information and / or data included in the identified marker LM can be compared with the marker LM and / or location information and / or data included in and / or associated with the mapped findings.
[0194] If a match is found, the LM flag, particularly location information and / or data, can be added to the mapped discovery.
[0195] At step 51, viewing preferences for medical images (particularly patient-specific images PI) can be received (specifically from user U). User input can be received at the interface device AE. Viewing preferences can be determined based on user input.
[0196] In one implementation, viewing preferences are defined based on at least one of the anatomical display orientation of patient-specific images, windowing techniques, or rendering modes.
[0197] In step S52, a display layout for the medical images (particularly patient-specific images, PIs) is generated. The medical images may be enhanced with pathological locations based on viewing preferences.
[0198] At step S53, the display layout is output, specifically to the user U. In particular, an enhanced patient-specific image including the pathological location FS can be output to the user via a display device.
[0199] In step S31, the patient data PD is parsed into one or more vectors. Based on one or more vectors, the database CDB can search for similar data. In other words, one or more vectors can present a search string to the database CDB.
[0200] At step S32, a vector search is performed between the vector representing patient data (PD) and the vector representing a node in the clinical knowledge database (CDB). The similarity between one or more vectors and nodes in the CDB is determined.
[0201] In other words, a similarity vector search is performed within the Clinical Knowledge Database (CDB). A similarity vector search can involve comparing vectors representing patient data points (PDs) with vectors representing nodes in the CDB to find similarities. Essentially, it measures the proximity and / or similarity of vectors in a multidimensional space.
[0202] Vector search can involve, for example, determining cosine similarity, Euclidean distance, or other metrics that quantify the similarity between vectors. By performing this vector search, nodes in a clinical knowledge database that match patient data can be identified, taking into account comparison or similarity metrics such as scores, priority schemes, or thresholds.
[0203] Go to Figure 3 In step S63, anchoring markers in a patient-specific image PI are determined based on reference markers from the atlas.
[0204] Anchor markers can be determined by comparing reference markers from a graph with detectable markers within a patient-specific image (PI). Anchor marker determination can involve graph-based comparison methods where edges of a graph model are mapped to unstructured markers in the patient-specific image.
[0205] By identifying these anchoring markers, spatial relationships can be established between findings and anatomical structures within a patient-specific image PI, especially in the absence of matching markers for the findings.
[0206] At step S64, the mapped findings are associated with anchor marks identified within the patient-specific image PI. One or more matched anchor marks can be added to the mapped findings. In other words, the mapped findings can include matched anchor marks.
[0207] The pathological location FS of an atrial lesion (AD) can be estimated based on shared anchoring landmarks present in both atlas and patient-specific image (PI). By utilizing the spatial relationship between the AD and the anchoring landmarks, the target pathological location FS can be accurately determined without directly matching the landmark LM in the patient-specific image (PI).
[0208] This step can be implemented by a machine learning model to output correlation results between the findings and the anchored flags. The machine learning model is configured to analyze patterns and / or relationships within the findings and anchored flags to determine the correlation results.
[0209] At step S81, one or more spectra including multiple reference markers are received.
[0210] The atlas can be used as a comprehensive diagram and / or map including detailed anatomical reference landmarks. The atlas can be sourced from database A or a server. The atlas can be pre-generated and / or stored in memory or a storage unit (e.g., a computing unit). The atlas can be retrieved from memory or storage unit A and / or loaded into processing unit BM.
[0211] As an example, at step S81, a human reference map can be received from HubMAP at the processing unit BM.
[0212] Subsequently, at step S82, one or more mapped findings are matched with reference markers in the received atlas. Step S82 can be specifically performed if the pathological location FS cannot be determined from detectable markers in a patient-specific image PI. Specifically, in this case, the nearest matching reference marker can be found within the graph of the human model and / or the atlas using the atlas. Matching may include using multiple atlases, such as those for different sexes or age groups, and / or may include graph-based search. Specifically, the edges of the graph model of the atlas represent anatomical space-ontology relationships. By performing a graph search, the nearest anatomical reference marker is determined within relevant nodes at different levels.
[0213] At step S71, location pointer data associated with one or more mapped findings is filtered based on body structure lateralization and / or location. The location pointer data may include information and / or data regarding the body structure lateralization and / or location of one or more findings. The mapped findings may be analyzed to determine whether the finding is on the left or right side of the body and / or to identify specific anatomical regions within the body.
[0214] For example, if a lesion is found in the left hippocampus, the location pointer data is filtered to ensure that only information about the left side of the brain and hippocampal region is considered in the matching step. In other words, filtering narrows down the data to the information that allows for accurate localization of the pathological finding.
[0215] At step S72, the position pointer data is prioritized to determine the first position pointer. Specifically, if one or more position pointers are filtered out in step S71, the filtered position pointers can be prioritized based on a correlation source. The correlation source can be determined by a trained function. Step S72 can also be a sub-step of S71 and / or entirely comprised of S71.
[0216] For example, if multiple location pointers are available for a specific anatomical site or region, the most relevant and accurate location pointer can be the source of high correlation. If an association with this location pointer is found, it is selected as the first location pointer for estimating the target pathological location.
[0217] At step S73, at least a first position pointer of the mapped discovery is matched with the mapped discovery to determine the approximate target location as a pathological location FS. The mapped discovery may include pathological locations associated with nodes in the Clinical Knowledge Database (CDB). These pathological locations FS of the mapped discovery can be examined using the first position pointer to obtain matching pathological locations FS.
[0218] Step S73 may include comparing filtered and prioritized location pointer data with reference data associated with the discovery in the Clinical Knowledge Database (CDB). The most accurate target pathological location (FS) is identified for each discovery. For example, if a first location pointer indicates a discovery in the left hippocampus, that location pointer is matched with similar reference pathological locations in the database associated with that discovery as a mapped discovery to determine the precise target pathological location (FS).
[0219] Figure 4 and Figure 5 Depicting Figure 1 , Figure 2 or Figure 3 The data flow diagram illustrates an implementation of the method. The data flow corresponding to an implementation of the method is shown. The method includes several steps. The order of the steps does not necessarily correspond to the depicted data flow. Furthermore, a single step or a series of steps may be repeated.
[0220] Figure 4 The diagram illustrates a medical system 10 as the starting point of the data flow. Medical images captured by the medical system 10 can be directly provided as patient data (PD) to a machine learning model NN. A patient database (PDB) connected to the medical system 10 can also provide patient data PD to the machine learning model NN. Medical images captured by the medical system 10, along with associated medical information (e.g., reports), can be stored in the patient database (PDB). The machine learning model NN can be applied to the patient data PD to determine discovery AD. Discovery AD can be provided to a processing unit BM. The processing unit BM can be connected to a clinical database (CDB). The processing unit BM can be configured to determine and / or output pathological locations (FS) associated with the discovery AD. Pathological locations (FS) can be output to a user U via an interface device AE, particularly a display device. The user U can provide input to the interface device AE.
[0221] Figure 5Additional image data streams are shown. Patient-specific images (PIs) that can be included in patient data (PDs) are provided. The patient-specific images (PIs) contain markers (LMs). In addition to detecting atopic lesions (ADs), the markers (LMs) can be provided to the processing unit (BM). The processing unit (BM) is connected to the clinical database (CDB) and may also be connected to the atlas database (A). The markers (LMs) can also be provided to a machine learning model (NN2), which may be connected to or included by the processing unit (BM). The interface unit (AE) can be configured to output an image (PA) containing the pathological location (FS).
[0222] Figure 6 An example device 1 for outputting pathological locations (FS) is depicted. Device 1 is adapted to perform, for example, as described in the reference... Figures 1 to 3 The method according to one or more embodiments is further described.
[0223] Device 1 includes a temporary storage unit RM, a processing unit BM, and an interface unit AE. These units can be connected via a network.
[0224] Device 1 can create, train, update, and apply machine learning-based neural network (NN) models. These NN models can typically be configured to provide discovery of adverse events (ADs) based on patient data (PD). This can include detection discovery in medical image data, detection discovery in report data, and annotation discovery, such as in orientation or landmark detection.
[0225] The units RM, BM, and AE of Device 1 can be hosted on a server, which can be a cloud server or a local server. However, the units RM, BM, and AE of Device 1 can also be implemented using any other suitable computing device. Device 1 can connect to a database DB. The database DB may include a clinical knowledge database CDB, a patient database PDB, and / or a graph database A. The database DB can be configured to centrally store training data for evaluating the discovery AD of the machine learning model NN.
[0226] Device 1 and / or processing unit BM may be a computing unit or include a computing unit. A computing unit may include one or more processors and working storage devices. One or more processors may include, for example, one or more central processing units (CPUs), graphics processing units (GPUs), and / or other processing devices. A computing unit may also include a microcontroller or integrated circuit. Alternatively, a computing unit may include a real or virtual computer group, such as a so-called “cluster” or “cloud.” Working storage devices may include one or more computer-readable media, such as RAM for temporarily loading data. Working storage devices may also store information accessible by one or more processors to perform method steps according to one or more embodiments described herein. Working storage devices are also temporary storage units RM or include temporary storage units RM.
[0227] The interface unit AE may include any suitable component for interfacing with one or more networks, including, for example, a transmitter, receiver, port, controller or other suitable component.
[0228] The database DB can be implemented as a cloud storage device. Alternatively, the database DB can be implemented as a local or extended storage device, specifically at the location of device 1.
[0229] Databases (DBs) can be part of hospital information systems (HIS), radiology information systems (RIS), clinical information systems (CIS), laboratory information systems (LIS) and / or cardiovascular information systems (CVIS), image archiving and communication systems (PACS), etc.
[0230] As an example, data items in a database (DB), particularly patient data (PD), can be associated with laboratory test results and / or pathology data and / or medical imaging data, electronic medical records, and any combination thereof.
[0231] Where meaningful, various embodiments or aspects and features thereof may be combined or interchanged with each other without limiting or expanding the scope of the invention. Where applicable, advantages described with respect to one embodiment of the invention are also advantageous to other embodiments of the invention.
Claims
1. A computer-implemented method for outputting pathological locations, comprising: Receive (S10) patient data, (S20) Apply a trained machine learning model to the patient data to determine one or more findings. The one or more findings are mapped (S30) to nodes in the clinical knowledge database to determine one or more mapped findings, wherein the nodes include reference findings and / or reference data. Determine (S40) the pathological location associated with the one or more mapped findings, and Output (S50) the pathological location.
2. The computer-implemented method according to claim 1, wherein, The method further includes: Receive (S60) one or more patient-specific images associated with the patient data, Identify (S61) one or more markers within the patient-specific image; Match the one or more flags with the one or more mapped findings (S62) such that the mapped findings include one or more matched flags.
3. The computer-implemented method according to any one of the preceding claims, wherein, The method further includes: Receive (S81) includes one or more spectra including multiple reference markers. The one or more mapped findings are matched with the reference flags (S82) such that the mapped findings include one or more matched reference flags.
4. The computer-implemented method according to any one of the preceding claims, wherein, The method further includes: Receive (S60) one or more patient-specific images associated with the patient data, Receive (S81) includes one or more spectra including multiple reference markers. Based on the reference marker, determine (S63) the anchor marker in the patient-specific image. The mapped findings are correlated with anchor marks within the patient-specific image (S64) such that the mapped findings include one or more matched anchor marks.
5. The computer-implemented method according to claim 2 or 4, in, The markers and / or anchoring markers in the patient-specific images are determined based on at least one of imaging modality, acquisition parameters, acquisition orientation, image appearance, anatomical field of view, or classification and detection.
6. The computer-implemented method according to any one of the preceding claims, wherein, Mapping the one or more findings to nodes in the clinical knowledge database includes: The patient data is parsed (S31) into one or more vectors. A vector search (S32) is performed between the vector representing the patient data and the vector representing the node of the clinical knowledge database to determine the similarity between the one or more vectors and the node of the clinical knowledge database.
7. The computer-implemented method according to any one of the preceding claims, wherein, Determining the pathological location associated with one or more of the mapped findings includes at least one of the following: Based on body structure lateralization and / or site filtering (S71), location pointer data associated with the one or more mapped findings, The filtered position pointer data is prioritized (S72) to determine the first position pointer. The mapped discovery of at least the first location pointer is matched with the pathological location associated with the node in the clinical knowledge database (S73) to determine the approximate target location as the pathological location.
8. The computer-implemented method according to any one of the preceding claims, wherein, Mapping the one or more findings to nodes in the clinical knowledge database includes: Verify the findings, and / or The findings are ranked based on the aforementioned clinical knowledge database.
9. The computer-implemented method according to any one of the preceding claims, wherein, Applying the trained machine learning model to the patient data to determine one or more findings includes: A portion of the patient data is identified, particularly radiological reports included in the patient data, and the trained neural network is applied to that portion.
10. The computer-implemented method according to any one of claims 2 to 9, wherein, Outputting the pathological location includes: outputting the pathological location in a patient-specific image on a display device.
11. The computer-implemented method according to any one of the preceding claims, wherein, Outputting the pathological location includes providing the pathological location for use in a data processing method.
12. The computer-implemented method according to any one of the preceding claims, wherein, The output of the pathological location includes: Receive (S51) medical image viewing preferences, (S52) Generate a display layout for a medical image enhanced with the pathological location based on the viewing preferences. Output (S53) the display layout described above.
13. An apparatus comprising: Device for receiving patient data A device for applying a trained machine learning model to the patient data to determine one or more findings. A means for mapping the one or more findings to nodes in a clinical knowledge database to determine one or more mapped findings. A means for determining the pathological location associated with the one or more mapped findings, and A device for outputting the pathological location.
14. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform at least one step of the method according to any one of claims 1 to 12.
15. A computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform at least one step of the method according to any one of claims 1 to 12.
