Medical image analysis techniques related to medical problems

Abstract

The invention relates to a technique for extracting medical image analysis data related to a medical question. A method (100) comprises the step of receiving (S102) a first medical data set related to a patient. The first medical data set comprises first medical image data acquired by means of a first medical imaging modality related to a medical question. A second medical data set related to the patient is received (S104). The second medical data set comprises context information related to the medical question. At least part of the received (S102) first medical data set is converted (S106) into a target format. The at least partly converted (S106) first medical data set and the received (S104) second medical data set are jointly analyzed (S108) in view of the medical question. Medical image analysis data related to the medical question is extracted (S110) from the result of the joint analysis (S108).

Description

Technical Field

[0001] This invention relates to a technique for extracting medical image analysis data related to medical problems, and particularly includes a method, a computing device, a unified platform, and a computer program product. Background Technology

[0002] Using prior images and data in conjunction with currently acquired imaging is crucial for image interpretation and the formation of meaningful diagnoses, and therefore for radiological reporting. Given the severe global shortage of radiologists, various efforts have been made on the topic of autonomous interpretation of radiological images. Recently, we have also seen greater openness to these efforts from the clinical community. While current AI systems have demonstrated state-of-the-art performance, surpassing expert radiologists in certain situations and tasks, their conversion to clinical routine remains problematic, particularly in the absence of radiologists, thus creating a significant demand for autonomous or semi-automated systems. However, these systems may suffer from the drawback that they do not adequately and comprehensively consider the context of information. In contrast to traditional automated data processing, radiologists and other clinical experts are able to place current imaging findings within the context of (1) non-imaging data (patient history, EHR, demographic information, etc.) and (2) prior images.

[0003] Comparing multiple examinations is challenging for traditional AI systems, primarily because these systems suffer performance degradation due to large shifts in data distribution. The multimodal nature of radiological workflows, such as prior examinations performed in a different imaging modality, further exacerbates this problem. The multimodal nature of clinical examinations in a patient's history often involves different imaging modalities. Furthermore, sometimes prior examinations exist only in the form of clinical reports, without any imaging.

[0004] Image homogenization techniques, largely borrowed from the computer vision field, have achieved moderate success in the literature. However, prior information is sometimes obtained in the form of clinical reports without any imaging. AI systems traditionally struggle to effectively process the diverse range of information sources, a task readily available to human experts. Summary of the Invention

[0005] Therefore, one object of the present invention is to provide a solution for (particularly at least semi-automatically) jointly analyzing health-related information of a patient in its context (e.g., in the patient's medical history). Alternatively or additionally, the objective is to facilitate the extraction, combination, and / or embedding of contextual health-related data. This data may originate from, for example, longitudinal and / or multimodal studies, such as image data and / or text data from various medical imaging modalities. Alternatively or additionally, supporting medical practitioners in improving diagnostic and / or treatment planning based on the collective medical history of patients is a publicly disclosed problem.

[0006] This objective is achieved by a method, computing device, unified platform, and computer program (and / or computer program product) for extracting medical image analysis data related to medical problems, according to the appended independent claims. Advantageous aspects, features, and embodiments, along with advantages, are described in the dependent claims and the following description.

[0007] In the following description, the solution according to the invention relates to both the claimed method and the claimed computing device. Features, advantages, or alternative embodiments mentioned with respect to the method can be assigned to other claimed objects (e.g., computer programs or devices, particularly unified platforms, computing devices, or computer program products), and vice versa. In other words, the system, apparatus, or device can be improved with features described or claimed in the context of the method, and vice versa. In this case, the functional features of the method are correspondingly embodied by the structural units of the apparatus or device or system, and vice versa. The method may refer to a software implementation, while the device may refer to a hardware implementation (having a spatial physical structure). Generally, in computer science, software implementations and corresponding hardware implementations (e.g., as embedded systems) are equivalent. Thus, for example, method steps for “storing” data can be performed using storage units and corresponding instructions for writing data to storage devices. To avoid redundancy, although the device may also be used in alternative embodiments described with reference to the method, these embodiments will not be explicitly described again with respect to the device. In principle, the corresponding device or apparatus claims are configured to perform the claimed method.

[0008] Regarding the method, a method (particularly computer-implemented) for extracting medical image analysis data relevant to a medical problem is provided. The method includes the step of receiving a first medical dataset related to a patient. This first medical dataset includes first medical image data acquired using a first medical imaging modality relevant to the medical problem. The method further includes the step of receiving a second medical dataset related to (e.g., the same or similar, for comparison purposes) a patient. This second medical dataset includes contextual information relevant to the medical problem. The method further includes the step of converting at least a portion of the received first medical dataset into a target format (and / or a format that both the first and second medical datasets can process, such as a shared space). The method further includes the step of jointly analyzing the at least partially converted first medical dataset and the received second medical dataset in light of the medical problem. The method further includes the step of extracting medical image analysis data relevant to the medical problem from the results of the joint analysis.

[0009] The present invention improves the extraction of medical image analysis data (and / or patient health-related information; also known as: patient-related proposed medical features) from two or more medical datasets, such as the presence and / or type of anomalies. Alternatively or additionally, performance degradation that traditionally occurs, for example, due to large shifts in data distribution, can be avoided.

[0010] The present invention enables the automatic reading of medical imaging data and / or non-medical imaging data, which may include historical (and / or previously acquired, and / or longitudinal) medical data and / or current (and / or more recently acquired) medical data. Therefore, for example, as a result of image analysis, changes in a patient's health status (e.g., abnormal progression, such as tumor growth or lesion shrinkage) can be detected and / or monitored.

[0011] Extracting medical image analysis data from the first medical dataset can be improved by leveraging the contextual information (also known as contextual data and / or contextual knowledge) included in the second medical dataset.

[0012] The first medical dataset includes first medical image data, particularly first medical image data acquired by a first medical scanner, such as using magnetic resonance imaging (MRT), computed tomography (CT), ultrasound (US), positron emission tomography (PET), X-ray imaging, and / or others. The second medical dataset may include second medical image data, particularly second medical image data acquired by a second medical scanner, which may also be of a different modality type relative to the first scanner. The first and second medical scanners may be independent or different, such as in terms of their medical image format and / or their technical specifications (e.g., including different magnetic field strengths for two different MRT scanners, and / or single-energy vs. dual-energy CT scanners). The first medical dataset (and optionally the second medical dataset) may include preliminary medical image analysis data and / or results of a perception task. This perception task may include detection (e.g., organ detection and / or object detection, such as the detection of stents or other implants) and / or segmentation (e.g., semantic segmentation).

[0013] Any (e.g., first and / or second) medical imaging data may include raw data acquired by a medical scanner using a corresponding medical imaging modality. Alternatively or additionally, any (e.g., first and / or second) medical imaging data may include preprocessed and / or reconstructed image data acquired by a medical scanner.

[0014] Alternatively or additionally, the first medical dataset and / or the second medical dataset may include textual data (also known as: clinical data and / or clinical reports), such as medical examination reports (hereinafter referred to as: medical reports; also known as: clinical reports), annotations of corresponding medical images, laboratory results and / or medications taken by patients (prescription and / or over-the-counter).

[0015] Contextual information may be relevant to the patient's medical problem, and particularly to the same medical problem for which the first medical dataset was acquired. Alternatively or additionally, contextual information may be patient-specific. Contextual information may include second medical imaging data and / or textual data (also referred to as: non-imaging data). Second medical imaging data may, for example, include previous studies related to the same patient and / or data that can be obtained using a second medical imaging modality, which may be the same as or different from the first medical imaging modality. Alternatively or additionally, textual data included in the second medical dataset may include electronic health records (EHRs), medical reports, examination results and / or laboratory results in text form (also referred to as: text format), and / or the medical history of the corresponding patient (i.e., the study subject of the first medical dataset).

[0016] Contextual information may include the results of artificial intelligence (AI) tools, such as the results of large language models (LLMs) of text data. Alternatively or additionally, one or more LLMs may be used to extract information from non-image sources (and / or text formats, and / or text data) that may be relevant to a clinical diagnosis.

[0017] Contextual information may alternatively or additionally include data as the output of one or more of a plurality of image analysis algorithms. For example, in one embodiment, prior studies, particularly those concerning the same medical problem (especially those concerning the same patient from whom the first medical image data was received), may be considered. These prior studies may have been processed using at least one image analysis algorithm (e.g., an AI tool, such as one for detection and / or segmentation). The output of this AI tool, preferably provided in a target format (e.g., text), may be provided as contextual information to be processed jointly (and / or in combination) with the received first medical dataset. Alternatively, if the output of the image analysis algorithm for the second medical dataset is not provided in the target format, a second transformation may be applied to convert it to the target format so that the first and second medical datasets can be processed jointly or together. In one example, the target format may be text, which enables the use and application of LLM. Alternatively or additionally, the target format may include segmentation (e.g., segmentation for a specific organ based on a medical problem).

[0018] The first and second medical datasets can be associated with the same patient. Alternatively or additionally, the first and second medical datasets can be associated with two different patients, such as for inter-cohort and / or intra-cohort comparisons of characteristics (e.g., characteristics associated with a particular type of lesion).

[0019] The first and second medical datasets can be obtained at different time instances. For example, the first medical dataset may include more recent medical data than the second medical dataset.

[0020] Any medical dataset can provide context, such as context in textual data. Alternatively or additionally, the first and second medical datasets can be obtained using different medical scanners, such as medical scanners of different modalities.

[0021] The first medical dataset can be received in the source format. Alternatively or additionally, the second medical dataset can be received in the target format.

[0022] This technique can be modal agnostic and / or applicable to multimodal extraction and / or extraction of health-related information from medical image analysis data. For example, by converting the received first medical dataset into a target format, different types of medical image data can be directly compared. For instance, three-dimensional (3D) medical images can be compared with two-dimensional (2D) medical images, such as by selecting (and / or constructing) the corresponding 2D projection (also known as: view).

[0023] In some embodiments, the target format may be a format that both the first medical dataset and the second medical dataset can at least partially convert to, and / or a format that both the first medical dataset and the second medical dataset can process or understand. Alternatively or additionally, conversion to the target format may involve the use of a shared space (e.g., a shared space shared between the first medical dataset and the second medical dataset).

[0024] Transforming at least a portion of the first medical dataset into the target format may include applying imaging AI (AI stands for Artificial Intelligence, embodied, for example, by neural network systems such as CNNs, U-Nets, and / or Transformers) and / or perception task models and / or image processing tools. The imaging AI, perception task models, and / or image processing tools may be configured for specific medical tasks, including organ detection (and / or object detection), and / or for (particularly semantic) segmentation. Alternatively or additionally, the imaging AI, perception task models, and / or image processing tools may be medical imaging modality-specific. Any (particularly modality-specific) imaging AI may be narrow artificial intelligence (ANI).

[0025] Alternatively or additionally, converting the first medical dataset into the target format may include applying LLM to the text data.

[0026] Alternatively or additionally, converting at least a portion of the first medical dataset into the target format may include converting at least a portion of the first medical image data (e.g., from the source format) into the target format.

[0027] According to some examples, the step of converting at least a portion of a first medical dataset into a target format may include determining the target format based on a second medical dataset (optionally, the source format of the second medical dataset), based on a clinical question, and / or based on a joint analysis step (optionally, the underlying model and / or analysis algorithm used in the joint analysis step, and further optionally, the format requirements of the input data for the underlying model and / or analysis algorithm used in the joint analysis step). In this way, the target format can be identified according to the requirements of further processing, which improves compatibility and efficiency.

[0028] Applying LLM to text data can include transforming the text data according to a predetermined medical ontology, and / or converting the text data into predetermined medical terms.

[0029] The first and second medical datasets transformed by the joint analysis may include at least a portion of the first medical image data from the joint analysis and contextual information.

[0030] The first and second medical datasets transformed by joint analysis can be performed with the help of a base model and / or analytical algorithms.

[0031] The LLM, analytics algorithm, base model, and / or perception task model can be trained, particularly through supervised learning using training pairs on medical datasets annotated by medical experts. Alternatively or additionally, when the method is performed, for example, based on acceptance, rejection, and / or ratings by medical experts on the extracted medical image analysis data, the LLM, analytics algorithm, base model, and / or perception task model can be fine-tuned.

[0032] Alternatively or additionally, training LLMs, analytics algorithms, base models, and / or perception task models can utilize self-supervised learning (SSL) and / or (especially two-branch) self-distillation.

[0033] The base model can be configured to jointly analyze textual data and / or medical imaging data. Alternatively or additionally, the base model can be configured to determine the medical data included in both first and second medical datasets, which can be jointly analyzed. Alternatively or additionally, the base model can be configured to extract medical questions from the first medical dataset and to carefully examine a second medical dataset, or both medical datasets, based on information related to the medical questions. For example, an indication of the medical question can be extracted from the metadata of (e.g., the first) medical dataset. Alternatively or additionally, an indication of the medical question can be extracted from the medical imaging protocol used to acquire medical imaging data.

[0034] Joint analysis of medical datasets may include selecting a subset of medical data from each individual medical dataset and focusing on medical data from two medical datasets that may be associated with a medical problem. For example, at least one of the first and / or second medical datasets may include medical imaging data of a patient's chest cavity, but the medical problem may involve only one of the imaging organs, such as only the lungs.

[0035] The extracted medical image analysis data (and / or health-related information) may include newly detected abnormalities and / or the progression of abnormalities (e.g., deterioration or improvement of health status).

[0036] According to one embodiment, medical image data, such as that obtained from a scanner (e.g., first and / or second), can be received, and image analysis can be performed upon receipt, for example, by using an AI tool such as one for converting at least a portion of a first medical dataset into a target format. In a further embodiment that can be combined with previous embodiments (e.g., for additional first and / or second medical image data), processed (e.g., first and / or second) and / or converted medical image data can be received, and joint analysis can be performed solely by the techniques of the present invention.

[0037] The objective of this invention (e.g., including a unified platform; also referred to as: a mutually understandable space, or simply: space) is to provide a unified framework that allows for comprehensive understanding, for example, for comparison. For instance, in one embodiment, a first medical dataset (also referred to as: source data) and a second medical dataset (also referred to as: target data) exist in two separate jurisdictions, and due to privacy concerns, the actual (specifically source and / or target) data may not be shared. In this embodiment, an AI tool can run locally (e.g., for the second medical dataset) and populate fields in this mutually understandable space (also referred to as: the unified platform). The populated fields of this space can then be communicated to another location for comparison (e.g., compared with the first medical dataset, or vice versa for both), rather than communicating the actual (specifically target and / or source) data. In another embodiment, data (e.g., the first medical dataset, or at least first medical image data, and / or the second medical dataset, or at least contextual information) can be received, and image analysis can then be performed using an AI tool.

[0038] In one embodiment, medical image information and contextual information may be used in combination to train computing devices or distributed computing systems and / or computing clouds, such as those provided in [4], for joint analysis of first and second medical datasets (and / or first medical image data and contextual information included therein, respectively).

[0039] The present invention offers advantages over existing technologies in terms of computational speed and / or reduced requirements for computational resources (particularly processing resources and / or memory space). Alternatively or additionally, the extracted medical image analysis data can be improved, for example, in terms of smaller error bars regarding the estimated size and / or shape of lesions.

[0040] The first (and / or second) medical dataset may include metadata (e.g., metadata in a predetermined format), such as the patient’s personal data, such as name, date of birth, sex, height, potential amputation and / or potential implants; and / or the date and / or type of the patient’s examination, such as imaging and / or non-imaging.

[0041] Alternatively or additionally, textual data may be provided in accordance with a predetermined medical ontology. Optionally, textual data may include patient-related medical history; medical conditions, particularly those previously reported and / or derived from examinations; medications, such as prescription and / or over-the-counter; and / or textual annotations of medical imaging data (e.g., measurements and / or estimated dimensions of lesions).

[0042] Annotations for medical imaging data can be provided from the perspective of predetermined organ categories and / or object categories (e.g., for implants, such as stents, pacemakers, dental implants, hip implants, knee implants, and / or any other type of implant); and / or from the perspective of predetermined (e.g., semantic and / or organ) segmentation categories.

[0043] Metadata may include administrative data that identifies patients and / or ensures that only medical datasets belonging to the same patient are compared. Alternatively or additionally, metadata may be used to chronologically rank medical datasets and / or determine the target format (and / or any standardized format) for joint analysis (and / or comparison of first and second medical datasets). For example, metadata may include an indication of whether the medical imaging data included in a medical dataset is 3D (also known as volumetric) or 2D (also known as planar). Alternatively or additionally, metadata may include an indication of the resolution of the medical imaging data.

[0044] In alternative embodiments, metadata may include data for purposes such as selecting a second medical dataset (and / or a second patient) based on a medical problem for joint analysis with a first medical dataset (and / or patient), such as data for comparison with previous (and / or second) patients treated for the same rare disease and / or rare cancer in the case of rare diseases and / or rare cancers.

[0045] Joint analysis of medical data from the first and second medical datasets may include comparing the medical data in a selected dimension (particularly 2D, if one of the two medical imaging datasets is 2D and the other is 3D) and / or a selected resolution (e.g., a coarser granularity between the two medical imaging datasets).

[0046] The received or converted first medical dataset may include the results of a perception task. Optionally, the perception task may include organ detection and / or semantic segmentation.

[0047] The perception task can be performed by a perception task model (and / or the result of the perception task can be obtained from the perception task model). The perception task model may include a detection model and / or a segmentation model. The perception task can be performed on first (and / or second) medical image data.

[0048] The segmentation model can be configured to perform semantic segmentation, specifically based on a predetermined set of segmentation categories. Alternatively or additionally, the detection model can be configured to perform organ detection and / or object detection (e.g., for detecting implants), specifically based on a predetermined set of object categories.

[0049] The transformation of at least a portion of the first medical dataset may depend on the received contextual information and / or medical questions.

[0050] Contextual information, medical questions, and / or any textual data can be analyzed using LLM.

[0051] The received second medical dataset may include text data. Optionally, the text data may include medical reports of medical examinations, patient medical history, patient medications, laboratory results data; and / or annotations related to medical image data acquired using the second medical imaging modality.

[0052] Alternatively or additionally, in addition to the first medical image data, the first medical dataset may include annotations of the first medical image data (e.g., segmentation results, detection results, and / or the results of measurements and / or estimates of lesions or organs, such as length or diameter).

[0053] The received second medical dataset may include second medical imaging data acquired with the aid of a second medical imaging modality, particularly second medical imaging data related to a medical problem.

[0054] The second medical image data may be acquired prior to the first medical image data. Alternatively or additionally, the second medical image data may have a lower dimension (e.g., 2D) and / or lower resolution than the first medical image data (e.g., it may be 3D).

[0055] The first medical imaging modality (and an optional second medical imaging modality) may include MRT, CT, single-photon emission CT (SPECT), PET, scintillation scanning, radiography (also known as X-ray imaging, XR), ultrasound (US), elastography, photoacoustic imaging, functional near-infrared spectroscopy (FNIR), and / or magnetic particle imaging (MPI). In some embodiments, the first medical imaging modality (and an optional second medical imaging modality) may include a combination of two medical imaging modalities, such as PET-CT.

[0056] (For example, first and / or second) medical imaging data may include 3D data (also known as volumetric data and / or volumetric images; acquired, for example, by means of MRT and / or CT) or 2D data (also known as planar data and / or planar images; acquired, for example, by means of radiographic imaging (also known as X-ray imaging)).

[0057] The conversion of the first medical imaging data into the target format can be determined on a case-by-case basis, particularly depending on the second medical dataset including second medical imaging data for joint analysis, and / or depending on the medical question. In some embodiments, volumetric medical imaging data may be particularly useful for answering different medical questions, and the target format can be selected based on one of these medical questions.

[0058] The second medical imaging modality may differ from the first medical imaging modality. The second medical imaging data may be acquired before the first medical imaging data.

[0059] By acquiring medical imaging data using different medical imaging modalities, the latest technological advancements and / or the availability of specific scanners can be leveraged when acquiring the more recent of two medical datasets (especially the first medical imaging modality). Alternatively or additionally, different medical imaging modalities may be optimal for detecting different types of abnormalities. Combining medical imaging data from two different medical imaging modalities can improve the detection rate and / or reliability of abnormalities. Alternatively or additionally, different medical imaging modalities can be used to obtain information about different types of detected abnormalities (e.g., supplementary information that can help differentiate between different diagnoses).

[0060] The second medical imaging data may be lower in dimensionality than the first medical imaging data. In particular, the second medical imaging data may include 2D medical images, and the first medical imaging data may include 3D medical images. Alternatively or additionally, converting at least a portion of the first medical dataset into the target format may include constructing 2D medical images from the first medical imaging data, particularly including views of anatomical structures similar to those included in the 2D medical images in the second medical dataset.

[0061] Converting at least a portion of the first medical dataset into the target format may include determining the dimensions of the second medical image data and / or determining the views included in the second medical image data (e.g., the viewing direction and / or location of at least one point on a 2D image, and / or a slice in a 3D medical image corresponding to the 2D image).

[0062] The similarity between a 2D medical image constructed from first medical imaging data and a 2D medical image included in second medical imaging data can provide similar anatomical structures for rendering and / or similar orientations for performing, for example, measurements of lesions. Therefore, the development of medical conditions (and / or lesions) can be monitored with improved accuracy and / or without the time-consuming manual selection of a suitable view of the first medical imaging data.

[0063] This method can be performed at least in part by a target-aware two-branch distillation (TDD) network. The TDD network may include a digital reconstructor. This digital reconstructor can be configured to transform first medical image data and to obtain a view similar to that provided by second medical image data. The digital reconstructor may further include a proposal extractor. This proposal extractor can be configured to identify at least one region of interest (ROI) in the transformed first medical image data and in the second medical image data. The digital reconstructor may further include a source adaptation (SA) block and a class target (TL) block. The SA block can be configured to perform medical image data analysis on the first medical image data, particularly upon receipt and / or before transformation to the target format. The TL block can be configured to perform medical image data analysis on the transformed first medical image data, particularly on at least a portion of the first medical image data transformed to the target format. Alternatively or additionally, the TL block can be configured to perform medical image data analysis on at least a portion of the second medical image data included in the second medical dataset. The digital reconstructor may also further include a target proposal perceptron. This target proposal perceptron can be configured to compare the medical image data analyses from the SA block and from the TL block.

[0064] The proposal extractor may include a Regional Proposal Network (RPN) for identifying ROIs.

[0065] SA blocks and / or TL blocks may include one or more fully connected (FC) layers and / or projection heads (HEADs).

[0066] The target proposal perceptron can perform iterative cross-attention between layers of the SA block and the TL block. The target proposal perceptron can be specifically used to jointly train SA and TL blocks, for example, on at least partially annotated (especially first) medical image data, and / or at least partially based on bi-branch self-distillation. Thus, domain knowledge from the source domain (and / or data in the source format) can be used for the target domain (and / or for data in the target format), and vice versa.

[0067] Specifically, supervised training (specifically, real data knowledge) can be used to train TDD for SA blocks and TL blocks respectively, and / or supervised training (specifically, real data knowledge) can be used to train self-distillation based specifically on the results of the target proposal perceptron.

[0068] Alternatively or additionally, the SL branch and TL branch can be trained in a unified teacher-student learning process. For example, the SL branch can treat the first medical image data (specifically in the source format) as input and the annotations as real data, and / or the SL branch can act as the teacher. The annotations can be transferred to the transformed first medical image data (specifically in the target format) to train the TL branch, and / or the TL branch can act as the student.

[0069] Two-branch self-distillation may include training with (e.g., a second) unannotated medical image data and / or using pseudo-annotations generated in (or by) the SL branch (e.g., as the teacher). These pseudo-annotations can then be used to train the TL branch (e.g., as the student).

[0070] If it is determined that the second medical dataset cannot answer the medical questions associated with the first medical dataset, the step of jointly analyzing at least partially transformed first medical datasets and received second medical datasets can be stopped. Alternatively or additionally, if it is determined that the second medical dataset cannot answer the medical questions associated with the first medical dataset, the step of extracting medical image analysis datasets can be stopped.

[0071] If it is determined that only contextual information about organs and / or medical conditions different from those included in the first medical dataset can be extracted, the second medical dataset may, for example, be unsuitable for answering the medical question. For instance, if the second medical dataset includes second medical imaging data of a patient's knee, but the medical question and the first medical imaging data pertain to the head or chest, the method may be discontinued.

[0072] The cessation of combined medical image analysis can be reminded (and / or notified) to medical experts (e.g., radiologists). Reminders can improve the efficiency of clinical workflows.

[0073] Medical experts can be alerted by using a user interface (UI), especially a graphical user interface (GUI).

[0074] The method may further include the step of providing the extracted medical image analysis data to a medical expert, specifically through a UI (preferably a GUI). Optionally, the method may further include the step of receiving feedback from the medical expert. Feedback may include acceptance, rejection, and / or rating or evaluation (e.g., regarding quality) of the medical image analysis data. Alternatively or additionally, acceptance and / or rating or evaluation may be implicit, for example, if the medical expert at least partially modifies the extracted medical image analysis data.

[0075] Based on feedback, medical image analysis data may (or may not) be used to draft examination reports related to the primary medical image data. For example, the extracted medical image analysis data may be provided in the examination report template, especially unless the medical expert denies the extracted medical image analysis data.

[0076] The acceptance, rejection, and / or evaluation or assessment of the extracted medical image analysis data can be used (e.g., continuously) to fine-tune LLM, analysis algorithms, base models, perception task models, and / or TDD networks to perform the method.

[0077] This method can be executed by computing devices, distributed computing systems, and / or in a computing cloud. Alternatively or additionally, the method can leverage task-specific AI-related "plug-and-play" configurations. For example, a first medical dataset may include a (particularly volumetric) chest CT dataset as first medical imaging data. The chest CT dataset can be automatically analyzed (e.g., measurements can be performed) by a dedicated AI tool (e.g., as an AI-driven radiology-assisted chest CT). As a further example that can be combined with the previous example, a second medical dataset may include a (particularly planar) chest X-ray dataset as second medical imaging data. The chest X-ray dataset can be automatically analyzed by another dedicated AI tool (e.g., for detecting radiographically detected chest X-rays).

[0078] On another front, a unified platform is provided for extracting medical image analysis data relevant to medical problems. This unified platform is configured to store multiple (particularly a second) medical datasets. The unified platform is further configured to be accessed to perform methods according to the methodological aspects. For example, the unified platform may include data storage space and at least one computing device, a distributed computing system, and / or a computing cloud.

[0079] The present invention can include a modality-agnostic (and / or unified) platform for storing clinical information, eliminating the need to compare actual data at hand, such as with another target and / or another point in time, once the platform is complete. The present invention can be particularly used to extract clinical information where the relevant information arrives at two points in time from different modalities (also denoted as source and target). For example, a patient may have gone to hospital 1 and received a chest X-ray, and six months later may have gone to a different hospital (e.g., hospital 2) and received a (specifically, chest) CT scan. Radiologists are perfectly capable of comparing both X-rays and CT scans, and have been doing so. However, comparing X-rays and CT scans is not so easy for machines. According to the present invention, an AI system can fill out a machine-readable form (and / or fields of a unified platform) at hospital 1, and then a system at hospital 2 will fill out a similar form (and / or fields of a unified platform). Finally, when it comes to comparing two examinations from different locations and different modalities (specifically, as a human radiologist would do), the AI ​​is comparing the two forms. In particular, there is no need to worry about one scan being a chest X-ray and the other a chest CT scan.

[0080] Specifically, the unified platform may include data storage space, specifically a database, capable of storing data (and / or information) from various sources such as medical scanners (e.g., medical imaging and non-medical imaging), measurement results (e.g., including electrocardiograms, ECGs, electroencephalograms, EEGs, and / or laboratory results), and / or clinical reports (also known as medical reports). Fields in the data storage space (specifically, fields in the database) may be in text format (e.g., for medical reports and / or laboratory results), graphic format (e.g., for ECGs and / or EEGs), 2D image format (e.g., for X-ray images), and / or 3D image format (CT scans and / or MRT scans). The database may be or may include a system collection of electronically stored data. It can contain any type of data, including text, numbers, images, videos, and files. Software (called a database management system (DBMS)) may be provided for and used by the database to store, retrieve, and / or edit the data.

[0081] Alternatively or additionally, the unified platform may include an AI orchestrator for orchestrating and / or managing data flow between external AIs (such as one or more LLMs, and / or one or more imaging AIs, particularly at least one imaging AI for each medical imaging modality) and a database. The AI ​​orchestrator may be embodied by at least one (e.g., depending on the device) computing device, a distributed computing system, and / or a computing cloud.

[0082] AI orchestrators can be trained using background knowledge from the medical field, for example, through self-supervised learning (SSL), transfer learning, and / or reinforcement learning. Alternatively or additionally, AI orchestrators can be configured to understand causality, such as comorbidities, to detect conflicting data, and / or to resolve conflicts in the data (e.g., by evaluating the confidence level of the source, specifically for each medical imaging data and / or medical non-imaging data).

[0083] Regarding the device, a computing apparatus is provided for extracting medical image analysis data related to a medical problem. The computing apparatus includes a first interface configured to receive a first medical dataset related to a patient. The first medical dataset includes first medical image data acquired using a first medical imaging modality related to the medical problem. The computing apparatus further includes a second interface configured to receive a second medical dataset related to the patient. The second medical dataset includes contextual information related to the medical problem. The computing apparatus further includes a conversion module configured to convert at least a portion of the received first medical dataset into a target format. The computing apparatus further includes an analysis module configured to jointly analyze the at least partially converted first medical dataset and the received second medical dataset in light of the medical problem. The computing apparatus further includes an extraction module configured to extract medical image analysis data related to the medical problem from the results of the joint analysis.

[0084] The computing device may include a providing module and / or a third interface configured to provide extracted medical image analysis data to medical experts, particularly through the use of a GUI. The computing device may further include a fourth interface configured to receive feedback from the medical expert. This feedback may include acceptance, rejection, and / or rating (also referred to as: evaluation) of the medical image analysis data.

[0085] The computing device can be configured to perform any of the steps disclosed in the context of the method, and / or may include any of the features disclosed in the context of the method.

[0086] In another aspect, a computer program product is provided. This computer program product includes program elements that, when loaded into the memory of a computing device, cause the computing device (e.g., according to the device aspect) to perform steps of a method for extracting medical image analysis data related to a medical problem, according to the method aspect.

[0087] In a further aspect, a computer-readable medium is provided on which program elements are stored that can be read and executed by (e.g., according to the device aspect) a computing device, so that when the program elements are executed by the computing device, the steps of a method for extracting medical image analysis data related to a medical problem are performed according to the method aspect.

[0088] The features, characteristics, and advantages of the present invention, as well as the ways in which they are implemented, will become clearer and more readily understood from the following description and embodiments, which will be described in more detail in the context of the accompanying drawings.

[0089] The following description does not limit the invention to the included embodiments. In different figures, the same components or parts may be labeled with the same reference numerals. Generally, the figures are not drawn to scale.

[0090] It should be understood that the preferred embodiments of the present invention may also be any combination of the dependent claims or the above embodiments with the corresponding independent claims.

[0091] These and other aspects of the invention will become apparent and will be explained with reference to the embodiments described below. Attached Figure Description

[0092] Figure 1 This is a flowchart of an exemplary method for extracting medical image analysis data related to medical problems; Figure 2 This is an overview of the structure and architecture of an exemplary computing device for extracting medical image analysis data relevant to medical problems, which can be configured to perform... Figure 1 Methods; Figure 3 ,Right now Figure 3A and 3B An example of a unified platform populated with medical imaging and non-medical imaging data at different time points is shown, which can be used as a basis for execution. Figure 1 The method is used in the medical dataset or is included in the medical dataset; Figure 4 This demonstrates the execution of actions based on contextual information included in prior medical imaging data. Figure 1 Examples of methods; and Figure 5 The combination is shown Figure 3A Unified platform Figure 1 A further example of how the method works.

[0093] Any reference numerals in the claims should not be construed as limiting the scope. Detailed Implementation

[0094] Figure 1 An exemplary flowchart, specifically computer-implemented, is schematically illustrated for a method of extracting medical image analysis data relevant to a medical problem. This method is generally referenced by reference numeral 100.

[0095] Method 100 includes a step S102 of receiving a first medical dataset related to a patient. The first medical dataset includes first medical image data acquired using a first medical imaging modality related to a medical problem. Method 100 further includes a step S104 of receiving a second medical dataset related to a patient. The second medical dataset includes contextual information related to the medical problem. Method 100 further includes a step S106 of converting at least a portion of the first medical dataset received in S102 into a target format. Converting at least a portion of the first medical dataset in S106 into the target format may include constructing a 2D medical image from the first medical image data, particularly including anatomical views similar to those included in the 2D medical images in the second medical dataset.

[0096] Method 100 further includes step S108 of jointly analyzing at least a partially transformed first medical dataset S106 and receiving a second medical dataset S104 in view of a medical problem.

[0097] Method 100 further includes step S110 of extracting medical image analysis data related to the medical problem from the results of the joint analysis S108.

[0098] Method 100 may further include the step of providing the extracted medical image analysis data of S110 to a medical expert in S112, particularly by using a user interface (UI), such as a graphical user interface (GUI).

[0099] Optionally, method 100 includes a step S114 of receiving feedback from medical experts. Feedback may include acceptance, rejection, and / or rating or evaluation of the medical image analysis data (e.g., in view of quality, and / or by medical practitioners, particularly radiologists).

[0100] Figure 2 An exemplary architecture of a computing device for extracting medical image analysis data relevant to a medical problem is schematically illustrated. The computing device is generally referred to by reference numeral 200.

[0101] The computing device 200 includes a first interface 202 configured to receive a first medical dataset related to a patient. The first medical dataset includes first medical image data acquired using a first medical imaging modality related to a medical problem. The computing device 200 further includes a second interface 204 configured to receive a second medical dataset related to the patient. The second medical dataset includes contextual information related to the medical problem. The computing device 200 further includes a conversion module 206 configured to convert at least a portion of the received first medical dataset into a target format. The computing device 200 further includes an analysis module 208 configured to jointly analyze, in light of the medical problem, the at least partially converted first medical dataset and the received second medical dataset. The computing device 200 also further includes an extraction module 210 configured to extract medical image analysis data related to the medical problem from the results of the joint analysis.

[0102] The computing device 200 may include a providing module (not shown) and / or a third interface 212 configured to provide extracted medical image analysis data to medical experts. The extracted medical image analysis data may be provided, particularly on a UI, preferably on a GUI.

[0103] The computing device 200 may further include a fourth interface 214 configured to receive feedback from a medical expert. The feedback may include acceptance, rejection, and / or rating (and / or evaluation) of the medical image analysis data.

[0104] The computing device 200 may include an input-output interface 216. Any one of a first interface 202, a second interface 204, an optional third interface 212, and / or an optional fourth interface 214 may be included in the input-output interface 216.

[0105] The computing device 200 may include a processor 218. The conversion module 206, the analysis module 208, the extraction module 210, and / or the optional provisioning module may be embodied by the processor 218.

[0106] The computing device 200 may further include a memory 220, for example for storing computer program code for performing method 100, and / or for storing medical image analysis data and / or any intermediate results.

[0107] The computing device 200 can be configured to execute method 100.

[0108] The techniques of this invention (e.g., including method 100 and / or computing device 200) can be viewed as a supplementary arrangement and / or extension of a conventional common data element (CDE) such as RadElement (https: / / www.radelement.org / ) for radiology. For example, a unified platform can be represented as RadElement+, on which method 100 is performed (and / or by accessing the unified platform). Alternatively or additionally, the techniques may include methods for constructing a unified platform and / or a modal agnostic framework to enable AI to process information for prior comparisons.

[0109] RadElement is an example of a platform that provides a directory of radiology CDEs indexed by title and controlled terminology such as SNOMED CT, LOINC, and RadLex. CDEs can be grouped into "sets" that list the CDEs used in a particular application. CDEs can be reused and therefore can belong to more than one set. The website's web service (using REST and JSON) makes it easy for automated systems to discover and apply radiology CDEs.

[0110] It is important to note that the actual operation of RadElement and the techniques of this invention (e.g., including method 100 and / or computing device 200) do not closely overlap. The RadElement platform is designed to standardize radiology reports using Common Data Elements (CDEs), which can provide a unified platform for communication among medical professionals (e.g., radiologists and / or referring physicians). Furthermore, depending on the role, information within the RadElement platform comes from different medical professionals to complete individual medical scenarios.

[0111] The technology of this invention (e.g., including method 100 and / or computing device 200), even though referred to by the inventors as RadElement+, is not merely an extension of the existing RadElement platform designed for communication between different medical professionals, but possesses additional features. The technology of this invention provides a unified platform for AI, rather than for medical professionals, based on a similarly inspired and / or complementary approach. According to the technology of this invention, different (particularly “plug-and-play” styles) AIs designed and fine-tuned for different tasks can communicate and / or interact, for example, to confirm or refute individual medical analyses.

[0112] The techniques of this invention (e.g., including method 100 and / or computing device 200) facilitate the generation of a unified framework (e.g., complementary to, similar to, and / or an extension of RadElement) for information extracted by an AI system from different multimodal sources (such as CT, XR, MRT, US, and / or clinical reports, also referred to as: medical reports). This allows the framework to be agnostic to the AI ​​used to perform different tasks (e.g., performing segmentation on medical image data included in, for example, a first and / or a second medical dataset), and / or to the modality of how prior information (such as contextual information and / or the second medical dataset) is delivered.

[0113] Individual AI systems demonstrate state-of-the-art performance on highly specialized tasks (e.g., segmentation and / or detection from a specific modality, and / or text understanding via a large language model LLM), but traditionally struggle to perform more than one task, more general tasks, and / or tasks once the scope of the task is expanded. The techniques of this invention (e.g., including method 100 and / or computing device 200) allow for plug-and-play scenarios where state-of-the-art AI suitable for the task can be replaced once a better solution becomes available.

[0114] Figure 3A and 3B An example of the technique of the present invention (also referred to as: framework) is illustrated. This framework is agnostic regarding how and which technique is used to populate the relevant uniform information at each point in time. For example, if at different points in time (also referred to as: time points, TP), a chest X-ray is performed during visit 1 (or TP1) to monitor for nodules, and a CT scan is performed during visit 2 (or TP2), separate X-ray and CT AI systems can be used to handle those cases (e.g., performing segmentation on medical imaging data included in the second and first medical datasets, respectively), and extracting relevant information that can subsequently be used to perform joint analysis (and / or comparison). Similarly, if only clinical reports are available for previous examinations (e.g., the patient was transferred from another institution, and / or included as contextual information in the second medical dataset), a fine-tuned LLM can be used to extract information from the clinical report that is similar (or identical) to the information extracted by the imaging AI from the current examination (and / or the first medical dataset).

[0115] exist Figure 3AIn this example, LLM 304, fine-tuned LLM 306, and imaging AI 302 (e.g., for performing segmentation and / or detection, such as organ detection) are exemplarily shown for populating the unified platform 300. The unified platform 300 includes multiple fields that can be arranged in different categories, such as (e.g., standardized) clinical data 310, (e.g., standardized) health data 312, (e.g., standardized) derived elements 316, (e.g., standardized) health economics 318, (e.g., standardized) metadata 314, and / or (e.g., standardized) vocabulary 320.

[0116] exist Figure 3B The diagram illustrates an example of populating a unified platform 300 at three different time points (specifically denoted as Visit 1, Visit 2, and Visit 3). The corresponding categories populated at each time point X (and / or Visit X; where X = 1, 2, 3) are exemplarily represented as (e.g., standardized) health data 312-X, (e.g., standardized) derived elements 316-X, and (e.g., standardized) vocabulary 320-X. (For example, standardized) metadata 314 may include, for example, CDM_source (which has a CDM for a common data model) and / or any other (for example, conventional) metadata.

[0117] (For example, standardized) vocabularies 320 (e.g., the vocabulary used in visits 1, 2 and 3, denoted as 320-1, 320-2 and 320-3 respectively) may include, for example, concepts, (e.g., conventional) vocabularies, domains, concept classes, concept relationships, relationships, concept synonyms, concept ancestors, source-to-concept maps, and / or drug strengths.

[0118] (For example, standardized) health economics 318 may include, for example, costs and / or payer-plan-period.

[0119] (For example,) the standardized) derived element 316 (e.g., the derived elements used in visits 1, 2 and 3, respectively, with reference numerals 316-1, 316-2 and 316-3) may include, for example, a Condition_era, a Drug_era, a Dose_era and / or a result schema filled, for example, by the fields: Cohort and / or Cohort_definition.

[0120] (For example,) standardized health data 312 (e.g., health data used in visits 1, 2 and 3, referred to as 312-1, 312-2 and 312-3 respectively) may include, for example, the field Location (optionally having a subfield: Location_history), the field: Care_site and / or Provider.

[0121] (For example,) standardized clinical data 310 may include, for example, the field Person, which has subfields: Observation_period and / or visit_occurrence and / or additional subfields such as Visit_detail, Condition_occurrence, Drug_exposure, Procedure_occurrence, Device_exposure, Measurement, Note, Note_NLP (which has NLP for natural language processing), Survey_conduct, Observation, Specimen and / or Fact_relationship.

[0122] For each medical examination (also known as a visit), entries can be generated, for example, in category (e.g., standardized) vocabulary, (e.g., standardized) derived elements, and / or (e.g., standardized) health data.

[0123] Standardization in this article may refer to digital data formats (e.g., text formats, graphic formats, 2D image formats, and / or 3D image formats). Alternatively or additionally, standardization may refer to a type of content, such as medical expressions and / or medical relationships based on medical ontology, such as systematic medical nomenclature (SNOMED), specifically SNOMED CT.

[0124] The techniques of the present invention (e.g., including method 100 and / or computing device 200) can be used for robust merging of prior image data (e.g., included in a second medical dataset).

[0125] Traditional methods suffer from two main problems: (1) performance degradation due to large shifts in data distribution, even if prior examinations are performed using the same imaging modality, and (2) a lack of instance-level annotations in the target domain, such as if something (e.g., anomalies, lesions, and / or tumors) was previously present and / or not present, or if it has grown and / or shrunk in size, since prior and current examinations (e.g., supplied as second and first medical datasets, respectively) are used for comparative purposes in almost all cases. Traditional methods primarily address either of these two challenges, even though they are tightly coupled in cross-domain object detection. To address this problem, the present invention can utilize a target-aware dual-branch distillation (TDD) network (also known as: TDD framework). The TDD framework takes (multiple) current and prior studies as input, specifically as a first medical dataset and one or more second medical datasets, respectively. The current and prior studies may come from, may not come from (and / or need not come from) the same modality, but the present invention (e.g., including method 100 and / or device 200; also simply: system or framework) can assume that the scope of detection in all modalities is the same. In one embodiment, if a clinical question can only be answered in one imaging modality, the system can reject the case. Comparisons in this scenario would be meaningless.

[0126] In different modalities, the system can route higher-dimensional images (current, e.g., included in a first medical dataset, or previous) to a digital reconstructor to produce a lower resolution image. In one example embodiment, the framework then integrates the detection branches of both the source and target domains into a unified teacher-student learning scheme. The techniques of this invention can effectively reduce domain shifts and generate reliable supervision. In particular, distinct target proposal perceptrons can be introduced between the two domains. By fully utilizing the target proposal context from iterative cross-attention, it can adaptively enhance source detection to perceive objects in the target image. Subsequently, a concise two-branch self-distillation strategy is designed for model training, which progressively integrates complementary object knowledge from different domains via self-distillation in both branches, such as according to self-distillation described in [1, 2].

[0127] Figure 4A TDD network 400 is schematically illustrated. The TDD network 400 includes a digital reconstructor 402. The digital reconstructor is configured to receive and select a (particularly 2D) view 414 from (particularly 3D, such as including CT scans) first medical image data 412 that is similar to (particularly 2D, such as including X-ray images) a view provided by second medical image data 418 (particularly 2D, such as including X-ray images). In other words, if the source 412 and the target 418 come from different modalities (such as CT and radiography), the digital reconstructor 402 is used to reconstruct the lower-dimensional modality. Many techniques (such as those described in [3]) can be used for this.

[0128] Subsequently, the source domain image 414 is transferred to the domain 416 of the class target, and / or style transfer 420 is performed on the view 414 of the first medical image data 412. The style transfer 420 is based on the style of the second medical image data 418, and as a result, the transformed first medical image data 416 (also represented as the first medical image data 416 of the class target) is obtained.

[0129] After the selection of the corresponding view 414 and / or the dimensionality reduction of the first medical image data 412 are performed, the portion of the TDD network 400 starting from style transfer 420 can operate similarly to the TDD for transferring the weather and / or lighting-related domain of 2D camera images of traffic scenes described in [1].

[0130] The TDD network 400 further includes (particularly, a shared) proposal extractor 404, to which all images 414, 416, 418 from three (or two, if target and class targets are counted as one) domains are fed to obtain proposals and proposal features. The shared attribute may involve shared weights of a set of Region Proposal Networks (RPNs), or the same RPN may be used for all three input images 414, 416, 418, as indicated by the dashed lines within the proposal extractor 404. The proposal extractor 404 is configured to identify at least one region of interest (ROI) and / or obtain proposal features in a view 414 of the first medical image data (also referred to as source 414), the transformed first medical image data 416, and the second medical image data 416.

[0131] The TDD network 400 further includes a source adaptation (SA) block (also referred to as: SA branch) 406 and a target class (TL) block (also referred to as: TL branch) 408. The SA block 406 is configured to perform medical image data analysis on a (particularly, source) view 414 of first medical image data 412 and on second medical image data 418, and the TL block 408 is configured to perform medical image data analysis on the first medical image data after transformation (also referred to as: target class) 416 and also on the second medical image data 418. The TDD network 400 further includes a target proposal perceptron 410. The target proposal perceptron 410 is configured to compare the medical image data analyses from the SA block 406 and from the TL block 408, particularly when performed on the same second medical image data (also referred to as: target) 418.

[0132] SA block 406 and TL block 408 each include one or more fully connected (FC) layers and one or more projection heads (HEADs). The same HEAD or HEADs with shared weights are used to input two types of medical image data (referred to as images) into the corresponding blocks, namely: source image 414 and target image 418 input into SA block 406 and target image 416 and target image 418 input into TL block 404.

[0133] The proposed features from source image 414 and class target image 416 are used to train corresponding branches 406 and 408, respectively, under supervision with ground truth data, as indicated by reference numeral 422. Furthermore, the proposed features from the ground target domain image 418 are fed into both branches 406 and 408 to learn object knowledge from both the source and class target domains. Because image 418 from the target domain is unannotated, the model is optimized through self-distillation, also as indicated by reference numeral 422.

[0134] Therefore, the TDD network 400 is trained by dual-branch supervision 422, taking advantage of the fact that the target image 418 is processed by both the SA branch 406 and the TL branch 408, and the first medical image data 412 is processed by the same view, but is processed once in the source style 414 and once in the target-like style 416 by the SA branch 406 and the TL branch 408 respectively, and annotations (e.g., segmentation of the first medical image data 412) are used to train both the SA branch 406 and the TL branch 408.

[0135] Two-branch self-distillation can include three stages: joint domain pre-training, cross-domain distillation (e.g., using a teacher-student approach), and dual-teacher refinement.

[0136] Figure 5A further example of a unified platform 300 including a database of extracted information is shown, the database having (e.g., standardized) fields for medical imaging findings and medical non-imaging findings (and / or clinical findings) as indicated by reference numerals 310, 312, 314, 316, 318, 320. Figure 5 The unified platform 300 further includes an AI orchestrator 502. The AI ​​orchestrator 502 acts as different AIs (e.g., Figure 3 The AI ​​orchestrator 502 is configured, for example, to determine what to “listen” (and / or submit to the database) (e.g., which medical imaging data and medical non-imaging data) and what to ignore (and / or not submit to the database).

[0137] exist Figure 5 The example illustrates three distinct time points (e.g., medical visits): TP1, TP2, and TP3, which provide different types of potential inputs to the database (specifically, entry categories 310, 312, 314, 316, 318, and 320). For example, multiple different input data are shown at TP1, and one type of input data is shown at TP2 and TP3. In this exemplary embodiment, each potential input is provided by artificial intelligence in a narrow sense (ANI).

[0138] exist Figure 5 In the example, at TP1, different ANIs provide three different findings from chest radiography (CRX). At reference numeral 504-1A, ANI 1a provides information on lung opacities from CRX 1; at reference numeral 504-1B, ANI 1b provides information on lung nodules from CRX 1; at reference numeral 504-1C, integration using ANI 1 and CRX 2 is performed; and at reference numeral 504-1D, ANI 1 using CRX 3 provides information on pleural effusion.

[0139] exist Figure 5 At TP2 in the attached figure, marked 504-2A, a CT report is provided to LLM ANI.

[0140] exist Figure 5 At TP3, in reference numeral 504-3A, ANI 2 is used to analyze CT, and at reference numeral 504-3B, ANI 2 is provided, exemplarily, with outputs and / or intelligent reports of what can already be provided (e.g., by analyzing previous examinations, such as those at TP1 and / or TP2).

[0141] Any individual ANI used at reference numerals 504-1A, 504-1B, 504-1C, 504-1D, 504-2A, and 504-3A can be considered “plug-and-play” and / or can be independently replaced by updated and / or new ANIs (particularly updated and / or new ANIs associated with AI orchestrator 402). Alternatively or additionally, the bidirectional arrows between reference numerals 504-1A, 504-1B, 504-1C, 504-1D, 504-2A, 504-3A and AI orchestrator 502 can indicate bidirectional transmission of information (at least the possibility of bidirectional transmission). For example, AI agents (and / or ANIs) 504-1A, 504-1B, 504-1C, 504-1D, 504-2A, and 504-3A can submit information to AI orchestrator 502 for evaluation and subsequent storage. Alternatively or additionally, AI agents (and / or ANIs) 504-1A, 504-1B, 504-1C, 504-1D, 504-2A, and 504-3A may request information from AI orchestrator 502 to perform tasks, such as providing (and / or obtaining) the results of previous checks.

[0142] exist Figure 5 In this process, any CRX or CT data at TP1, TP2, and TP3 can respectively serve as the first medical dataset according to method step S102, wherein, according to method step S104, any other received findings related to these medical image data are used as candidates for contextual information. AI orchestrator 502 is configured to execute method steps S106 and S108. Figure 5 At reference numeral S110 in the attached figure, medical image analysis data related to the medical problem is provided as output, and a report is sent to the user (e.g., a radiologist or oncologist). The report may include results from ANI 2 (specifically with respect to CT performed at TP3) at reference numeral 504-3A and / or comparisons with previous results such as data 504-1A, 504-1B, 504-1C, 504-1D based at TP1 and data 504-2A based at TP2.

[0143] AI orchestrator 502 can be a basic generalization of a visual-language model of general artificial intelligence [5] and / or [6] for medical image analysis. The system (and / or unified platform 300) of the present invention differs from the prior art, particularly in the modality-agnostic nature of how information is received and stored. AI orchestrator 502 enables this by coordinating multiple (e.g., imaging) modalities and different AI agents (and / or ANIs).

[0144] Figure 5The AI ​​orchestrator 502 is trained with background knowledge in the medical field that enables it to understand causality—coexisting diseases—and to perform abstract thinking. The AI ​​orchestrator 502 can be trained through self-supervised learning (SSL), transfer learning, and / or reinforcement learning. Alternatively or additionally, the AI ​​orchestrator 502 can undergo temporal changes (e.g., using continuous training and / or fine-tuning, and / or responding to new “plug-and-play” ANIs that provide input data), and / or can be configured to register temporal changes in patient-related data. Alternatively or additionally, the AI ​​orchestrator 502 can be configured to handle conflicting information, for example, if two nodule AIs provide different findings about the same organ, such as one AI indicating “normal” while the other indicates “nodule.”

[0145] The unified platform 300, AI orchestrator 502 and / or database (specifically entry categories 310, 312, 314, 316, 318, 320) can be updated (e.g., periodically and / or event-based, such as if a new imaging AI or a new LLM provides different types of input data (e.g., different data formats and / or information types not previously included in the database).

[0146] The present invention (e.g., including method 100 and / or computing device 200) provides a unified platform for storing and comparing information currently extracted from medical images or medical images accompanied by non-imaging data. The present invention, including the unified platform, is agnostic about how the information was initially collected. The platform is also universally adaptable and encompasses or has the ability to store all necessary information. The source of this information can vary depending on the imaging modality source (e.g., CT, XR, MR, US, and / or clinical reports) or the AI ​​system used.

[0147] A simple real-world example involves storing a patient's height or weight. Once that information is stored in a table, the tool used to measure the height becomes irrelevant. This is not typically the case for imaging-based AI systems: information extracted using an imaging modality and / or an AI system cannot traditionally be easily translated to different modalities and / or systems, so rendering previously extracted information is often useless.

[0148] Because AI systems are complex and often benefit from highly specialized tasks (e.g., segmentation and / or detection from specific modalities, and / or text understanding via LLM), creating a general AI for comprehensive diagnostics becomes complex. The present invention (e.g., including method 100 and / or computing device 200) provides a unified platform for different AIs to submit information they are good at and / or designed for. The present invention allows other AI systems to consume information from this unified platform. Therefore, the present invention provides a unified (also known as: agnostic universal) platform, specifically extending traditional platforms such as RadElement-like frameworks, and / or providing a unified, commonly used image space.

[0149] The universal applicability of this invention ensures that any AI network capable of extracting useful information from medical images can be used.

[0150] This invention improves accuracy. Traditional solutions focus only on the intensity harmonization of the source and target domains without quantifying the underlying primary tasks, such as detection (e.g., health-related information given a specific medical question).

[0151] This invention improves reliability. Lack of performance in the source domain leads to no or incorrect detections, rendering the entire prior comparison system unusable. For example, an anomaly detected as "nodule" in one domain and "pneumonia" in another can make comparisons with previous examinations difficult. This invention can draw the attention of medical professionals (e.g., radiologists) to this problem.

[0152] The present invention improves upon the universality of the technology. Traditional methods assume that current and previous examinations are performed in the same domain and / or modality (e.g., chest radiography, CXR, CT). This is a major, traditional bottleneck in clinical workflows, where different modalities are used to perform examinations depending on needs, institutional preferences, etc. The proposed framework enables cross-modal operation.

[0153] Where not explicitly described, the various embodiments or aspects and features thereof described in conjunction with the accompanying drawings may be combined or interchanged with each other without limiting or expanding the scope of the described invention, provided that such combination or interchange is meaningful and in the sense of the invention. Advantages described with respect to specific embodiments of the invention or with respect to specific figures are also advantages of other embodiments of the invention wherever applicable.

[0154] Referenced existing technology [1] He, M., Wang, Y., Wu, J., Wang, Y., Li, H., Li, B.,...&Qiao, Y. (2022). Cross-domain target detection by target-aware bibranch distillation. IEEE / CVF Computer Vision and Pattern Recognition Conference Collection of Argumentative Essays (Pages 9570-9580) [2] Liang, S., Wang, W., Chen, R., Liu, A., Wu, B., Chang, EC, ... & Tao, D. (2024). Target Detectors in Open Environments: Challenges, Solutions and Prospects arXiv preprint arXiv :2403.16271. [3] Pyrros, A., Chen, A., Rodríguez-Fernández, JM, Borstelmann, SM, Cole, PA, Horowitz, J., ...&Koyejo, S. (2023). Feasibility study of deep learning-based digital reconstruction tomography of the chest in the assessment of solitary pulmonary nodules. Radioactivity , 30(4), 739-748. [4] Hong-You Chen and Zhengfeng Lai and Haotian Zhang and Xinze Wang and Marcin Eichner and Keen You and Meng Cao and Bowen Zhang and Yinfei Yang and Zhe Gan. Contrastive Localization Language-Image Pretraining. arXiv:2410.02746 [cs.CV] [5] Li, X., Zhao, L., Zhang, L., Wu, Z., Liu, Z., Jiang, H., ... & Shen, D. (2024). Artificial general intelligence for medical image analysis. IEEE Biomedical Engineering Review . [6] Zhang, J., Huang, J., Jin, S., & Lu, S. (2024). Visual-language models for visual tasks: a review. IEEE Transactions on Pattern Analysis and Machine Intelligence .

Claims

1. A computer-implemented method (100) for extracting medical image analysis data related to a medical problem, comprising the following method steps: - Receive (S102) the first medical dataset related to the patient, wherein, The first medical dataset includes first medical image data acquired using a first medical imaging modality related to a medical problem; - Receive (S104) a second medical dataset related to the patient, wherein the second medical dataset includes contextual information related to the medical problem; - Convert at least a portion of the received (S102) first medical dataset into the target format (S106); - To jointly analyze (S108) the at least partially transformed (S106) first medical dataset and the received (S104) second medical dataset, based on medical problems; and - Extract (S110) medical image analysis data related to the medical problem from the results of the joint analysis (S108).

2. The method (100) according to claim 1, wherein, The received (S102) first medical dataset or the transformed (S106) first medical dataset includes the results of the perception task, optionally, The perception task includes organ detection and / or semantic segmentation.

3. The method (100) according to any one of the preceding claims, wherein, The transformation of at least a portion of the first medical dataset (S106) depends on the context information received (S104) and / or the medical question.

4. The method (100) according to any one of the preceding claims, wherein, The received (S104) second medical dataset includes text data, optionally, wherein the text data includes: • Medical reports from medical examinations; • The patient's medical history; • The patient's medication; • Laboratory results data; and / or • Annotations related to medical image data acquired using a second medical imaging modality.

5. The method (100) according to any one of the preceding claims, wherein, The received (S104) second medical dataset includes second medical image data acquired by means of a second medical imaging modality related to the said medical problem.

6. The method (100) according to any one of the preceding claims, wherein, The first medical imaging modality and the optional second medical imaging modality include at least one of the following: - Magnetic resonance imaging (MRT); - Computed tomography (CT); - Single-photon emission CT, SPECT - Positron emission tomography (PET); - Scintillation scanning technique; - Radiographic imaging; - Ultrasound, US; - Elastography; - Photoacoustic imaging; - Functional near-infrared spectroscopy, FNIR; and - Magnetic particle imaging, MPI.

7. The method (100) according to claims 5 and 6, wherein, The second medical imaging modality is different from the first medical imaging modality, and wherein the second medical image data is acquired before the first medical image data.

8. The method (100) according to any one of claims 5 to 7, wherein, The second medical image data is lower dimensional than the first medical image data, specifically, Wherein, the second medical image data includes two-dimensional (2D) medical images, and wherein, the first medical image data includes three-dimensional (3D) medical images; and / or The conversion (S106) of at least a portion of the first medical dataset into the target format includes constructing 2D medical images from the first medical image data, particularly including anatomical views similar to those included in the 2D medical images in the second medical dataset.

9. The method (100) according to any one of claims 5 to 8, wherein, The method (100) is performed by a target-aware dual-branch distillation TDD network (400), wherein the TDD network (400) comprises: - A digital reconstructor (402) configured to transform (S106) the first medical image data and obtain a similar view to the view provided by the second medical image data; - A proposal extractor (404) is configured to identify at least one region of interest (ROI) in the transformed first and second medical image data; - A source-adaptive SA block (406) and a class-target TL block (408), wherein the SA block (406) is configured to perform medical image data analysis on the received (S102) first medical image data, and wherein the TL block (408) is configured to perform medical image data analysis on the transformed (S106) first medical image data, and optionally on the second medical image data included in the second medical dataset; and - Target proposal sensor (410), which is configured to compare medical image data analysis from the SA block (406) and the TL block (408).

10. The method (100) according to any one of the preceding claims, wherein, If it is determined that the second medical dataset cannot answer the medical questions associated with the first medical dataset, then the steps of joint analysis (S108) of the at least partially transformed first medical dataset and the received second medical dataset and / or extraction (S110) of the medical image analysis dataset are stopped.

11. The method (100) according to any one of the preceding claims further includes the step of providing (S112) the extracted (S110) medical image analysis data to a medical expert, particularly by using a graphical user interface (GUI); optionally, The method (100) further includes receiving (S114) feedback from the medical expert, wherein, The feedback includes at least one of the following: (The feedback includes the medical image analysis data) • accept; • Negation; and • Rating or assessment.

12. A unified platform (300) for extracting medical image analysis data related to medical problems, wherein, The unified platform is configured to store multiple, particularly second medical datasets, and wherein the unified platform is further configured to be accessed to perform the method (100) according to any one of the preceding method claims.

13. A computing device (200) for extracting medical image analysis data related to a medical problem, comprising: - A first interface (202) is configured to receive a first medical dataset related to a patient, wherein the first medical dataset includes first medical image data acquired by means of a first medical imaging modality related to a medical problem; - A second interface (204) is configured to receive a second medical dataset related to the patient, wherein the second medical dataset includes contextual information related to the medical problem; - A conversion module (206) configured to convert at least a portion of the received first medical dataset into a target format; - Analysis module (208), configured to jointly analyze, in view of the medical problem, at least partially transformed first medical dataset and received second medical dataset; and - Extraction module (210), which is configured to extract medical image analysis data related to the medical problem from the results of the joint analysis.

14. The computing device (200) according to the preceding claim is further configured to perform any one of the steps of any one of the method claims 2 to 11, and / or include any one of the features of any one of the method claims 2 to 11.

15. A computer program product including program elements that, when loaded into the memory of a computing device (200), cause the computing device (200) to perform the steps of a method for extracting medical image analysis data related to a medical problem according to any one of the preceding method claims.

Images