Error Reduction Reading by Machine Learning-Assisted Alternative Search Proposal

Abstract

A preprocessor (PP) component and related method for a machine learning system (MLS) for processing medical data. The preprocessor includes an input interface (IN) for receiving initial findings generated by a human for a patient, and a medical image to which the findings relate. An encoder (ENC) of the preprocessor encodes the findings and the medical image into encoded data including encoded image data and encoded finding data. A combiner (COM) component of the preprocessor combines the encoded findings and the encoded image data into combined encoded data. An output interface (OUT) provides the combined encoded data to the machine learning system. The proposed preprocessor (PP) enables achieving more robust machine learning performance.

Description

Technical Field

[0001] The present invention relates to a preprocessor component for a machine learning model for processing medical data, a related method, a machine learning device including the preprocessor component and the machine learning model, a training system for training the machine learning model, a method for training the machine learning model, a computer program element, and a computer-readable medium.

Background Art

[0002] The interpretation of radiological studies (referred to herein as "reading" or "review") is a difficult task.

[0003] It is estimated that at least 5% of patients experience some form of diagnostic error and contribute up to 17% of the harmful errors in hospitals.

[0004] Approximately 75% of these errors are centered around radiology. Most research dedicated to preventing radiological errors has focused on avoiding false negatives, with particular attention paid to the fatigue of radiologists and the impact on the perception and interpretation of images, as well as their work ergonomics.

[0005] However, since it is assumed that radiologists have sufficient information to provide a correct interpretation when not due to fatigue, the accuracy of radiologists' interpretations has rarely been verified so far.

[0006] However, there are literature reports indicating that even without fatigue, radiologists' interpretations are not always correct. Certain radiological findings can be easily confused with each other. Examples reported in the literature include confusion between hematomas and misinterpreted lymphomas, or between various causes of pulmonary sclerosis such as water leakage, pus exudation, blood hemorrhage, or cell tumors / chronic inflammation.

[0007] The causes of misinterpretation range from inappropriate radiological experience, confusion in patient history, or inaccurate imaging. The consequences or image reading errors can include the need for re-imaging, unnecessary surgery, or even patient death.

SUMMARY OF THE INVENTION

PROBLEM TO BE SOLVED BY THE INVENTION

[0008] Therefore, it may be necessary to facilitate reducing the error rate in image reading.

MEANS FOR SOLVING THE PROBLEM

[0009] The object of the present invention is achieved by the subject matter of the independent claims, and further embodiments are incorporated in the dependent claims.

[0010] It should be noted that the aspects described below of the present invention are equally applicable to related methods, preprocessor components and machine learning configurations including machine learning models, training systems for training machine learning models, methods for training machine learning models, computer program elements, and computer-readable media.

[0011] According to a first aspect of the present invention, a preprocessor component for a machine learning system for processing medical data is provided.

[0012] At least one input interface for receiving an initial finding generated by a human about a patient and a medical image related to the finding, and an encoder for encoding the finding and the medical image into encoded data, including encoded image data and encoded finding data.

[0013] A combiner component for combining the encoded findings and the encoded image data into combined encoded data, and an output interface for providing the combined encoded data to a machine learning system.

[0014] In an embodiment, an input interface receives context data, provides context information related to a report and / or an image, an encoder encodes at least a part of the context data into encoded data, and a combiner combines the encoded context data with an image and an encoded report to obtain combined data.

[0015] In an embodiment, the context data includes any one or more of i) a patient history, ii) an imaging request for an image, and iii) statistical data related to misdiagnosis.

[0016] In an embodiment, the combiner and / or the encoder is implemented as a machine learning model.

[0017] In an embodiment, a machine model for the encoder includes a processing channel configured for iterative processing.

[0018] In an embodiment, the processing channel is configured to process at least the encoded patient history.

[0019] In an embodiment, the expected dimensional size of the encoded patient history is variable.

[0020] In an embodiment, the encoded data includes at least one of a matrix or at least one vector. This enables the implementation of efficient calculations.

[0021] In some embodiments, at least one vector includes one heat vector, but other coding techniques may be used instead.

[0022] In another aspect, a machine learning device is provided that includes a preprocessor component according to any one of the above embodiments and a machine learning system.

[0023] In an embodiment, it includes a machine learning model configured to convert an encoded output combined with a machine learning system into output data indicating at least one second finding, where the second finding is either an alternative to the initial finding or equal to the initial finding.

[0024] In some embodiments, the output includes a natural text string or a medical finding code.

[0025] In an embodiment, the apparatus includes a localizer configured to map the output data to an image position within an image.

[0026] In another aspect, a training system configured to train one or both of the machine learning models of any one of the above embodiments based on training data is provided.

[0027] In another aspect, a method for preprocessing medical data for machine learning, receiving an initial finding of a patient generated by a human and a medical image related to the finding; encoding the finding and the medical image into encoded data including encoded image data and encoded finding data; combining the encoded finding and the encoded image data into combined encoded data; providing the combined encoded data to the machine learning system is provided.

[0028] In another aspect, a method for processing provided composite encoded data by a further machine learning model is provided. Specifically, this method can include converting a combined encoded output into output data indicating at least one second finding, where the second finding is either an alternative to the initial finding or equal to the initial finding.

[0029] In another aspect, a method for training any one of the machine learning models in the above-described embodiments or all of the aspects based on training data is provided.

[0030] In another aspect, a computer program element adapted to cause at least one processing unit to execute a preprocessing method or a training method when executed by the at least one processing unit is provided.

[0031] In another aspect, at least one computer-readable medium storing the program element or storing the machine learning model is provided.

[0032] Medical findings are decisions regarding medical conditions taken based on partially available information. For example, such a decision could be formulated as "Does this 47-year-old patient have a heart attack?" Contextual data makes it possible to add potentially relevant information (such as "the patient is male" and / or "has a history of heavy smoking", etc.). By using such contextual data, the decision-making process becomes more robust, but may sacrifice speed or computational resources. However, there is information that does not improve (is irrelevant to) the robustness of the system while still taking processing time. The preprocessor proposed herein preprocesses information from different sources in order to balance this information according to its relevance in order to improve its relevance to the desired output (findings). The adverse impact on computation time is preferably mitigated by a parallelizable algorithm that can be executed on special hardware such as a GPU or others.

[0033] The encoded combined findings generated by the preprocessor are preferably elements of a vector space. The encoded combined findings preferably include encoded contributions from one or more (such as all) of the initially received data types, such as the input image, the initial findings, and optionally, context data. This represents balancing the input data such that it can be more robustly processed by a transducer to selectively rebalance the relevance of varying data types for the findings to be computed.

[0034] The preprocessor is preferably used in real time in a proposed machine learning (“ML”) module that works with a radiologist, for example, to propose alternative interpretations / findings when filling out a report, thus reducing reading errors.

[0035] The ML-based recommender module proposed herein, in an embodiment, analyzes the radiologist's interpretation of findings and proposes different possible readings that result from the radiologist's attenuation. Using this option for double-checking can reduce reading errors. By helping to reduce reading errors, costs can be reduced. The current cost of misdiagnosis is an astounding $17 to $29 billion per year, and the healthcare sector can spend elsewhere with far more benefits. Such reduction in reading errors is expected to lead to overall better patient care by reducing the number of re-imaging procedures or by reducing the number of unnecessary interventions caused by misdiagnosis.

[0036] The proposed ML module can process radiologist reports in freehand form or in any type of unstructured form. Structured reports such as tables, checkmarkable lists, etc. are not required herein.

[0037] The proposed system and method can be applied to any type of radiation modality such as, for example, chest X-ray, CT, MRI, PET or ultrasound examination.

[0038] The use of a preprocessor in such a module is preferred herein, but such use does not exclude other uses including stand-alone use, and the data of the preprocessor can be used by itself, such as in medical data analysis, to explore the interrelationships between data from different sources.

[0039] "User" relates to a person who operates an imaging device or supervises an imaging procedure, such as a medical professional or other person, and who conducts an image review / reading session, such as a radiologist. In other words, the user is generally not a patient.

[0040] Generally, the term "machine learning" includes computerized devices (or modules) that implement machine learning ("ML") algorithms. Some such ML algorithms operate to adjust a machine learning model configured to perform a task (i.e., "learn"). Other MLs act directly on the training data, without necessarily using a model or the like. The adjustment or update of this training data corpus is called "training". Generally, the task performance by the ML module can be measurably improved using the training experience. The training experience can include appropriate training data and the exposure of the model to such training data. The performance of the task can be improved such that the data better represents the task to be learned. The training experience helps improve performance when the training data well represents the distribution of examples on which the final system performance is measured. The performance may be objectively test-measured based on the output generated by the module in response to supplying test data to the module. The performance can be defined in terms of a specific error rate to be achieved for a given test data. See, for example, T. M. Mitchell, "Machine Learning", page 2, section 1.1, page 6, section 1.2.1, McGraw―Hill, 1997.

[0041] Here, exemplary embodiments of the present invention will be described with reference to the following drawings, which are not to scale unless otherwise specified.

Brief Description of the Drawings

[0042]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Mode for Carrying Out the Invention

[0043] Next, refer to FIG. 1, which shows a block diagram of a medical device MAR for processing medical data, in particular measurements related to a patient. Generally, this device includes a medical measurement setup such as a medical imaging device IA that generates measurement values related to the patient.

[0044] Measurements taken on the patient may include medical images. Based on the medical images, a computer-implemented medical recommender module MA, preferably machine learning, operates to calculate one or more alternative findings that are alternatives to the findings provided by a human medical user. The operation of the medical recommender module MA is based on the initial findings provided by the user and the measurement values in the image I on which the user's initial findings were particularly based.

[0045] Therefore, the medical recommender MA is preferably configured to operate with a radiologist to suggest alternative findings that replace those reached by a human radiologist when examining the same image for the same patient, based on machine learning herein. When inspecting measurement values such as images, the user activity is called "reading" in order to reach his or her findings. In that case, generally, the recommender module MA helps to reduce the risk of errors in the reading of medical measurement values, especially in the reading of images. In the following, with the understanding that the principles described herein apply equally to other types of medical measurements such as laboratory data (e.g., blood samples), ECG, EEG, or any other such medical data that describes the (medical) state of a patient, reference is made specifically to medical images as an example of such medical measurements.

[0046] Accordingly, the machine learning recommender MA analyzes the image and, in addition, analyzes the initial findings of a medical user (such as a radiologist) to derive / infer possible alternative findings (if any). If there are no alternative findings, i.e., if the findings derived by the recommender module MA are the same as or sufficiently similar to the findings provided by the radiologist, this fact can be appropriately flagged by a confirmation signal or otherwise. If the findings generated by the machine learning recommender MA differ from the user's initial findings, they are alternative findings, and this fact can also be presented graphically, numerically, or in any other form on the display device DID or by using any other appropriate transducer. Instead of or in addition to visualization, the alternative findings calculated by the recommender can be stored or otherwise processed. In the following, reference is made to "findings" in the singular, with the understanding that multiple findings such as those generated by the recommender MA or the user can be involved. Accordingly, the reference to "findings" in this specification should not necessarily mean a single finding (although this is not excluded in this specification) but should be interpreted as a reference to "at least one finding". Generally, as understood in this specification, "to find" is a standard in medical terms, in coded form, or in natural language, or graphically, etc., and describes one or more aspects of a patient's medical condition. Accordingly, findings can include signs of a disease, condition, illness, etc. related to the patient, or the absence thereof ("the patient is healthy").

[0047] Before describing the operation of the configuration MAR, particularly the operation of the pusher module MA, in more detail, first, some components of the imaging device IA will be described. Generally, the imaging device IA may include a signal source SS and a detector device DD. The signal source SS generates a signal, for example, an interrogation signal, which interacts with the patient to generate a response signal, and this response signal is then measured by the detector device DD and converted into measurement data such as the medical image. An example of an imaging device or imaging device is an X-ray based imaging device such as an X-ray imaging device configured to generate a protection image. Volume tomography (cross-sectional) images are not excluded in this specification, such as via a C-arm imaging device or a CT (computer tomography) scanner, or others.

[0048] During the imaging session, the patient PAT can reside on the patient support PS (patient bed, bed, etc.), but this is not necessary as the patient may stand, squat, sit, or take other body postures within the examination area during the imaging session. The examination area is formed by a part of the space between the signal source SS and the detector device DD.

[0049] For example, in a CT set, during an imaging session, the X-ray source SS rotates around the examination area together with the patient in order to acquire projection images from different directions. The projection images are detected by the detector device DD, in this case an X-ray sensitive detector. The detector device DD can rotate opposite to the examination area using the X-ray source SS, but such co-rotation is not necessarily required as in the case of a fourth-generation or higher-generation CT scanner. A signal source SS, such as an X-ray source (X-ray tube), is activated so that the X-ray beam XB exits from the focal point within the tube during rotation. The beam XB traverses the examination area and the patient tissue therein and interacts with them to generate modified radiation. The modified radiation is detected as intensity by the detector device DD. The detector DD device is coupled to a collection circuit such as a DAQ to capture the projection image digitally as a digital image. The same principle applies to (planar) X-ray imaging, except that there is no rotation of the source SS during imaging. In such a radiographic set, it is this projection image that can be examined by a radiologist. In a tomographic / rotational setting, first, the multi-directional projection images are processed by a reconstruction algorithm that converts the projection images from the projection area into cross-sectional images within the image area. The image area is located in the examination area. The projection images or the reconstructed images are not distinguished anymore in this specification, but are simply collectively referred to as the (input) image I or the input image I. It is the input image I that can be processed by the recommender MA.

[0050] However, the input image I is not necessarily obtained from X-ray imaging. In contrast to the aforementioned transmission imaging modalities, other imaging modalities such as emission imaging, for example SPECT or PET, are also envisioned herein. Further, in some embodiments, magnetic resonance imaging (MRI) is also envisioned herein.

[0051] In an embodiment of MRI, a signal source SS functions as a detector device DD configured to receive a high-frequency response signal emitted by a patient present in a magnetic field in a reception mode, and is formed by a high-frequency coil. Such a response signal is generated in response to a previous RE signal transmitted by the coil in a transmission mode. However, in some embodiments, instead of the same coil being used in the different modes, dedicated transmission and reception coils may be present.

[0052] In radiography, a source SS is within a patient in the form of a previously administered radioactive tracer that emits radiation that interacts with patient tissue. This interaction results in a gamma result detected by a detection device DD, in this case a gamma camera, which is preferably arranged in a ring around an examination area in which the patient resides during imaging.

[0053] Instead of, or in addition to, the modalities described above, ultrasound (US) is also envisaged, and the signal source and detector device DD are appropriate acoustic US transducers.

[0054] The image I generated by either modality IA may be passed to the (non-volatile) memory MEM via the communication interface CI and stored for subsequent review or other processing. However, the online setting is not excluded herein if the image is reviewed when it is generated by the imaging device. However, in many cases, the offline setting may be sufficient or more appropriate when the image is preferably first stored in the memory MEM in relation to each patient's ID. The image memory may be non-volatile, for example, a medical image database such as PACS. When the user desires to review (or "read") the stored images of the patient of interest are accessed by an appropriate database query system using, for example, the patient's ID or other markers. The accessed images may be passed to the viewer software VIZ. The viewer software can be operated to generate a visualization of the image as a graphics display device, and the graphics display device is then displayed on the display device DID. The above review can be performed on any suitable computing platform, mobile (laptop, desktop, tablet), or stationery (desktop, workstation, etc.) where the visualizer VIZ and the DB query system can be executed or, for example, remotely executed on a server while being controlled therefrom.

[0055] A radiologist can review images in one or more review sessions. The radiologist can enter his or her findings medically into a structured or free-text report file. The report file is a data structure that resides in the computer memory of a review system (not shown). The report file contains data (text strings, codes, etc.) indicating the findings. The report file is simply referred to as a "report" herein. The report may preferably be stored in relation to the image(s) on which the review was based and / or in relation to the respective ID of the patient(s) from whom the image(s) I was taken. Thereby, all relevant information can be conveniently retrieved later if needed, such as during a review. The report may be stored in the same database as the image(s), or in a different database such as a patient record database (HIS or others).

[0056] Human reviewers are often under stress and have to read large amounts of image material (e.g., in a busy clinic where new patients keep coming in continuously, some of whom may be trauma patients and require immediate attention). In such an environment with high stress and high workload, there is a possibility that the reviewer may unfortunately make a reading error, associating incorrect findings with the image, which can potentially have tragic consequences for the patient. False positives, or even worse, false negative types of errors, can occur in this way. The occurrence of such incorrect readings may be due to user fatigue and / or may be caused by poor-quality images. For example, important image information can be hidden, for example, by artifacts, or by forgotten tags, text information boxes, annotations, etc., or by other such widgets that are later added to the image, for example, by overlay.

[0057] Examples of such misinterpretations or misdiagnoses include lymphoma misreported as a hematoma, or different causes of pulmonary sclerosis: hydrothorax, empyema, hemorrhage, or confusion between cell - tumor / chronic inflammation. The causes of the misunderstandings are diverse. They range from insufficient radiological experience, confused patient histories, and inaccurate imaging, to name just a few examples.

[0058] To reduce the possibility of such misinterpretations, the recommendation module MA processes the radiologist's report (including initial findings) together with the input image I to calculate alternative findings. The initial findings and the image to which the findings relate form the core input data c.

[0059] The alternative findings calculated by module MA can draw the radiologist's attention as described above. The alternative findings can be sounded or displayed instead of, or alongside, the original / initial findings previously assigned by the radiologist. The radiologist can then choose to accept the alternative findings and thus modify the original findings, or the radiologist can choose to reject the alternative findings and maintain the original findings. Such acceptance and rejection operations can be performed via a suitable user interface (UI) device, such as via a keyboard, touch screen, pointer tool (stylus, computer mouse), etc. Instead of such alternative findings, the original findings may be confirmed as described above.

[0060] Preferably, the recommender MA operates while the radiologist is reviewing the visualized image. The visualized image is generated, for example, by a visualization device VIZ on a display device DID. Thus, the recommendation module MA can interact with the visualizer VIZ to provide alternative or confirmatory findings in a suitable visualization, preferably simultaneously with the image being reviewed and / or simultaneously with the report being displayed.

[0061] As will be described in more detail below, in some embodiments, alternative findings can be localized by a localizer LC within a review-based image. Localization can be implemented by displaying an overlay widget, such as a bounding box or other graphical indicator widget, to indicate the portion of the input image I to which the alternative finding pertains. The alternative finding can be displayed in relation to the bounding box as text, in code, in free text, or otherwise, preferably inside the bounding box, adjacent to it, or in any other spatially associable manner.

[0062] Next, the operation of the recommender module MA is shown in more detail in the block diagram of FIG. 2.

[0063] As can be seen, the recommender MA processes input data. The input data includes core input data that includes an image I reviewed by a human reviewer, and original / initial findings such as those included in a radiologist report RP. The report RP can include text or other information indicating the findings arrived at by the radiologist based on the input image I. The input image I and the report RP pertain to the same patient. Although this is assumed in some embodiments herein, it is not necessary to include such a report RP in the core input data. Instead, it may be sufficient to extract the original findings from the report RP by an NLP (Natural Language Processing) pipeline, string matching, or other text or string processing, and include data indicating the findings so extracted in the input core data. The NLP pipeline itself can be implemented by a dedicated machine learning model such as a BERT-type NN model, or other fully connected architecture configured for natural language processing.

[0064] Optionally, the context data is processed as context input data x together with the core input data c. The context input data x provides context regarding the patient and / or findings. This enables the machine-based recommended MA to operate more robustly. The context data x can function as a regularizer during training. The mode of training will be described in more detail below.

[0065] The context data x can include several different data items, either alone or in any combination. For example, the context data x can include the radiology examination request RQ that resulted in the generation of the input image. This request includes information that guided the radiologist or other medical user to request the imaging session in which the input image was generated. The imaging RQ data can include text data in a structured or unstructured format that represents the reason for requesting the imaging of the image I. Typically, the imaging request data RQ can include suspicious diagnostic clues based on the fact that an imaging session was requested for the image I to be generated by the modality IA.

[0066] Alternatively or additionally, the context data x can include patient reference data PH, such as the patient's medical history, e.g., patient records, prior images, etc., or extracts therefrom. This medical history data PH can be retrieved from a database where this information regarding the patient at hand can be stored.

[0067] In addition or alternatively, the context data x can include statistical data ST or a statistical analysis of previous misdiagnoses. This data type is related to previous cases where misdiagnosis / misreading occurred, i.e., where the original findings generated by the radiologist were later found to be actually incorrect, not necessarily regarding this patient but in a cohort of other patients.

[0068] This type of data ST may typically include the correct findings that would be found by a subsequent investigation, typically conducted when it becomes clear that the original findings were incorrect. Thus, the review data of misdiagnosed cases contains valuable information where the findings generated by the user can correlate with the respective correct findings. Thus, it can be derived from such data, for example, by statistical analysis, which type of (diagnostic) findings is more likely to be misdiagnosed than others. Preferably, the core input data c (including initial findings such as in image I and report RP) enhanced by context data x = (request RQ, statistics ST, patient history PH) is processed by a machine learning-based module MA to generate, for example, one or more alternative findings W1, W2. Here too, not necessarily all of the described context data x need be used. Any of the above sub-combinations are contemplated herein. However, in particular, it is included in the context data x of imaging requests RQ and / or statistical data ST that have been found to yield good results. An example could be a trauma patient, and the findings could include "fracture", "organ laceration", and "internal hemorrhage". The radiologist reports the trauma. However, in April 2020, statistical trends could map, for example, the impact of COVID 19, and fatigued individuals sent requests as chest X-rays of infectious patients. In this case, the system should discard or downweight the context and not provide a "pneumonia" recommendation. Five days later, when the patient begins to develop pneumonia on a control X-ray, the system should ignore or downweight the context that the patient is a trauma patient and should consider the findings of an infectious trauma patient.

[0069] The visualizer VIZ can show the findings W1, W2 in text form TX1, TX2. Two such findings are shown, but there may be more or less than two. The alternative findings may be superimposed on the input image in question, such as on the images I1, I2, as exemplarily shown on the right side of FIG. 2. The bounding boxes BB1 and BB2 may be used to indicate parts of the image that contributed more than other image parts in the course of the calculations performed by the machine learning module MA when generating the alternative findings W1, W2. The hot map technique may be used to derive in the bounding boxes, as will be described in more detail below. However, any other localization technique may be used, for example, if the input data includes explicit coordinates indicating parts of the image that support the findings, and such coordinates are learned together with the images. The localization may be achieved by a localizer LC, the operation of which may be described in more detail below.

[0070] The textual representations TX1 and TX2 of the alternative findings W1 and W2 may include text strings such as tree text form, natural language text, or may be in the form of medical terminology coding such as the World Health Organization's International Classification of Diseases (ICD) coding such as version 11 or earlier versions, or future versions. However, any other suitable medical coding may be used instead, and generally, coding as used herein differs from encryption, which is irreversible without knowledge of a cryptographic key, whereas initially, at a lower computational load, it is reversible without the need for such a key.

[0071] Referring now to the block diagram of FIG. 3, further details of the machine learning-based recommender module MA are shown. Conceptually, there is a potential relationship between the input data (i.e., the core data c potentially enhanced with the context data x) and the alternative findings associated therewith. This relationship can be considered to be implicitly represented in the existing training data including the historical images and the associated historical patient records. Such training data can be found in a medical database and associated with a pre-examination for a suitably representative cohort of patients.

[0072] The potential relationship is classically, analytically, and ad hoc difficult and even impossible to model. For this purpose, a machine learning model that does not require such explicit modeling is used. In the learning / training process, a standard model with parameters adapted based on the training data can be used. The parameters may be iteratively adjusted until the adapted model is considered to be a good enough approximation of the potential relationship (in a predefined manner based on a cost function). The mode of learning will be described in more detail below.

[0073] Continuing to refer to FIG. 3, and more particularly, this figure represents a model architecture that can be used to construct the recommender module MA. After appropriate training on the training data, the model can compute alternative findings (if any) given at least the core input data c as described above. Generally, the model of the recommender MA can include a preprocessor PP and, downstream thereof, a postprocessor, which is referred to herein as the recommender machine learning system MLS.

[0074] Both the preprocessing processor PP and the postprocessing recommender machine learning system MLS can be implemented by respective dedicated machine learning models, as will be described in more detail below.

[0075] Broadly speaking, the post-processor PP processes the input data v k (including the core data c ⊂ v k ) in any of training, testing, or deployment (i.e., real-world clinical post-training applications), and calculates the intermediate output e(v k ). The intermediate output is passed to the post-processor stage MLS, which then converts the intermediate output into a desired finding, e.g., an alternative finding w if any, or a confirmation finding. The pre-processor PP and / or the post-processor MLS can be implemented as their respective trained machine learning models. Hereinafter, it is assumed that the model has been trained. The training aspects will be described in more detail in FIGS. 4 and 6.

[0076] Looking at the pre-processor PP in more detail, the raw input data rk is applied to the encoder ENC part of the pre-processor PP. The raw data is in digital form and may be the result of being converted into such digital data by data capture, A / D conversion, and character recognition, etc. For example, a handwritten report can be captured as an image and then OCR processed. Alternatively, a report or other text data is generated by a word processing module or the like. The raw data is encoded by the encoder ENC into the input data vk for the second stage of the pre-processor PP. This second stage is configured as a combiner COM. The encoded input data v k for the combiner COM includes the core data c and, optionally, the context data x. The combiner COM combines the encoded input data vk into the combined encoded data e(Vk) + .

[0077] The combiner COM of the pre-processor PP can be configured as a set of computational nodes n ij of an artificial neural network ("NN"). Each node is associated with parameters ("weights") that have been pre-adapted in training based on training data. Node n ijmay be arranged in a cascade within the layer. In some embodiments, the combiner COM is configured as a convolutional neural network (“CNN”), and the computational nodes implement convolutional operators.

[0078] The layer may include an input layer IL, IN, one or more hidden layers HL, and an output layer OL, OT. The encoded input data is supplied to the input layer at v k where it is processed and propagated through one or more hidden layers HL to the output layer OL. The output of the output layer OL can include a feature map representing intermediate data representing the encoded data e(v k ) of the input v k . By way of example only, an architecture having three layers is shown in FIG. 3, although more than three may be present. As shown in FIG. 3, it has been observed that a fairly shallow network having only a single hidden layer functions well. Making the network shallow with a single or a small number (such as 2, 3 or 4) of hidden layers allows for higher responsiveness as less computation is required.

[0079] The inputs and outputs between the hidden layer HL and the outputs of the input layer IL and output layer OL may be referred to herein as feature maps. Feature maps can be represented as matrices (a “tensor”) of two or more dimensions for computational and memory allocation efficiency.

[0080] Preferably, some or all of the layers IL, HL are convolutional layers, i.e., they include one or more convolutional filters that process the input feature maps from previous layers into an intermediate output sometimes called logits. Any bias terms may be applied, for example, by addition. The activation operator of a given layer non-linearly transforms the logits into the next generation of feature maps, which are then output to the next layer, passed as input, and so on. The activation operator may be implemented as a rectified linear unit (“ReLU”), or as a softmax function, sigmoid function, tan function, or any other suitable non-linear function. Optionally, other functional layers such as pooling layers or dropout layers may be present to facilitate more robust learning. A pooling layer reduces the dimensionality of the output, while a dropout layer disconnects the connections between nodes from different layers. The combined or “hybrid” encoding e(vk) generated by the combiner COM + is, in embodiments, one or more such feature maps. Preferably, the encoding e(vk) + is equal to the number (vk) of data streams encoded as a supply to the input layer IL. However, such combined encodings e(vk) + may be present more or less.

[0081] Node n ij 's computing function is typically a (weighted) linear combination of the logits from the previous layer, and also includes applying non-linearity to the logits so combined. However, other functional combinations that are not necessarily linear combinations are also envisioned.

[0082] The postprocessor MLS may include a transformer TRF stage. The transformer processes the combined encoding e(vk) +It operates to convert into the output finding w either the first finding or any alternative of the confirmation. Similar to the preprocessor PP, the transformer TRF can be configured as a trained ML model. For example, a general setup of the transformer TRF can be similar to the setup of the combiner COM of the preprocessor PP, but the layers of the transformer TRF are preferably not convolutional. Instead, especially when classification results are required, such as in the classification of the input feature map from the combiner COM to the output finding, they are two or more fully connected layers. When the input is regressed to the finding, again, fully connected layers may be used, but in some embodiments, a specially configured convolutional kernel with spatially adapted convolution and padding may be used instead as needed. However, a fully connected architecture is preferred herein.

[0083] More generally, the transformer stage TRF can include an attention mechanism for learning the spatial dependencies that tend to be lost in the convolutional setup. The attention mechanism can be implemented by matrix multiplication or normalization, or others. The attenuation mechanism, for example, enables the reweighting of portions (plural) of sequential inputs to model the language context. An autoencoder (AE) or variational AE (VAE) can be used. The attenuation mechanism can be implemented as a submodel such as a fully connected layer inserted between the encoder and decoder of the AE or VAE. The fully connected submodel as the attention mechanism can receive inputs from the encoder and from the output of the decoder. Such an attention mechanism can provide its output as an input to the decoder. Variants of the attention mechanism can be implemented as dot products, query-key values, or others. Such mechanisms recombine the inputs on the encoder side and redistribute these effects to each target output.

[0084] Preferably the combiner COM is convolutional or hybrid, both fully connected and convolutional, and the transformer TRF is fully connected.

[0085] The transformer TRF, which is a component of the downstream machine learning system MLS, converts the encoded information e(v k ) and generates an output recommendation w. The output recommendation w may be, for example, an alternative to the first recommendation in the report PR. In some cases, the output is essentially equal to the first finding, thus reassuring the user. If there is an alternative finding, the user can have a second, more detailed look in a second review session and then choose, as described above, to accept or reject the alternative finding using an appropriate UI device such as a GUI.

[0086] The transformer TRF can operate as a regression network or as a classifier network. One such example of the transformer TRF includes a fully connected NN, in contrast to the CNN that can be used in a preprocessor PP architecture. A BERT-type network can be used, for example, as described in A Smit et al "CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT", arXiv:2004.09167, which is available online at 2021-10-12 https: / / arXiv.org / abs / 2004.09167. The transformer TRF is preferably a language model such as BERT or others. Its input (encoded e(v k ) + ) is preferably converted to a text output. The transformer TRF can embed such an input into, for example, a feature vector. Thus, in some embodiments, the transformer TRF can be described as a text-embedding language model.

[0087] The output of the transformer TRF is not necessarily a single finding, although this may be the case in an embodiment. A plurality of such findings may be output in an embodiment. The format may be a vector format such as some of the inputs (vk), where each index is coded such that it represents a finding, and each entry represents a score for that finding "i". The entry need not necessarily be a probability, as two or more findings may be applicable. The transformer TRF combines the hybrid encoding e(vk) + with a softmax layer, or any other normalizer, to be able to represent each entry of the output vector as such a score. The final output provided to the user can include only the index of the finding having the top score within the output vector. Alternatively, a list of m findings for the first m scores (m > 1) is provided. The vector index may match an NL string describing a finding in NL, such as "PNEUMONIA", or a code, numerical value, or other means may be provided. The string w can be displayed, stored, and transmitted in a text message, email, sound out, etc.

[0088] The localizer LC can be implemented as a separate component or as a component included in the transformer TRF or the pre-preprocessor PP. The localizer LC is operable to identify one or more portions of the input image I. The identified portions include pixels that contribute to the arriving output w more than other neighboring pixels or overall across the image plane. The contribution is determined by thresholding or by considering the gradient behavior graf * of the entire model, including the pre-preprocessor PP and the transformer TRF, as a function w = g(I, θ I g with respect to the parameters θ of the model *It can be measured by observing together. The localizer can be implemented by heatmap techniques such as GradCAM, GradCAM++, or other class activation mapping techniques. A suitable technique envisioned herein is available online at https: / / arxiv.org / abs / 1610.02391 and is described in "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization" by Ramprasaath R. et al., published at arxiv: 1610.02391 [cs.CV]. Reference is also made to "Generalized Gradient-based Visual Explanations for Deep Convolutional Networks" by Chattopadhyay A. et al., available online at https: / / arxiv.org / abs / 1710.11063v2 and published at arxiv: 1710.11063 [cs.CV]. Also, the concept of saliency maps, envisioned herein and available online at https: / / arxiv.org / abs / 1312.6034 and described in "Deep Inside Convolutional Networks: Visualizing Image Classification Models and Saliency maps" by Simonyan K. et al., published at arXiv:1312.6034 [cs.CV], is also referenced.

[0089] The identified portion can be graphically rendered by the above-described bounding box, for example, in interaction with the output w provided by the recommender module MA, so as to be caused by the visualizer VIZ.

[0090] The localization lc(w) to find w can instead be achieved by training the model on the output (w, p), where p is a set of image coordinates that define the part within the input image to which the output finding w relates. Thus, in this embodiment, the localization information p is native. Thus, the coordinates p are provided within the training data and considered by the objective function during training. Thus, in this embodiment, the localizer LC is an implicit function in the trained model PP,COM.

[0091] An alternative finding w may relate to a completely different organ or tissue type in the patient, compared to the initial / original finding. However, this is not necessarily required. The alternative finding may still relate to the same organ as the radiologist's initial finding, but instead represent a different diagnosis or medical insight regarding the same organ.

[0092] The preprocessor PP and the downstream recommended machine learning system MLS (including the Transformer TRF) can be implemented on the same processing unit, such as a server or other computing device. However, this is not necessarily assumed in all embodiments of this specification. A distributed implementation is also envisioned. For example, the preprocessor PP may be implemented on one computing unit PU, while the downstream machine learning system MLS including the transformer TRF is implemented on another processing unit PU' (not shown). Preferably, the processing units PU, PU' are communicatively coupled to each other such that the combined encoded e(vk)+ generated by the preprocessor PP can be provided for combination by the downstream machine learning system MLS. For example, the processing units PU, PU' can be geographically remote from each other. The preprocessor PP can be implemented on one (or more) servers, while the system MLS can be implemented on a user's terminal device, such as on a handheld device (laptop, smartphone, etc.) or on a fixed device such as a desktop or workstation. Alternatively, it is the MLS that is implemented on one or more servers, and it is the preprocessor PP that is executed on such a user's terminal device.

[0093] Preferably, to achieve good throughput, the computing device PU includes one or more processors (CPUs) that support parallel computing, such as those with a multi-core design. In one embodiment, one or more GPUs (graphics processing units) are used.

[0094] Next, referring to the preprocessor PP of FIG. 3 in more detail, in the encoder part ENC, the raw input data item rk is processed in a separate strand or channel (shown as a set of parallel lines on the left side of FIG. 3) into each of the respective "pure" encodings (vk) that include the encoding of the core data. Specifically, the raw input data rk is encoded into the respective encoded input data vk in each pure form in separate processing channels. Then, such pure encodings can be combined by the combiner COM into the intermediate result e(v k )+ sometimes called the hybrid encoding. The intermediate encoding result e(v k ) is a set encoding, but each is no longer pure and instead includes the mutual contributions combined from pure encodings from other channels / strands. This is shown in FIG. 3 by the multiple line feedings to the node n 3j of the output OL, where each line carries a contribution that originated at least in part from some or all of the encoded input data vk. Thus, the combiner COM mixes or equalizes the contributions from other / different channels vk.

[0095] The number of such combined encoding results e(v k )+ can vary. In the case of a dimension reducer type combiner COM, it can be made to be a number less than the number of encoded items v k . The encoder ENC can be implemented as a mapper that maps the raw input to the elements v k of a vector space such as a respective vector or matrix or tensor in some cases.

[0096] The combiner COM then mixes or equalizes the contributions from the pure encodings v k into the intermediate result which is a vector space element e(v k ) + having mutual contributions from various channels. Such an intermediate result can include a feature map when the combiner COM is configured as a CNN. In some embodiments, some or each of the intermediate results e(v k)+ may include contributions from some or all of the other data items v from other channels k’!= k k’ from.

[0097] For example, if the length / dimension of the raw data within each channel is known, it can be implemented as a LUT, so the encoder part ENC does not necessarily require a machine learning element. However, the combiner COM is preferably implemented as an ML model such as a neural network NN, preferably a CNN, a fully connected network, or a hybrid network, including the computational node n ij as described above.

[0098] Some data types such as the patient history PH may have variable lengths or dimensions, which may not be known a priori. For such types of data, preferably, an ML-based sequential data processor SDP can be provided within the strand / channel for processing the patient history data PH. This can be implemented as a recurrent neural network, probably of the convolutional type or the long short-term memory (LSTM) type. In general, RNNs describe all forms of NNs that return their outputs to their inputs. This includes not only networks with a "loopback" function (fully connected or convolutional), but also more complex architectures such as LSTM or gated recurrent units (GRUs) that send two types of information: the output h of the nodes in a layer called the cell state C t and. t In addition, the second-stage combiner COM of the preprocessor PP may be configured as a feedforward network.

[0099] The raw data (r k) may include text data (such as report PR, requirement RQ, patient history PH, etc.), numerical data such as statistical data ST, or spatial pixel / voxel data such as input image I. In some channels, such encoding is not required. Such channels can be implemented as unity operators that pass their input directly to the combiner COM. For example, the input image data I may not need to be encoded in the first stage ENC because the image data is already essentially properly encoded as a matrix of pixel values. However, in embodiments, the channel for the image channel may still include trained ML components such as an image classifier that classifies the image into vectors such as status, disease, etc. Thus, the input image I can be encoded into a vector. A CNN-type network with a classifier output layer (a normalizer such as a softmax layer) can be used to constitute such an ML component and is, for example, properly pre-trained on some appropriately labeled training images. This classifier and / or the sequence data processor SDP can be co-trained with the combiner model COM and / or the transformer model TRF. Generally, the ML components of the encoder stage ENC can be pre-trained or trained using the models of the combiner COM and the transformer TRF.

[0100] The encoding in the encoder stage ENC and / or the combined encoding of the combiner COM may, in some cases, be provided in tensor format, matrix format, or vector format. In the following embodiments, the encoding (v k ) of different types of raw data (r k ) in the first-stage encoder ENC is described in vector format, but this is exemplary and not limiting in this specification.

[0101] For example, a natural language processing (NLP) method can be applied to retrieve different findings from a radiologist's freehand report. A BERT-type network can be used, which is a trained neural network of the type that converts a freehand report into radiological findings, for example, converting it into a list of "fracture, consolidation, dilated mediastinum, no finding, pleura and others, cardiomegaly, pneumothorax, atelectasis, atelectasis, support device, edema, pleural effusion, lung lesion, lung opacity". These findings can be encoded by the encoder ENC into, for example, a one-hot vector such as TIFF2025520268000002.tif1791.

[0102] The position of 1 indicates the presence of a finding, which in this case is "no finding". In addition to the utilization of the radiologist's report RP, this type of model can be used to analyze the requirements RQ for radiological research. By transfer learning, the same network such as CheXbert or other BERT-type networks can be adapted to process the requirements RQ for radiological research into classes including "airway research, trauma, oncology, device verification" and others, and encoded into a one-hot vector such as TIFF2025520268000003.tif1869.

[0103] Finally, the same or a similar NLP method can be transferred to process the patient history PH into respective vectors. However, in this case, since the number of diseases in each patient history is different and the time from the remission state is likely to be a beneficial factor, it is more appropriate not to use one-hot vectors. The vectors for patient history PH encoding can instead be encoded, for example, like TIFF2025520268000004.tif16100 for duplicate values, one-coding for disease types, and a second encoding of the elapsed time from the remission state.

[0104] In this example, the disease code "22" has been in remission for 168 months, the disease code 5 has been in remission for 17 months, while the disease code 64 has not yet been in remission, as indicated by, for example, a negative number. Such an implementation can be based on a look-up table (LUT) of diseases. Such a LUT can make it possible to establish a correspondence between the codes and the states, for example, "22" can code for "cancer", 5 can code for "fracture", and 64 can code for "pneumonia".

[0105] Instead of passing the image through the discriminator operator as described above, the input image I processing channel can include its own NN, such as a CNN, configured to process the image I into classes. Again, a BERT-type setting such as CheXbert can be used, but image data can be used as input. The output provided to the combiner may be configured again as a one-hot vector, where each entry represents the absence or presence of TIFF2025520268000005.tif1486.

[0106] As described above, the classes into which the image I is classified can represent medical conditions / diseases, etc.

[0107] Thus, the first-stage encoder ENC can use its own ML network for each channel to provide different vectors encoding the diseases / conditions, and the combiner COM, when the vectors / matrices are called, integrates these "votes" into a balanced vote by cross-sectional contributions from different channels to obtain the above hybrid encoding e(vk) + as can be seen. Thus, the second-stage combiner COM is understood as a mathematical function f, and each "vote" (v k ) encoded as a vector or other index structure i is composed of TIFF2025520268000006.tif1582.

[0108] The location-specific component v pAlthough it may be included in this specification, this is optional. Localization data can be fed into the hidden layer as shown in the illustration.

[0109] The statistical data ST can be encoded as a list (vector) or matrix containing relevant statistical descriptors (percentages for things like the estimated misdiagnosis probability, optionally including means, variances, etc.) to describe misdiagnosis for each medical condition. For example, the indices of the encoded vector can be associated with each medical condition, and the entry at that index can be related to the statistical descriptor of the state at that index. Although not shown in FIG. 3, there may be a channel for the statistical data ST in addition to any one of the context data channels.

[0110] Consistent with the notation used in this specification, the context data channels are schematically indicated by "x" in FIG. 3, and the core data channels are indicated by "c". The specific order (top to bottom) of the channels in FIG. 3 is not important.

[0111] Next, refer to FIG. 4 showing a training system TS that can be used to train the machine learning architecture described in FIG. 3.

[0112] The preprocessor PP (encoder and / or combiner) and the downstream transformer TRF may be trained together as a whole (which may be preferred in some situations), or may be trained separately. In the latter case, a transfer training set-up can be used. Whether to train together or separately may depend on the exact structure of the loss function, and either option is specifically contemplated in this specification in the embodiments.

[0113] Also, it should be understood that the input data in training or deployment or testing can be provided separately in different channels as matrix or vector data as described. However, in alternative embodiments, these can be integrated, for example, into a multi-dimensional "cube" (tensor) and then processed together.

[0114] As described above, the training data can be supplied from a medical database TD that includes patient records from previous patients, preferably from a cohort of patients appropriately varied, where v_ represents the training input data corresponding to FIG. 3 (vk), and w_ preferably represents relevant findings (alternative or confirmatory) from historical cases that can serve as training labels in a supervised machine learning setup, although unsupervised training setups are not excluded herein.

[0115] Such alternative findings can be found as described in a medical surveillance database where misdiagnosed cases investigated in the past are recorded. These records are likely to include the correct (alternative) results that can be used as the label w_ associated with the training input data v_. On the other hand, the training input data v_ includes the aforementioned core elements such as the initial submitted report, including the first incorrect or wrong findings, and the image data it was based on. As described above, one or more appropriate ones of the mentioned context data c can be used herein. Thus, the training data includes instances of each historical initial finding (which may be incorrect) and subsequent correct findings so that the training system TS can learn the aforementioned potential relationships. Preferably, the training data includes some correct cases where the findings in the input v_ and the label w_ are the same, providing better robustness and learning performance. Thus, the training system TS is also exposed to samples of the "correct" diagnostic training material for better separation from material representing misdiagnosis.

[0116] As described above as a separate input channel to the machine learning model, including the statistical data ST for misdiagnosis for each finding type as context data is understood to help achieve certain robustness, accuracy, and rapid learning. However, the statistical data ST is generally no longer necessary in deployment or testing. The same is true for other types of context data c. The application of the core input in deployment is sufficient, but the user may still choose to "re-open" one or more additional input channels to add context data in deployment if performance improvement is required in some embodiments in some cases.

[0117] Therefore, two processing phases can be defined for the machine learning model, namely, the training phase and the subsequent deployment (or inference) phase.

[0118] In the training phase, prior to the deployment stage, the model is trained by adapting its parameters based on training data. Once trained, the model can be used in the deployment stage to calculate alternative findings if any. Training may be a one-off operation or repeated after new training data becomes available.

[0119] In the training phase, a set of initial weights is pre-input into the architecture of a machine learning model M = PP,TRF such as the NN network shown in FIG. 3. The weights θ of the model NN parameterize M θ and it is the purpose of the training system TS to optimize and thus adapt the parameter θ based on the training data (v― k ,w― k ). In other words, learning can be mathematically formulated as an optimization scheme where the cost function F is minimized, but a dual formulation that maximizes a utility function can be used instead.

[0120] Training is the process of adapting model-based parameters to training data. An explicit model is not always necessary, as in some cases the training data itself that constitutes modes such as clustering techniques or k-nearest neighbors is the explicit model. In explicit modeling such as NN-based approaches and many other approaches, the model can include at least partially interconnected inputs and / or outputs as a system of model functions / calculation nodes. The model functions or nodes are associated with parameters θ that are adapted during training. The model functions can include convolvers and / or weights of non-linear units such as RELU, as described above in Figure 3 in relation to NN-type models. In the case of an NN, the parameters θ can include the weights of the operator CV and / or the convolution kernels of the non-linear units. The parameterized model can be formally described as M θ =PP, TRF. Parameter adaptation can be carried out by numerical optimization procedures. The optimization can be iterative. An objective function F can be used to guide or control the optimization procedure. The parameters are adapted or updated such that the objective function is improved. The input training data v― i is applied to the model. The model responds to produce the training data output M(v― i )= TIFF2025520268000007.tif928. The objective function is a mapping from the parameter space to a set of numbers. The objective function f measures the combined deviation between the training data output TIFF2025520268000008.tif97 and each respective target w― i . The parameters are iteratively adjusted such that the combined deviation decreases until a stopping condition preset by the user or designer is met. The objective function may quantify the deviation using a distance metric ||.||.

[0121] In some, but not all, embodiments, the combined deviation is over the training data instances / pairs (v― i , w― i ) iBased on this, it may be implemented as a sum over some or all of the residuals, and the optimization problem regarding the objective function may be Formulated as TIFF2025520268000009.tif17133.

[0122] Although not limited in this specification, in setup (1), the optimization is formulated as the minimization of the cost function F, but a dual formulation that maximizes the utility function may be used instead. The sum is for training data instance i.

[0123] The cost function F can be pixel / voxel-based, such as the L1 - or smoothed L1 norm, L2 norm, Hubert, or soft margin cost function. For example, in the least squares method or a similar approach, the (squared) Euclidean distance - type cost function of (1) can be used for the above - mentioned regression task for regression to the output findings. When configuring the model as a classifier that classifies the findings, the sum of (1) is formulated as cross - entropy or negative log - likelihood (NLL) divergence or something similar.

[0124] The exact functional configuration of the update UP depends on the optimization procedure being executed. For example, using an optimization method such as back / forward propagation or other gradient - based methods, all or a subset (v - k , w - k ) of the combined residuals from the complete training data set can be used to adapt the parameters θ of the model M to reduce them. Such a subset may be called a batch, and the optimization may proceed batch - wise until all of the training data set is exhausted or a predefined number of training data instances are processed.

[0125] The training may be a one - off operation or may be repeated when new training data becomes . Available.

[0126] Referring to FIG. 5, FIG. 5 particularly shows a flowchart of a computer-implemented method for calculating a confirmed finding or an alternative finding w based on core input data c including an initial finding (such as a medical report PR) generated by a user and image data I based on the finding generated by the user. Optionally, context data x can be used together with core data c. However, the method described herein is not necessarily tied to the above architecture, and it will be understood that the following can be understood as a teaching in itself. This method is preferably based on an ML model. Such a model is assumed to be trained on training data.

[0127] In step S510, during testing or deployment, input data including the initial finding and its associated image I is received. Optionally, any of the above types of context data may be further used.

[0128] More specifically, in step S510, input data such as a report including an initial finding generated by a radiologist and a medical image I related to and based on the report is received. Alternatively, the original finding is first extracted from the report and the finding so extracted is received. Thus, since the report itself may contain useful context data herein, this can still be done, but it is not necessary to process all reports.

[0129] In step S520, the input data is co-processed by a machine learning model to generate an output finding. The output finding may be an alternative to or a confirmation of what is included in the initial input data.

[0130] If any, in step S530, an alternative finding is output. Alternatively, it can be confirmed that the output finding matches (is equal to) the initial finding, and this can also be output.

[0131] In step S540, the output finding is displayed, sounded, or otherwise draws the user's attention by controlling an appropriate transducer based on the calculated output finding w.

[0132] The machine learning model used in step S520 can perform a two-step operation that first includes a preprocessing S520_1 and then a conversion S520_2 operation. The preprocessing can include an encoding operation S520_11 and then a combiner operation S520_12. In particular, the preprocessing can include using an ML model to encode (raw) input data by the encoder part of the model. The encoded data is then combined by the combiner part of the model into combined encoded data. The combined encoding is then provided at S520_13 for further ML processing. Specifically, next, in the second step S520_2, the combined encoded data is transformed by the transformer stage of the machine learning model.

[0133] The transformation operation transforms the combined encoded output into an output finding. The transformation can include regression or classification to text data or encoded data that, if applicable, represents the output finding. The output finding may be a confirmation of an alternative finding or an initial finding.

[0134] The preprocessing and transformation operations may each be configured as separate ML models. The two models can be considered part of a supermodel that includes the two models. Some or each model may include submodels. The encoder of the preprocessing model can include partial ML models (such as sequential data processing or image classifiers), and the combiner can include a separate ML model. The transformer model may itself be implemented as a sub-ML model. Thus, the super ML model can include multiple ML submodels, which are nested and / or arranged in sequence.

[0135] The combined encoded data generated by step S520_12 includes contributions from both the input image and the initial findings, and optionally, contributions from context data. For example, the combined encoded data may be represented as a feature map in a CNN or other NN model. The encoder encodes the data into matrix or vector form. The encoder may be used at least in part for some or each channel as the separate one-hot encoding scheme described above. The transformation operation or combiner can include a weighted linear combination of logits from the previous layer, with non-linearity applied to generate contributions, which can then be passed to the nodes of the next layer.

[0136] A recurrent network may be used in the encoding operation to process in particular a portion of the input data that includes patient record history data, which has variable size or dimensions. Thus, other data types within different channels where the expected dimension / size, such as the length of the history data, is variable can also be processed.

[0137] The encoding operation can include classifying the input image (where the initial findings are relevant) into a classification vector. The indices of this vector can represent things such as medical conditions, and each entry represents the corresponding score for each state. This vector can be "one-hot" (binarized) by setting the index with the maximum score to "1" and invalidating all other entries.

[0138] As optional step S550, the calculated alternative findings are localized to the input image of the initial input data. This can include using a heatmap technique or the like that measures the contribution of pixels or pixel regions within the input image to the final result (alternative findings) and indicates those image pixel regions or areas that contribute more than a given threshold compared to the remaining pixels for localization. Gradient-based techniques applied to ML models treated as differentiable functions can be used as described above.

[0139] The output, i.e., the confirmatory or alternative finding, may be in text form, or may be a medical code or something else.

[0140] Referring now to the block diagram of FIG. 6, this shows a flowchart of a method for training the above-described machine learning models PP, COM. Preferably, both are trained together.

[0141] The training data v−, w− were described above with reference to FIG. 4.

[0142] At S610, training data is received in the form of pairs (v− k , w− k ). Each pair includes a training input x k and an associated target w− k , v− k as defined in FIG. 5.

[0143] In step S620, the training input v− k is applied to the initialized machine learning model M to generate a training output.

[0144] The deviation or residual of the training output M(v− k ) from the associated target w− k is quantified at S630 by a cost function F. One or more parameters of the model are adapted in one or more iterations within an inner loop at step S640 to improve the cost function. For example, the model parameters are adapted to reduce the residual measured by the cost function. The parameters particularly include the weights of the convolutional operator when a convolutional model M is used.

[0145] Next, the training method returns to step S610 in the outer loop, where the next pair of training data is supplied. In step S620, the parameters of the model are adapted such that the aggregated residuals for all pairs considered decrease, and in particular are minimized. The cost function quantifies the aggregated residues. Backpropagation or a similar gradient-based technique may be used in the inner loop.

[0146] Examples of gradient-based optimization include gradient descent, stochastic gradient, conjugate gradient, maximum likelihood, EM maximization, Gauss-Newton, etc. Methods other than gradient-based methods such as Nelder-Mead, Bayesian optimization, simulated annealing, genetic algorithms, and Monte Carlo methods are also envisioned.

[0147] More generally, the parameters of model NN are adjusted to improve an objective function F that is either a cost function or a utility function. In an embodiment, the cost function is configured to measure the aggregated residuals. In an embodiment, for all pairs for which the aggregation of the residues is considered, it is performed by a sum over all or some of the residues. In particular, in a preferred embodiment, the outer sum proceeds in batches (a subset of training instances), and the aggregated residuals are all considered at once when adjusting the parameters within the inner loop. Then the outer loop then proceeds to the next batch until the required number of training data instances have been processed. Instead of processing each pair, the outer loop accesses multiple pairs of training data terms at once, and the loop is batch-wise. Thus, the sum over the index "k" in the above equation (1) can be extended batch-wise over each entire batch.

[0148] Although mainly referred to NN above, the principles disclosed herein are not limited to NN. For example, instead of using NN for preprocessor PP, another technique such as a hidden Markov model (HMM), or a sequential dynamic system (SDS) may be used, among which in particular a cellular automaton (CA) may be used.

[0149] Similarly, for example, instead of using NN for the combiner COM, another ML approach such as a support vector machine (SVM) or a boosted decision tree can be used.

[0150] The components of the recommender MA are implemented as one or more software modules and executed on one or more general-purpose processing units PU, such as a workstation associated with the imager IA, or can be executed on a server computer associated with a group of imagers.

[0151] Alternatively, some or all of the components of the recommender MA can be configured as hardware, such as a properly programmed microcontroller or microprocessor, for example an FPGA (field programmable gate array), or as a hard-wired IC chip integrated into the imaging system IA, an application-specific integrated circuit (ASIC). In a further embodiment, the recommender MA can be implemented partly in software and partly in hardware.

[0152] The different components of the recommender MA can be implemented on a single data processing unit PU. Alternatively, some or multiple components are implemented on different processing units PU, and in some cases, are remotely located within a distributed architecture and can be connected within a suitable communication network such as a cloud setup or a client-server setup.

[0153] One or more of the features described herein can be configured or implemented as a circuit encoded within a computer-readable medium, or using a circuit, and / or as a combination thereof. The circuit can include discrete and / or integrated circuits, system-on-chip (SOC), and combinations thereof, machines, computer systems, processors and memories, computer programs.

[0154] In another exemplary embodiment of the present invention, there is provided a computer program or a computer program element, which is adapted to execute the method steps of the method according to one of the foregoing embodiments on a suitable system.

[0155] Accordingly, the computer program element may be stored in a computer unit which may be part of an embodiment of the present invention. This computing unit may be adapted to execute or induce the execution of the steps of the method described above. Further, it may be adapted to operate the components of the device described above. The computing unit may be adapted to operate automatically and / or to execute the order of a user. The computer program may be loaded into the working memory of a data processor. Accordingly, the data processor may be equipped to execute the method of the present invention.

[0156] This exemplary embodiment of the present invention includes both a computer program for using the present invention from the start and a computer program for changing an existing program into a program using the present invention by an update.

[0157] Furthermore, the computer program element can provide all the steps necessary to fulfill the procedure of the exemplary embodiment of the procedure described above.

[0158] According to a further exemplary embodiment of the present invention, a computer-readable medium such as a CD-ROM is presented, the computer-readable medium having computer program elements stored thereon, and the computer program elements being described by the previous section.

[0159] A computer program can be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium, which is supplied together with or as part of other hardware, in particular, but not necessarily, a non-transitory medium, and can also be distributed in other forms, such as via the Internet or other wired or wireless electrical communication systems.

[0160] However, a computer program may also be presented via a network such as the World Wide Web and downloaded from such a network into the working memory of a data processor. According to a further exemplary embodiment of the present invention, a medium is provided for making available computer program elements for downloading, and these computer program elements are configured to execute a method according to one of the foregoing embodiments of the present invention.

[0161] It should be noted that the embodiments of the present invention are described with reference to different subjects. In particular, some embodiments are described with reference to method-type claims, and other embodiments are described with reference to apparatus-type claims. However, unless otherwise notified, those skilled in the art will gather from the above and following descriptions that any combination of features belonging to one type of subject, as well as any combination between features related to different subjects, will be considered to be disclosed in this application. However, combining all features may provide a synergistic effect greater than the simple sum of the features.

[0162] Although the present invention has been illustrated and described in detail in the drawings and the foregoing description, such illustrations and descriptions should be regarded as exemplary or illustrative and not restrictive. The present invention is not limited to the disclosed embodiments. Other modifications to the disclosed embodiments can be understood and implemented by those skilled in the art when implementing the claimed invention from the study of the drawings, the disclosure, and the dependent claims.

[0163] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit can fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used advantageously. No reference signs in the claims shall be construed as limiting the scope. Such reference signs may consist of numbers, letters, or any combination of alphanumeric characters.

Claims

1. A machine learning device for processing medical data, comprising a preprocessor (PP) component and a machine learning system (MLS), The aforementioned preprocessor (PP) component is: An input interface (IN) for receiving initial findings generated by a human user for a patient and medical images relating to those findings, Encoded data including encoded image data and encoded finding data includes an encoder (ENC) for encoding the findings and the medical image, A coupler (COM) component for combining the encoded findings and encoded image data into combined encoded data, An output interface (OUT) for providing the combined encoded data to the machine learning system and It has, The machine learning system includes a machine learning model (TRF) configured to convert the combined encoded output into output data exhibiting at least one second finding, wherein the at least one second finding is a substitute for the initial finding. The aforementioned at least one alternative finding indicates a medical condition or disease, The machine learning device ensures that the at least one alternative finding is directed to the user's attention. Machine learning device.

2. The apparatus according to claim 1, wherein the input interface (IN) is configured to receive context data providing context information relating to a report and / or the image, the encoder is configured to encode at least a portion of the context data into encoded data, and the coupler is configured to combine the encoded context data with the image and the encoded report to obtain the combined data.

3. The apparatus according to claim 1, wherein the context data includes one or more of the following: i) patient history, ii) requests for imaging the image, and iii) statistical data relating to misdiagnosis.

4. The apparatus according to claim 1, wherein the coupler (COM) and / or the encoder (ENC) are implemented as machine learning models, respectively.

5. The apparatus according to claim 4, wherein the mechanical model for the encoder (ENC) includes a processing channel (SDP) configured for iterative processing.

6. The apparatus according to claim 5, wherein the processing channel (SDP) is configured to process at least the encoded patient history.

7. The apparatus according to claim 1, wherein the output includes a natural string or a medical finding code.

8. The apparatus according to claim 7, further comprising a localizer (LC) configured to map the output data to image positions within the image.

9. A training system (TS) configured to train the machine learning model (ENC, COM, TRF) described in claim 4 based on training data.

10. A method for preprocessing medical data for machine learning, The steps include receiving initial patient findings generated by a human user and medical images related to those findings, The steps include encoding the aforementioned findings and the aforementioned medical images into encoded data which includes encoded image data and encoded finding data, The steps include combining the encoded findings and the encoded image data into combined encoded data, The steps include providing the combined encoded data to the machine learning system. A step of using the machine learning system to convert the combined encoded output into output data showing at least one second finding, wherein the at least one second finding is a substitute for the initial finding, and the at least one substitute finding indicates a medical condition or disease. A step of ensuring that at least one alternative finding is brought to the attention of the user. A method having.

11. A method for training the machine learning model (ENC, COM, TRF) described in claim 4 based on training data.

12. A computer program element adapted to cause at least one processing unit (PU) to perform the method described in claim 10 when executed by at least one processing unit.

13. At least one computer-readable medium storing the program element described in claim 12, or the machine learning module described in claim 4.

Images