Read error reduction by machine learning assisted alternative discovery suggestions

Abstract

A preprocessor (PP) component and related method for a machine learning system (MLS) for processing medical data. The preprocessor comprises an input interface (IN) for receiving a human-generated initial discovery for a patient and a medical image related to the discovery. An encoder (ENC) of the preprocessor encodes the discovery and medical images into encoded data, including encoded image data and encoded discovery data. A combiner (COM) component of the preprocessor combines the encoded discovery and encoded image data into combined encoded data. An output interface (OUT) provides the combined encoded data to a machine learning system. The proposed preprocessor (PP) may be utilized to achieve more robust machine learning performance.

Description

Technical Field

[0001] The present invention relates to a preprocessor component for a machine learning model for processing medical data, a related method, a machine learning device including a preprocessor component and a machine learning model, a training system for training a machine learning model, a method for training a machine learning model, a computer program unit, and a computer-readable medium. Background Art

[0002] The interpretation (referred to herein as "reading" or "review") of radiological studies is a difficult task.

[0003] It is estimated that at least 5% of patients experience some form of diagnostic error, which contributes to up to 17% of adverse errors in hospitals.

[0004] Approximately 75% of those errors are centered around radiology practice. Most research devoted to preventing radiology errors has focused on avoiding false negatives, paying particular attention to the radiologist's fatigue and its impact on image perception and interpretation, as well as his or her work ergonomics.

[0005] However, to date, radiologists’ correctness has rarely been verified, as it is assumed that radiologists have enough information to provide a correct interpretation, if not due to fatigue.

[0006] However, there are reports in the literature that even in the absence of fatigue, the radiologist's interpretation may not always be correct. Certain radiological findings may be easily confused with each other. Examples reported in the literature include lymphoma misinterpreted as hematoma, or confusion between different causes of pulmonary consolidation, such as watery exudate, purulent exudate, hemorrhage, or cellular tumor / chronic inflammation.

[0007] Causes of misinterpretation range from inadequate radiologist experience, confusing patient histories, or incorrect imaging changes. Errors in findings or image readings can include the need for re-imaging, unnecessary surgery, or even patient death. Summary of the invention

[0008] Therefore, there may be a need to facilitate reducing error rates in image reading.

[0009] The objects of the invention are achieved by the subject-matter of the independent claims, wherein further embodiments are incorporated in the dependent claims.

[0010] It should be noted that the aspects of the invention described below also apply to related methods, machine learning arrangements comprising a preprocessor component and a machine learning model, training systems for training machine learning models, methods for training machine learning models, computer program units and computer-readable media.

[0011] According to a first aspect of the present invention, there is provided a pre-processor component for a machine learning system for processing medical data, comprising:

[0012] at least one input interface for receiving human-generated initial findings for a patient and medical images associated with the findings,

[0013] an encoder for encoding findings and the medical image into encoded data, the encoded data comprising encoded image data and encoded findings data;

[0014] a combiner component for combining the encoded discovery and encoded image data into combined encoded data, and

[0015] An output interface for providing the combined encoded data to a machine learning system.

[0016] In various embodiments, an input interface is used to receive context data, the context data providing context information related to a report and / or an image, an encoder is configured to encode at least a portion of the context data into encoded data, and a combiner is used to combine the encoded context data with the image and the encoded report to obtain combined data.

[0017] In various embodiments, the contextual data includes any one or more of: i) patient medical history, ii) imaging requests for images, iii) statistics related to misdiagnosis.

[0018] In various embodiments, the combiner and / or encoder are implemented as a machine learning model.

[0019] In various embodiments, a machine model for an encoder includes a processing channel configured for loop processing.

[0020] In various embodiments, the processing channel is configured to process at least an encoded patient medical history.

[0021] In various embodiments, the expected size of the encoded patient history is variable.

[0022] In various embodiments, the encoded data comprises at least one entry in a matrix or at least one vector. This allows for efficient computational implementation.

[0023] In various embodiments, at least one of the vectors comprises a one-hot vector, although other encoding techniques may alternatively be used.

[0024] On the other hand, a machine learning device is provided, comprising a pre-processor component and a machine learning system according to any one of the embodiments mentioned above.

[0025] In various embodiments, the machine learning system includes a machine learning model configured to convert the combined encoded output into output data indicative of at least one second finding that is a replacement for or equal to the initial finding.

[0026] In various embodiments, the output includes: a natural text string or a medical finding code.

[0027] In various embodiments, the arrangement comprises a localizer configured to map the output data to image locations in the image.

[0028] In another aspect, a training system is provided, which is configured to train a machine learning model of any or both of the mentioned embodiments based on training data.

[0029] In another aspect, a method for preprocessing medical data for machine learning is provided, comprising:

[0030] receiving an initial human-generated finding for a patient and a medical image associated with the finding,

[0031] encoding the findings and the medical image into coded data, the coded data comprising the coded image data and the coded finding data;

[0032] combining the coded discovery and coded image data into combined coded data, and

[0033] The combined encoded data is provided to a machine learning system.

[0034] In another aspect, a method for processing combined encoded data provided by another machine learning model is provided. Specifically, the method may include converting the combined encoded output into output data indicating at least one second finding, the second finding being a replacement for or equal to the initial finding.

[0035] On the other hand, a method for training a machine learning model of any one of all the embodiments or aspects mentioned above based on training data is provided.

[0036] In another aspect, a computer program element is provided which, when executed by at least one processing unit, is adapted to cause the at least one processing unit to perform a pre-processing method or a training method.

[0037] In another aspect, at least one computer-readable medium is provided, having a program element stored thereon, or having a machine learning model stored thereon.

[0038] A medical discovery is a decision about a medical condition that is made based on partially available information. For example, a decision such as "Did this 47-year-old patient have a heart attack?" Contextual data allows the addition of potentially relevant information (such as "the patient is a male" and / or "has a heavy smoking history", etc.). By using such contextual data, the decision process becomes more robust, but may be at the expense of speed or computing resources. However, there is (irrelevant) information that does not improve the robustness of the system while still spending processing time. A preprocessor such as the one proposed in this article preprocesses information from different sources in order to balance the information according to its relevance in order to improve its relevance to the desired output (discovery). The adverse effect on computing time is mitigated by preferably parallelizable algorithms that can be run on dedicated hardware such as GPUs or other.

[0039] The encoded combined findings produced by the preprocessor are preferably elements of a vector space. The encoded combined findings preferably include encoded contributions from multiple (such as all) data types initially received (such as one or more of the input image, initial findings, and optionally context data). This represents a balance of input data that can be more robustly processed by the converter to selectively rebalance the relevance of varying data types for the findings to be computed.

[0040] The preprocessor may be used in the proposed machine learning ("ML") module to work with the radiologist, preferably in real time, to suggest, for example, alternative interpretations / findings while he or she is filling out a report, thereby reducing reading errors.

[0041] The ML-based recommendation module as proposed in the embodiments herein analyzes the radiologist's interpretation of the study and suggests different possible interpretations that draw the radiologist's attention. By this double-checking option, reading errors can be reduced. By helping to reduce reading errors, costs can be reduced. The current cost of misdiagnosis is a staggering 1.7 to 2.9 billion USD per year, which the health department might otherwise spend elsewhere with more benefits. It is expected that this reading error reduction will translate into better patient care overall by reducing the number of re-imaging processes or the number of unnecessary interventions caused by misdiagnosis.

[0042] The proposed ML module is able to process radiologist reports in freehand or any type of unstructured form. There is no need for structured reports such as tables, lists of checkable marks, etc.

[0043] The proposed system and method can be applied to all types of radiological modalities, such as chest X-ray, CT, MRI, PET or ultrasound studies.

[0044] Although use of the preprocessor in such a module is preferred herein, such use does not exclude other uses (including standalone uses), where the data of the preprocessor can be used by itself, such as in medical data analysis to explore relationships between data from different sources.

[0045] A "user" refers to a person who operates the imaging device or oversees the imaging process, conducts the image review / reading session, such as medical staff or other persons, such as a radiologist. In other words, a user is typically not a patient.

[0046] In general, the term "machine learning" includes a computerized arrangement (or module) that implements a machine learning ("ML") algorithm. Some such ML algorithms operate to adjust a machine learning model that is configured to perform ("learn") a task. Other MLs operate directly on training data without having to use a model such as a model. This adjustment or update of the training data corpus is called "training". In general, the task execution of an ML module can be measurably improved using training experience. The training experience can include suitable training data and model exposure to such training data. Task execution can be improved, and the data better represents the task to be learned. If the training data well represents the distribution of examples that measure the performance of the final system, the training experience helps to improve performance. Execution can be measured by objective testing based on the output generated by the module in response to feeding test data to the module. Execution can be defined according to a specific error rate to be achieved for a given test data. See, for example, TM Mitchell, "Machine Learning", page 2, section 1.1, page 6, section 1.2.1, McGraw-Hill, 1997. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Exemplary embodiments of the present invention will now be described with reference to the following drawings, which are not to scale unless otherwise specified, and in which:

[0048] Figure 1 is a block diagram of a medical arrangement for processing medical data including medical images;

[0049] Figure 2 Yes, you can Figure 1 a machine learning module for use in an arrangement of and capable of generating an alternative medical finding that can replace a finding in input data that can be received for processing by the module;

[0050] Figure 3 Shows Figure 2 A block diagram of the architecture of the machine learning module in ;

[0051] Figure 4 It is shown that the training Figure 2or a computer-implemented training system for a machine learning model used in module 3;

[0052] Figure 5 A flow chart illustrating a computer-implemented method of processing medical data, in particular for calculating alternative findings that may replace findings included in initial input data; and

[0053] Figure 6 A flow chart of a computer-implemented method for training a machine learning model is shown. DETAILED DESCRIPTION

[0054] Reference now Figure 1 , Figure 1 A block diagram of a medical arrangement MAR for processing medical data, in particular measurements about a patient is shown.Broadly, the arrangement comprises a medical measurement device, such as a medical imaging device IA, which generates measurements about a patient.

[0055] The measurements taken on the patient may include medical images. Based on the medical images, the computer-implemented medical recommender module MA, preferably machine learning-implemented, is operable to calculate one or more alternative findings that may replace the findings provided by the human medical user. The operation of the medical recommender module MA is based on the initial findings provided by the user and on the measurements, in particular on the images I on which the initial findings of the user are based.

[0056] Thus, the medical recommender MA is configured herein to operate with the radiologist, preferably based on machine learning, to suggest alternative findings to one or more findings that a human radiologist would have made when examining the same images regarding the same patient.

[0057] The activity of a user examining a measurement (such as an image) in order to arrive at his or her findings is referred to as "reading". Broadly speaking, the recommender module MA helps to reduce the risk of errors in reading medical measurements, in particular in reading images. In the following, we will exclusively refer to medical images as one example of such medical measurements, while understanding that the principles described herein are equally applicable to other types of medical measurements, such as laboratory data (e.g., blood samples), ECG, EG or any other such medical data describing the (medical) state of a patient.

[0058] Thus, the machine learning recommender MA operates to analyze the imagery and, in addition, to analyze the initial findings of a medical user (such as a radiologist) to derive / infer possible alternative findings, if any. If there is no alternative finding, that is, if the finding derived by the recommender module MA is the same or sufficiently similar to the finding provided by the radiologist, this fact may be appropriately marked by a confirmation signal or other signal. If the finding generated by the machine learning recommender MA is different from the user's initial finding, it is an alternative finding, and this fact may also be indicated on the display device DID in a graphical, numerical or any other form or by using any other suitable transducer. Alternatively or in addition to visualization, alternative findings calculated by the recommender may be stored or otherwise processed. In the following, we will refer to "finding" in the singular, with the understanding that there may be multiple findings involved, such as those generated by the recommender MA or a user. Therefore, references to "finding" herein do not necessarily mean a single finding (although this is not excluded herein), but should be interpreted as references to "at least one finding". Generally, as understood herein, a "finding" is a specification of medical terminology, in coded form or in natural language or graphically, etc., that describes one or more aspects of a patient's medical status. Thus, a finding may include an indication of a disease, condition, ailment, etc., about a patient, or the absence thereof ("the patient is healthy").

[0059] Before explaining the arrangement MAR and in particular the operation of the recommender module MA in more detail, some components of the imaging apparatus IA will first be explained. Typically, the imaging apparatus IA may include a signal source SS and a detector device DD. The signal source SS generates a signal, such as an interrogation signal, which interacts with the patient to generate a response signal, which is then measured by the detector device DD and converted into measurement data, such as the medical image. An example of an imaging device or imaging apparatus is an X-ray based imaging device, such as a radiographic device configured to generate protective images. Volumetric tomographic (cross-sectional) imaging, such as via a C-arm imager or a CT (computed tomography) scanner or others is not excluded herein.

[0060] During the imaging session, the patient PAT may reside on a patient support PS (such as a patient couch, bed, etc.), but this is not required, as the patient may also stand, squat, or sit, or assume any other body posture in the examination area during the imaging session. The examination area is formed by the spatial portion between the signal source SS and the detector device DD.

[0061] For example, in a CT setting, during an imaging session, an X-ray source SS is rotated around an examination region with a patient therein to acquire projection images from different directions. The projection images are detected by a detector device DD, in this case an x-ray sensitive detector. The detector device DD can be rotated relative to the x-ray source SS around the examination region, but such a joint rotation is not necessarily required, such as in CT scanners of the 4th generation or higher. A signal source SS, such as an x-ray source (X-ray tube), is activated so that an x-ray beam XB is emitted from a focal point in the tube during the rotation. The beam XB passes through the examination region and the patient tissue therein and interacts with the examination region and the patient tissue, resulting in modified radiation to be generated. The modified radiation is detected by the detector device DD according to intensity. The detector DD device is coupled to an acquisition circuit, such as a DAQ, to capture the digital projection image as a digital image. The same principle applies to (planar) radiography, except that the source SS is not rotated during imaging. In such a radiographic setting, it is the projection image that can then be examined by the radiologist. In a tomography / rotation setting, the multi-directional projection images are first processed by a reconstruction algorithm that converts the projection images from the projection domain into cross-sectional images in the image domain. The image domain is located in the examination area. Projection images or reconstructed images will no longer be distinguished in this article, but will simply be referred to as (input) images I or input images I. It is such input images I that can be processed by the recommender MA.

[0062] However, the input image I may not necessarily be generated by X-ray imaging. In contrast to the previously mentioned transmission imaging modalities, other imaging modalities are also contemplated herein, such as radiographic imaging, such as SPECT or PET, etc. Furthermore, in some embodiments, magnetic resonance imaging (MRI) is also contemplated herein.

[0063] In an MRI embodiment, the signal source SS is formed by a radio frequency coil, which may also serve as a detector device DD, configured to receive, in a receive mode, a radio frequency response signal emitted by a patient located in a magnetic field. Such a response signal is generated in response to a previous RF signal emitted by the coil in a transmit mode. However, in some embodiments, there may be dedicated transmit and receive coils, rather than using the same coils in the different modes.

[0064] In radiographic imaging, the source SS is in the patient's body in the form of a previously administered radiotracer which emits radioactive radiation which interacts with the patient's tissues. This interaction generates gamma signals which are detected by a detection device DD (in this case a gamma camera) which is preferably arranged in a ring around the examination region in which the patient is located during imaging.

[0065] As an alternative or in addition to the above mentioned modalities, ultrasound (US) is also envisaged, wherein the signal source and detector devices DD are suitable acoustic US transducers.

[0066] The image I generated by any modality IA can be transferred to the (non-volatile) memory MEM via the communication interface CI, where the image I can be stored for later viewing or other processing. However, this document does not exclude an online setting, in which the image is viewed when it is generated by the imaging device. Nevertheless, in many cases, an offline setting may be sufficient or easier, in which the image is first stored in the memory MEM, preferably associated with the corresponding patient ID. The image memory can be non-volatile, such as a medical image database such as PACS. Once the user wishes to view (or "read"), the stored image of the patient of interest is accessed through a suitable database query system using, for example, the patient ID or other tags. The accessed image can be passed to the viewer software VIZ. The viewer software can be operated to generate a visualization of the image as a graphical display, and then display the graphical display on the display device DID. The above viewing can be completed on any suitable mobile (laptop, desktop computer, tablet computer) or stationary (desktop computer, workstation, etc.) computing platform, on which the visualizer VIZ and DB query system can be run, or can be controlled from the computing platform when the visualizer VIZ and DB query system are remotely running, for example on a server.

[0067] The radiologist can view the images in one or more viewing sessions. The radiologist can enter his or her findings in medical terms into a structured or free text report file. The report file is a data structure located in the computer memory of the viewing system (not shown). The report file includes data (such as text strings, codes, etc.) indicating the findings. The report file will be referred to as a "report" in this article. The report can preferably be stored in association with the image on which the comment is based, and / or in association with the corresponding ID of the patient who took the image I. This allows all relevant information to be easily retrieved at a later time when needed, such as during viewing. The report can be stored in the same database as the image, or it can be stored in a different database, such as a patient record database (HIS or other).

[0068] Human viewers are often under stress and have to read a large amount of image material, such as in a busy clinic with a constant influx of new patients, some of whom may be trauma patients requiring immediate attention. In such an environment with high stress and high workload, reading errors may occur, wherein unfortunately the viewer may associate erroneous findings with the image, with potentially disastrous consequences for the patient. Errors of the false positive or even worse false negative type may occur in this way. Such erroneous readings may occur due to user fatigue, and / or may be caused by poor quality images. For example, key image information may be obscured, for example, by artifacts, misplaced labels, text information boxes, annotations, etc., or other such widgets that are later added to the image, for example, by overlays.

[0069] Examples of such misreadings or misdiagnoses include lymphomas that are mistakenly reported as hematomas, or confusion between different causes of pulmonary consolidation: watery exudates, purulent exudates, hemorrhages, or cell tumors / chronic inflammation. The causes of misunderstandings may be numerous. They can range from inadequate radiology experience to confusing patient histories and incorrect imaging, to name just a few.

[0070] To reduce the possibility of such erroneous readings, the recommender module MA processes the radiologist's report (which includes the initial finding) together with the input image I to compute alternative findings. The initial finding and the image associated with the finding form the core input data c.

[0071] As mentioned above, the alternative findings calculated by module MA can be brought to the attention of the radiologist. The alternative findings can be output in the form of sound, or can replace the original / initial findings previously assigned by the radiologist, or be displayed together with the original / initial findings. The radiologist can then choose to accept the alternative findings, thereby correcting the initial findings, or the radiologist can choose to reject the alternative findings and maintain the initial findings. Such acceptance and rejection operations can be completed by a suitable user interface (UI) arrangement, such as via a keyboard, touch screen, pointer tool (stylus, computer mouse), etc. In place of such alternative findings, the initial findings can be confirmed, as described above.

[0072] Preferably, the recommender MA operates while the radiologist is examining the visualizations. The visualizations are generated, for example, by a visualizer VIZ on the display device DID. The recommender module MA can thus interact with the visualizer VIZ to provide alternative or confirmed findings in a suitable visualization, preferably simultaneously with the image being viewed, and / or simultaneously with the displayed report.

[0073] As will be explained in more detail below, in some embodiments, the alternative findings may be located by the locator LC in the imagery upon which the review is based. The location may be achieved by displaying an overlay widget, such as a bounding box or other graphical indicator widget, to indicate the portion of the input image I that is associated with the alternative findings. The alternative findings may be indicated as text in code, free text or other form, preferably displayed relative to the bounding box, such as inside or adjacent to the bounding box or in any other spatially associable manner.

[0074] Now in Figure 2 The operation of the recommender module MA is shown in more detail in the block diagram of FIG.

[0075] As can be seen, the recommender MA processes input data. The input data includes core input data, which includes an image I viewed by a human viewer and original / initial findings such as included in a radiologist's report RP. The report RP may include text or other information indicating the findings made by the radiologist based on the input image I. The input image I and the report RP belong to the same patient. It is not necessary to include the report RP in the core input data in this document, although this is envisioned in some embodiments of this document. Instead, it may be sufficient to extract the initial findings from the report RP through an NLP (natural language processing) pipeline, string matching or other text or string processing and include the data indicating the findings so extracted in the input core data. The NLP pipeline itself can be implemented by a dedicated machine learning model (such as a BERT-type NN model) or other fully connected architectures configured for natural language processing.

[0076] Optionally, context data is processed as context input data x along with the core input data c. The context input data x provides context related to the patient and / or the findings. This may help the machine-based recommender MA to operate more robustly. The context data x may act as a regularizer during training. The training aspect is described in more detail later.

[0077] The context data x may include a plurality of different data items, either individually or in any combination. For example, the context data x may include a request RQ for a radiology study that resulted in the production of an input image. The request includes information that has resulted in a radiologist or other medical user requesting an imaging session during which the input image was produced. The imaging RQ data may include text data in a structured or unstructured format that indicates the reason for requesting imaging for image I.

[0078] Alternatively or additionally, the context data x may include patient reference data PH including or extracted from the patient's medical history, such as patient records, previous imaging, etc. This medical history data PH may be extracted from a database that may store this information about the patient at hand.

[0079] Additionally or alternatively, the context data x may include statistical data ST or a statistical analysis of previous misdiagnoses. This data type relates to previous cases, not necessarily about the instant patient, but to a group of other patients where misdiagnoses / wrong readings occurred, i.e., an initial finding produced by a radiologist was later found to be actually wrong.

[0080] Data ST of this type may typically include correct findings, as may have been discovered by subsequent investigations, which typically occur when it appears that the initial findings were actually wrong. Therefore, the review data of misdiagnosed cases includes valuable information, in which user-generated findings are related to their corresponding correct findings. Therefore, it can be derived from such data, for example by statistical analysis, which types of (diagnostic) findings are more prone to misdiagnosis than other findings. The core input data c (including images I and initial findings such as in reports RP), preferably enriched by contextual data x = (request RQ, statistics ST, patient history PH), can be processed by a machine learning-based module MA to produce, for example, one or more alternative findings W1, W2. Similarly, all described contextual data x may not necessarily be used. Any sub-combination of the above is contemplated herein. However, in particular, imaging requests RQ and / or statistical data ST that have been found to produce good results are included in the contextual data x. An example may be a trauma patient: findings may include "fractures", "organ lacerations" and "internal bleeding". Radiologists report trauma, as it should be. However, in April 2020, statistical trends may draw the impact of, for example, COVID 19, and the exhausted individual sends a request for a chest X-ray as an infectious patient. In this case, the system should discard the context or downweight the context and not provide a "pneumonia" recommendation. Five days later, when the patient begins to develop pneumonia on control X-rays, the system should ignore the context that the patient is a trauma patient or downweight that context and consider the finding of an infectious disease trauma patient.

[0081] The visualizer VIZ may indicate findings W1, W2 in text form TX1, TX2. Two such findings are shown, but there may be more or less than two findings. Alternative findings may be superimposed on the input image in question, such as in Figure 2On the images I1, I2 shown on the right. Bounding boxes BB1 and BB2 can be used to indicate image portions that contribute more than other image portions in the calculation process performed by the machine learning module MA when generating alternative findings W1, W2. As will be described in detail below, heat map techniques can be used to derive at the bounding boxes. However, any other localization technique can be used, such as where the input data includes explicit coordinates indicating portions of the image that support the discovery, and such coordinates are learned together with the image. Localization can be achieved by a localizer LC, the operation of which can be explained in more detail below.

[0082] The textual representations TX1 and TX2 of the alternative findings W1 and W2 may comprise text strings (such as in tree text form, natural language text), or may be in the form of medical terminology codes, such as the ICD (International Classification of Diseases) codes of the WHO (World Health Organization), such as version 11 or an earlier version or a future version. However, any other suitable medical coding may be used instead. In general, encoding as used herein is distinguished from encryption, in that the latter is irreversible without the knowledge of (multiple) encryption keys, while the former is recoverable, does not require such keys, and has a lower computational burden.

[0083] Reference now Figure 3 The block diagram in Figure 3 More details of a machine learning based recommender module MA are shown. Conceptually, there is a potential relationship between input data (i.e., core data c, possibly enriched with contextual data x) and its associated alternative findings. This relationship can be considered to be implicitly expressed in existing training data including historical images and related historical patient records. Such training data can be found in medical databases and can involve previous examinations for appropriately representative patient groups.

[0084] The underlying relationships may be difficult (or even impossible) to model classically and analytically in a self-organizing manner. For this reason, a machine learning model is used that does not require such explicit modeling. A standard model with parameters adapted during the learning / training process based on the training data can be used. The parameters can be adjusted in iterations until the adapted model is considered to be a sufficiently good approximation of the underlying relationship (in a predefined manner based on a cost function). Various aspects of learning will be described in more detail below.

[0085] Continue to refer Figure 3 , and in more detail, the figure represents a model architecture that can be used to build a recommender module MA. After appropriate training on the training data, the model is able to compute alternative discoveries (if any) given at least the core input data c as described above. Broadly, the model for the recommender MA can include a preprocessor PP and its downstream postprocessor, the latter of which is referred to herein as a recommender machine learning system MLS.

[0086] Both the pre-processor PP and the post-processing recommender machine learning system MLS may be implemented by dedicated respective machine learning models, as will be described in more detail below.

[0087] Broadly speaking, a post-processor PP processes input data v during training, testing, or deployment (i.e., in a real-world clinical post-training application) k (It includes core data ), and calculate the intermediate output e(v k ). The intermediate output is passed to the post-processor stage MLS, which then converts the intermediate output into a desired finding, such as an alternative finding w (if any) or a determined finding. The pre-processor PP and / or the post-processor MLS may be implemented as a corresponding trained machine learning model. In the following, it is assumed that the model has been trained. Figure 4 , Figure 6 Aspects of training are described in more detail in .

[0088] Referring to the pre-processor PP in more detail, raw input data rk is applied to the encoder ENC portion of the pre-processor PP. The raw data is in digital form and may be the result of conversion into such digital data by data capture, A / D conversion and character recognition, etc. For example, a handwritten report may be captured as an image and then subjected to OCR processing. Alternatively, the report or other textual data is generated by a word processing module, etc. The raw data is encoded by the encoder ENC into input data v for the second stage of the pre-processor PP. k The second stage is configured as a combiner COM. The encoded input data v for the combiner COM k The composer COM converts the encoded input data v into k Combined into the combined coded data e(Vk) + .

[0089] The combiner COM of the preprocessor PP can be configured as a set of computational nodes n of an artificial neural network (“NN”) ij Each node is associated with parameters (“weights”) that were previously adapted in training based on the training data. ij The layers may be arranged in a cascaded manner. In some embodiments, the combiner COM is arranged as a convolutional neural network ("CNN"), with the computing nodes implementing convolution operators.

[0090] The layers may include an input layer IL, IN, one or more hidden layers HL and an output layer OL, OT. The encoded input data is provided to the input layer v k, where it is processed and propagated to the output layer OL through one or more hidden layers HL. The output of the output layer OL may include a feature map representing intermediate data, which represents the input v k The encoded data e(v k ). For illustration only, Figure 3 , but there may be more than three layers. Figure 3 The fairly shallow network shown with only a single hidden layer performs well. Keeping the network shallow with a single or a few (such as 2, 3 or 4) hidden layers allows for high responsiveness because less computation is required.

[0091] The input and output between the hidden layer HL and the output of the input layer IL and the output layer OL may be referred to herein as feature maps. For computational and memory allocation efficiency, feature maps may be represented as two-dimensional or higher-dimensional matrices ("tensors").

[0092] Preferably, some or all of the layers IL, HL are convolutional layers, i.e., comprise one or more convolutional filters that process the input feature maps from earlier layers into intermediate outputs, sometimes called logits. An optional bias term may be applied, for example, by addition. The activation operator of a given layer processes the logits in a nonlinear manner into the next generation of feature maps, which are then output and passed as input to the next layer, and so on. The activation operator may be implemented as a rectified linear unit ("RELU"), or as a soft-max function, a sigmoid function, a hyperbolic tangent function, or any other suitable nonlinear function. Optionally, there may be other functional layers, such as pooling layers or derivation layers to facilitate more robust learning. Pooling layers reduce the size of the output, while derivation layers cut off connections between nodes from different layers. The combined or "mixed" code e(vk) produced by the combiner COM + In an embodiment, it is one or more feature maps. Preferably, the encoding e(vk) + is equal to the number of encoded data streams (vk) that are fed to the input layer IL. However, there may be more or fewer such combined encodings e(vk) + .

[0093] Node n ij The computation function of is typically a (weighted) linear combination of the Logie values ​​from the previous layers, and also includes applying nonlinearities to the Logie values ​​so combined. However, other combinations of functions are also envisaged, not necessarily linear combinations.

[0094] The post-processor MLS may include a converter TRF stage. The converter is operable to convert the combined code e(vk) +Transformed into an output finding w, which is a replacement for the initial finding or a confirmation thereof. Similar to the preprocessor PP, the transformer TRF can be configured as a trained ML model. For example, the general setup of the transformer TRF can be similar to that of the combiner COM of the preprocessor PP, but the layers of the transformer TRF are preferably not convolutional, but two or more fully connected layers, especially in cases where a classification result is sought, such as in classifying the input feature map from the combiner COM into the output finding. If the input is to be regressed into the finding, then again, fully connected layers can be used, although if necessary, specially configured convolution kernels with spatially adapted convolutions and padding can be used instead in some embodiments. However, a fully connected architecture is preferred in this article.

[0095] More broadly, the transformer-level TRF can include an attention mechanism to learn spatial dependencies, which tend to be lost in a convolutional setting. The attention mechanism can be implemented by matrix multiplication or normalization or other. The decay mechanism allows the re-weighting of parts of the sequential input to, for example, model the language context. An autoencoder (AE) or a variational AE (VAE) can be used. The decay mechanism can be implemented as a submodel, such as a fully connected layer inserted between the encoder and decoder of an AE or VAE. The fully connected submodel as an attention mechanism can receive input from the outputs of the encoder and decoder. Such an attention mechanism can provide its output as input to the decoder. Variants of the attention mechanism can be implemented as dot products, query keys, or other. Such a mechanism reorganizes the input on the encoder side to redistribute those effects to each target output.

[0096] Preferably, the combiner COM is convolutional or hybrid, fully connected and convolutional, and the transformer TRF is fully connected.

[0097] Transformer TRF, a component of the downstream machine learning system MLS, converts the encoded information e(v k ), and generates an output recommendation w. For example, the output recommendation w may be a replacement for the initial recommendation in the report PR. In some cases, the output w is substantially equal to the initial finding, thus reassuring the user. If there is an alternative finding, the user may have more second details in the second viewing session, and may then select to accept or reject the alternative finding using a suitable UI arrangement (such as a GUI, etc.) as previously described.

[0098] The Transformer TRF can operate as a regression network or a classifier network. One such example of a Transformer TRF includes a fully connected NN instead of a CNN that can be used in a preprocessor PP architecture. For example, a BERT-type network such as that described by A Smit et al. in "CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT", arXiv:2004.09167, available online at https: / / arxiv.org / abs / 2004.09167 on 2021-10-12 can be used. The Transformer TRF is preferably a language model such as BERT or other. Its input (encoding e(v k ) + ) is preferably converted to a text output. For example, the transformer TRF may embed such an input into a feature vector. Thus, in some embodiments, the transformer TRF may be described as a text embedding language model.

[0099] The output of the transformer TRF is not necessarily a single finding, although this may be the case in embodiments. In embodiments multiple such findings may be output. The format may be in vector form, such as some of the inputs (vk), and the detection encoding is such that each index represents a finding, and each entry represents a score for that finding "i". The entries may not necessarily be probabilities, as more than one finding may be applicable. The transformer TRF may include a soft-max layer or any other normalizer to encode the mixture e(vk) in such a way that each entry of the output vector can be represented as such a score + The final output provided to the user may include an indication of only the findings with the highest scores in the output vector. Alternatively, a list of the first m findings with m>1 scores is provided. The vector index may be matched to an NL string describing the findings in NL, such as "PNEUMONIA", or alternatively, a number or other form of code may be provided. The string w may be displayed, stored, sent in a text message, email, output as sound, etc.

[0100] The localizer LC may be implemented as a separate component or as a component included in the transformer TRF or the preprocessor PP. The localizer LC is operable to identify one or more parts of the input image I. The identified parts include pixels that contribute more to the resulting output w than other pixels in the neighborhood or globally across the entire image plane. The contribution can be determined by thresholding or by observing the gradient behavior grad of the entire model (including the preprocessor PP and the transformer TRF). I g, viewed as a function w = g(I, θ*), where θ* is a parameter of the model. The localizer can be implemented by heatmap techniques such as GradCAM, GradCAM++, or other activation mapping techniques. Suitable techniques contemplated herein are described in “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization” by Ramprasaath R et al., arxiv:1610.02391 [cs.CV], available online at https: / / arxiv.org / abs / 1610.02391. See also (GradCAM++) "Generalized Gradient-based Visual Explanations for Deep Convolutional Networks" by Chattopadhyay A. et al., arxiv:1710.11063 [cs.CV], available online at https: / / arxiv.org / abs / 1710.11063v2. See also the concept of saliency maps also contemplated herein and described in "Deep Inside Convolutional Networks: Visualizing Image Classification Models and Saliency Maps" by Simonyan K et al., arXiv:1312.6034 [cs.CV], available online at https: / / arxiv.org / abs / 1312.6034.

[0101] The identified parts may be graphically rendered by the mentioned bounding boxes, as caused by, for example, a visualizer VIZ interacting with the output w provided by the recommender module MA.

[0102] The localization lc(w) of the finding w can alternatively be achieved by training the model to output (w,p), where p is a set of image coordinates defining the part of the input image associated with the output finding w. Thus, in this embodiment, the localization information p is local. Thus, the coordinates p are provided in the training data and are considered by the objective function during training. Thus, in this embodiment, the localizer LC is an implicit function in the trained model PP, COM.

[0103] The alternative finding may relate to an entirely different organ or tissue type in the patient compared to the initial / initial finding. However, this may not necessarily be the case. The alternative finding may still relate to the same organ as the radiologist's initial finding, but may instead represent a different diagnosis or medical insight related to the same organ.

[0104] The preprocessor PP and the downstream recommender machine learning system MLS (including the converter TRF) can be implemented on the same processing unit, such as on a server or other computing device. However, this is not necessarily envisioned in all embodiments herein. Distributed implementations are also envisioned. For example, the preprocessor PP can be implemented on one computing unit PU, while the downstream machine learning system MLS including the converter TRF is implemented on another processing unit PU' (not shown). Preferably, the processing units PU, PU' are communicatively coupled to each other so that the combined encoding e(vk)+ generated by the preprocessor PP can be provided to be combined by the downstream machine learning system MLS. For example, the processing units PU, PU' may be geographically distant from each other. The preprocessor PP can be implemented on one (or more) servers, while the system MLS can be implemented on the user's terminal device, such as on a handheld device (laptop, smart phone, etc.), or on a fixed device (such as a desktop computer, workstation, etc.). Alternatively, the MLS is implemented on one or more servers, and the preprocessor PP is run on such a user's terminal device.

[0105] Preferably, in order to achieve good throughput, the computing device PU comprises one or more processors (CPUs) supporting parallel computing, such as those of a multi-core design. In one embodiment, a GPU (Graphics Processing Unit) is used.

[0106] Now refer to it in more detail Figure 3 In the preprocessor PP in the encoder part ENC, the original input data item rk is processed in separate chains or channels (shown as Figure 3In particular, the original input data rk is encoded in a corresponding pure form in a separate processing channel into corresponding encoded input data vk. Such pure encodings can then be combined by the combiner COM into the intermediate result e(v k )+, which can now be called a hybrid encoding e(v k ). The intermediate encoding result e(v k ) is an ensemble encoding, but each intermediate encoding result is no longer pure, but includes the combined cross-contributions of pure encodings from other channels / chains. This is Figure 3 The node n is fed to the output OL 3j , each line carrying a contribution originating at least in part from some or all of the encoded input data vk. Thus, the combiner COM mixes or balances contributions vk from other / different channels.

[0107] The encoding result of this combination is e(v k )+ can vary. For a dimensionality reducer type combiner COM, the number can be lower than the encoding term v k The encoder ENC can be implemented as a mapper that maps the original input into the elements v of the vector space k , such as the corresponding vector or matrix or tensor, as appropriate.

[0108] The combiner COM will then convert the code from the pure code v k The contributions of the channels are mixed or balanced into an intermediate result, which is now a vector space element e(v k ) + When the combiner COM is configured as a CNN, such intermediate results may include feature maps. In some embodiments, some or each intermediate result e(v k )+ may include some or all other data items v from other channels k'! = k k’ contribution.

[0109] The encoder part ENC does not necessarily require machine learning elements, since this can be implemented as a LUT, for example, if the length / dimension of the raw data in the corresponding channels is known. However, the combiner COM is preferably implemented as an ML model as described above, such as a neural network NN, preferably a CNN, a fully connected network or a hybrid network, comprising computing n ij .

[0110] Some data types, such as patient histories PH, may vary in length or dimension, which may not be known a priori. For this type of data, preferably, an ML-based sequential data processor SDP may be provided in a chain / channel for processing the patient history data PH. This may be implemented as a recurrent neural network, which may be of convolutional type or long short-term memory (LSTM) type. In general, RNN describes all forms of NNs that carry their output back to their input. This includes fully connected or convolutional networks with a "loopback" facility, but also more complex architectures such as LSTM or gated recurrent units (GRU), which transmit two types of information, the so-called cell state C and the cell state C. t The output h of the nodes in the and layer t In contrast, the second-stage combiner COM of the pre-processor PP may be configured as a feed-forward network.

[0111] The original data mentioned earlier (r k ) may include text data (such as in a report PR, a request RQ, a patient history PH, etc.), numerical data (such as statistical data ST), or spatial pixel / voxel data (such as an input image I). In some channels, such encoding is not required. Such a channel can be implemented as a unit operator that passes its input unchanged to the combiner COM. For example, the input image data I may not need to be encoded in the first-level ENC because the image data is already inherently and appropriately encoded as a matrix of pixel values. However, in various embodiments, the channel for the image channel may still include a trained ML component, such as an image classifier that classifies the image into a vector of conditions, diseases, etc. Therefore, the input image I can be encoded into a vector. A CNN-type network with a classifier output layer (normalizer, such as a soft-max layer, etc.) can be used to configure such an ML component, for example, pre-trained appropriately on some training images that are appropriately labeled. The classifier and / or the sequence data processor SDP can be trained together with the combiner model COM and / or with the transformer model TRF. Typically, the ML components of the encoder-level ENC can be trained in advance or using models for the combiner COM and the transformer TRF.

[0112] The encoding at the encoder stage ENC and / or the combined encoding of the combiner COM can be provided in the form of a tensor, a matrix or a vector, as appropriate. In the following example, different types of raw data (r ) at the first stage encoder ENC are described in the form of vectors. k )'s encoding (v k ), it should be understood that this is exemplary rather than restrictive.

[0113] For example, natural language processing (NLP) methods can be applied to retrieve different findings from a radiologist's handwritten report. A BERT-type network, a trained neural network that converts a handwritten report into radiological findings, such as a list of "fractures, consolidation, enlarged cardiac mediastinum, no findings, pleural other, cardiomegaly, pneumothorax, atelectasis, support devices, edema, pleural effusion, lung lesions, lung opacity" can be used. These findings can be encoded by the encoder ENC into a one-hot vector, for example:

[0114]

[0115] The position of 1 indicates the presence of a finding, in this case “no finding”. In addition to leveraging the radiologist’s report RP, this model can be used to parse the request RQ for radiology studies. Through transfer learning, the same network (such as CheXbert or other BERT-type networks) can be adapted to process the request RQ for radiology studies into categories including “airway study, trauma, oncology, device validation”, etc., which can also be encoded as a one-hot vector:

[0116]

[0117] Finally, the same or similar NLP methods can be transferred to process the patient history PH of the disease into a corresponding vector. However, in this case, it is more appropriate not to use a one-hot vector because the number of diseases in each patient's history is different, and the time since remission may be an informative factor. The patient history PH encoded vector can instead be encoded for a binary of values, one encoding for the type of disease and a second encoding for the time since remission, for example:

[0118]

[0119] In this example, disease code "22" has been in remission for 168 months, disease code 5 has been in remission for 17 months, and disease code 64 has not yet been in remission, as indicated, for example, by a negative number. Such an implementation may be based on a lookup table (LUT) of diseases. Such a LUT may allow for establishing a correspondence between codes and conditions, e.g., "22" may be the code for "cancer," 5 for "fracture," and 64 for "pneumonia."

[0120] Instead of passing the image through the identity operator as described before, the input image I processing pipeline can include its own NN, such as a CNN, which can be configured to process the image I into categories. Again, a BERT-type setup like CheXbert can be used, but with image data as input. The output provided to the combiner can again be configured as a one-hot vector, where each entry represents the absence or presence of a category:

[0121]

[0122] As described above, the categories into which the image I is classified may represent a medical condition / disease, etc.

[0123] Thus, it can be seen that the first stage encoder ENC can provide different vectors encoding the disease / condition per channel using its own ML network, and the combiner COM merges these "votes" as the vectors / matrices can be called balanced votes with cross contributions from different channels to obtain the above hybrid encoding e(vk) + : Thus, the second-level combiner COM can be understood as a mathematical function f, where each “vote” (v k ) are considered:

[0124]

[0125] In this article, the positioning component v can also be included p , but this is optional. As shown in the figure, the positioning data can be fed into the hidden layer.

[0126] The statistical data ST may be encoded as a list (vector) or matrix including relevant statistical descriptors describing the misdiagnosis of each medical condition (estimated percentage of misdiagnosis probability, etc., optionally including mean, variance, etc.). For example, an index of the encoded vector may relate to the corresponding medical condition, and the entry at that index may relate to the statistical descriptor of the condition at that index. Figure 3 Not shown in FIG, but in addition to any context data channels, there may also be a channel for statistical data ST.

[0127] Consistent with the notation used in this paper, the context data channel is Figure 3 The core data channels are schematically indicated by “x” in the figure, and the core data channels are schematically indicated by “c”. Figure 3 The specific order of the channels in (from top to bottom) is not important.

[0128] Reference now Figure 4 , Figure 4 It shows that it can be used for training Figure 3 Training system TS for the machine learning architecture described in .

[0129] The preprocessor PP (encoder and / or combiner) and the downstream transformer TRF can be trained together as a whole (and this may be preferred in some cases), or can be trained separately. In the latter, a transfer training setting can be used. Training together or separately may depend on the exact structure of the loss function, and either option is specifically contemplated in the embodiments herein.

[0130] It should also be understood that the input data in training or deployment or testing can be provided separately as matrix or vector data in different channels, as described. However, in alternative embodiments, these can be combined into, for example, multi-dimensional "cubes" (tensors) and then processed together.

[0131] As mentioned above, the training data may originate from a medical database TD, which may include patient records from previous patients, preferably from a large number of patients in a suitably varied group. v~ indicates the training input data, corresponding to Figure 3 (vk), and w~ indicates an associated finding (replacement or confirmation) from historical cases, which can be used as training labels in the supervised machine learning setting preferably envisioned in this article, although an unsupervised training setting is not excluded in this article.

[0132] Such alternative findings as mentioned can be found in a medical audit database, where misdiagnosed cases that have been investigated in the past are recorded. These records most likely include correct (alternative) findings, which can be used as labels w~ associated with the training input data v~. On the other hand, the training input data v~ includes the core elements mentioned previously, such as the submitted initial report including the initial erroneous or incorrect findings, and the image data on which the initial report is based. As mentioned above, one or more of the mentioned contextual data c can be used in this article. Therefore, the training data includes corresponding historical initial findings (which may be erroneous) and later correct findings. Examples enable the training system TS to learn the above-mentioned potential relationship. Preferably, for better robustness and learning performance, the training data includes multiple correct cases, where the findings in the input v~ and the label w~ are the same. Therefore, the training system TS is also exposed to samples of "correct" diagnosis training material to better separate from material representing misdiagnosis.

[0133] It should be understood that including the statistics ST of misdiagnosis for each of the previously mentioned lookup types as context data as a separate input channel to the machine learning model helps to achieve certain robustness, accuracy, and fast learning. However, the statistics ST are generally no longer needed in deployment or testing. The same applies to other types of context data c. Although the user still has the option of "reopening" additional one or more input channels to add context data in the deployment in some embodiments (if performance needs to be improved in some cases), the application of the core inputs in the deployment is sufficient.

[0134] Therefore, two processing phases can be defined with respect to machine learning models: a training phase and a later deployment (or inference) phase.

[0135] In the training phase, the model is trained by adapting its parameters based on the training data before the deployment phase. Once trained, the model can be used in the deployment phase to calculate alternative discoveries (if any). Training can be a one-time operation or can be repeated as new training data becomes available.

[0136] In the training phase, the machine learning model M = PP, TRF (such as Figure 3 The architecture of the NN network in Figure 1 is pre-populated with an initial set of weights. The weights θ of the model NN represent the parameterized M θ , and based on the training data (v~ k ,w~ k ) Optimizing the parameters θ and thus adaptation is the goal of the training system TS. In other words, learning can be formulated mathematically as an optimization scheme where a cost function F is minimized, although a dual formulation of maximizing a utility function may be used instead.

[0137] Training is the process of adapting the parameters of a model based on training data. An explicit model is not necessarily required, as in some examples it is the training data itself that constitutes the pattern, such as in clustering techniques or k-nearest neighbors. In explicit modeling, such as in NN-based methods and many other methods, the model can include a system of model functions / computation nodes, where the system is at least partially interconnected with inputs and / or outputs of the model functions / computation nodes. The model function or node is associated with a parameter θ that is adapted in training. The model function can include the above-mentioned NN-type model in Figure 3 The parameter θ may include the convolution operator and / or weights of the nonlinear unit (such as RELU) mentioned above. In the case of NN, the parameter θ may include the weights of the convolution kernel of the operator CV and / or the nonlinear unit. The parameterized model can be formally written as M θ=PP,TRF. Parameter adaptation can be achieved through a numerical optimization process. The optimization can be iterative. The objective function F can be used to guide or control the optimization process. The parameters are adapted or updated so that the objective function is improved. Input training data v~ i is applied to the model. The model responds to produce the training data output The objective function is a mapping from parameter space to a set of numbers. The objective function f measures the output of the training data. With the corresponding target w~ i The parameters are iteratively adjusted to reduce the combined deviation until a stop condition preset by the user or designer is met. The objective function can use a distance measure ||.|| to quantify the deviation.

[0138] In some embodiments, but not all embodiments, the combined bias can be implemented as a function of the training data instance / pair (v~ i ,w~ i ) i The sum of some or all residuals of , and the optimization problem in terms of the objective function can be formulated as:

[0139] argmin θ F=∑ k ||M θ (x k ),y k || (1)

[0140] In setting (1), the optimization is formulated as the minimization of a cost function F, but is not limited to this, as a dual formulation of maximizing a utility function may be used instead. The summation is over training data instance i.

[0141] The cost function F may be pixel / voxel based, such as L1- or smoothed L1-norm, L2-norm, Hubert or soft-margin cost function. For example, in least squares or similar methods, the (squared) Euclidean distance type cost function in (1) may be used for the regression task mentioned above, with regression as the output lookup. When the model is configured as a classifier for classification discovery, the sum in (1) is instead formulated as one of the cross entropy or negative log-likelihood (NLL) divergence or similar divergences.

[0142] The exact functional composition of the updater UP depends on the optimization procedure implemented. For example, an optimization scheme (such as back / forward propagation or other gradient-based methods) can then be used to adapt the parameters θ of the model M in order to reduce all or a subset of the training pairs (v~ k ,w~ k). Such subsets are sometimes called batches, and optimization can proceed in batches until all the training data in the training data set has been exhausted, or until a predetermined number of training data instances have been processed.

[0143] Training can be a one-time operation, or can be repeated as new training data becomes available.

[0144] Reference now Figure 5 , Figure 5 A flowchart of a computer-implemented method for calculating a confirmed or alternative finding w is shown, specifically based on core input data c including an initial finding generated by a user (such as in a medical report PR) and image data I on which the user-generated finding is based. Optionally, context data x can be used together with the core data c. However, it should be understood that the method described herein does not necessarily rely on the above architecture, and the following can be understood as its own teaching. The method is preferably based on an ML model. It is assumed that such a model has been trained on training data.

[0145] In step S510, during testing or deployment, input data is received including the initial findings and their associated images I. Alternatively, any type of context data among the above types may be used in addition thereto.

[0146] More specifically, at step S510, input data is received, such as a report including initial findings generated by a radiologist and a medical image I, which is associated with the report and the report is based on the medical image I. Alternatively, the initial findings are first extracted from the report, and the findings so extracted are received. Therefore, it is not necessary to process all reports in this article, although it is still possible to do so because the reports themselves can include useful contextual data.

[0147] At step S520, the input data is processed together by the machine learning model to produce an output finding. The output finding may be a replacement for the finding included in the initial input data, or may be a confirmation thereof.

[0148] The replacement finding (if any) is then output at step S530. Alternatively, the output finding confirms (is equal to) the initial finding, and this may also be output.

[0149] In step S540, the output finding is displayed, output in the form of sound, or otherwise brought to the user's attention by controlling appropriate transducers based on the calculated output finding w.

[0150] The machine learning model used at step S520 can implement a two-step operation, including first preprocessing S520_1, followed by a conversion S520_2 operation. The preprocessing may then include an encoding operation S520_11, followed by a combiner operation S520_12. Specifically, the preprocessing may include using the ML model to encode the (original) input data through the encoder portion of the model. The encoded data is then combined into combined encoded data by the combiner portion of the model. The combined encoded data is then provided to S520_13 for further ML processing. Specifically, the combined encoded data is then converted by the converter stage of the machine learning model in a second step S520_2.

[0151] The conversion operation converts the combined encoded output into an output finding. The conversion may include regression or classification into text or encoded data, which indicates the output finding, if applicable. The output finding may be a replacement finding or a confirmation of the initial finding.

[0152] The preprocessing and conversion operations can each be configured as a separate ML model. The two models can be considered as parts of a super model that includes the two models. Some or each model may include sub-models. The encoder of the preprocessing model may partially include an ML model (such as a sequential data processing or image classifier), and the combiner may include a separate ML model. The converter model itself can be implemented as a sub-ML model. Therefore, the super ML model can include multiple ML sub-models that are nested and / or sequential.

[0153] The combined encoded data produced by step S520_12 includes cross contributions from both the input image and the initial lookup and optionally from context data. For example, the combined encoded data can be represented as a feature map in a CNN or other NN model. The encoder encodes the data into a matrix or vector form. The encoder can be used at least in part for some or each channel as a separate unique hot encoding scheme as described above. The conversion operation or combiner can include a weighted linear combination of Logie values ​​from a previous layer, where nonlinearity is applied to produce a contribution, which can then be passed to the nodes of the next layer.

[0154] Recurrent networks can be used in encoding operations to process a portion of the input data, including in particular patient record history data, as this has a variable size or dimension. Therefore, other data types in different channels where the expected size / size (such as the length of the history data, etc.) can also be processed.

[0155] The encoding operation may include classifying the input image (to which the initial finding is relevant) into a classification vector. The index of this vector may represent a medical condition, etc., and each entry represents a corresponding score for the corresponding condition. The vector may be "one hot" (binarized) by setting the index with the largest score to "1" and invalidating all remaining entries.

[0156] As an optional step S550, the calculated alternative lookups are located in the input image of the initial input data. This may include using a heat map technique or similar technique, where the contribution to the final result (alternative finding) of pixels or pixel regions in the input image is measured, and those image pixel regions are located indicating that the contribution exceeds a given threshold compared to the remaining pixels. As described above, gradient-based techniques applied to ML models may be used, which are viewed as differentiable functions.

[0157] The output as a confirmation or alternative finding may be in text form or may be in medical code or other form.

[0158] Reference now Figure 6 The block diagram in Figure 6 The flowchart of the method for training the machine learning models PP and COM described above is shown. Preferably, the two are trained together. The training data v~ and w~ are the ones in Figure 4 Described in .

[0159] In step S610, training data (v~ k ,w~ k ). Each pair consists of the training input x k and the associated target w~ k . v~ k , as above Figure 5 As defined in .

[0160] At step S620, the training input v~ k Applied to the initialized machine learning model M to produce training outputs.

[0161] Training output M(v~ k ) and the associated target w~ k The deviation or residual of is quantified by a cost function F at S630. At step S640, one or more parameters of the model are adapted in one or more iterations in the inner loop to improve the cost function. For example, the model parameters are adapted to reduce the residual as measured by the cost function. In the case of using a convolutional model M, the parameters particularly include the weights of the convolution operator.

[0162] The training method then returns in an outer loop to step S610, where the next pair of training data is fed. In step S620, the parameters of the model are adapted so that the aggregate residual of all pairs considered is reduced, in particular minimized. A cost function quantifies the aggregate residual. Forward-backward propagation or similar gradient-based techniques may be used in the inner loop.

[0163] Examples of gradient-based optimization may include gradient descent, stochastic gradient, conjugate gradient, maximum likelihood method, EM maximization, Gauss-Newton, etc. Methods other than gradient-based methods are also contemplated, such as Nelder-Mead, Bayesian optimization, simulated annealing, genetic algorithms, Monte Carlo methods, etc.

[0164] More generally, the parameters of the model NN are adjusted to improve an objective function F which is a cost function or utility function. In various embodiments, the cost function is configured to measure aggregated residuals. In various embodiments, the aggregate residual is achieved by summing all or some of the residues of all pairs considered. In an embodiment, the aggregation of residuals is achieved by summing all or some of the residuals of all pairs considered. In particular, in a preferred embodiment, the outer summation is performed in batches (subsets of training instances), whose summed residuals are considered together when adjusting the parameters in the inner loop. The outer loop then proceeds to the next batch, and so on, until the necessary number of training data instances have been processed. Instead of processing pair by pair, the outer loop accesses multiple pairs of training data items at a time, and the loop is batched. Therefore, the sum of the index "k" in the above formula (1) can be extended in batches over the entire corresponding batch.

[0165] Although mainly reference is made above to NNs, the principles disclosed herein are not limited to NNs. For example, instead of using NNs for the preprocessor PP, another approach may be used, such as a Hidden Markov Model (HMM) or a Sequential Dynamical System (SDS), in particular a Cellular Automaton (CA).

[0166] Similarly, for example, instead of using a NN for the combiner COM, another ML approach may be used, such as a support vector machine (SVM) or a boosted decision tree.

[0167] The components of the recommender MA may be implemented as one or more software modules, running on one or more general purpose processing units PU, such as a workstation associated with the imager IA, or on a server computer associated with a group of imagers.

[0168] Alternatively, some or all components of the recommender MA may be arranged in hardware, such as a suitably programmed microcontroller or microprocessor, such as an FPGA (field programmable gate array) or as a hardwired IC chip, an application specific integrated circuit (ASIC) integrated into the imaging system IA. In further embodiments, the recommender MA may be implemented in both hardware and software (partially in software and partially in hardware).

[0169] The different components of the recommender MA may be implemented on a single data processing unit PU. Alternatively, some or more components are implemented on different processing units PU, possibly arranged remotely in a distributed architecture and connectable in a suitable communication network, such as in a cloud setup or a client-server setup or the like.

[0170] One or more features described herein may be configured or implemented as circuitry encoded in a computer-readable medium, or configured or implemented with the circuitry, and / or a combination thereof. The circuitry may include discrete and / or integrated circuits, systems on a chip (SOCs), and combinations thereof, machines, computer systems, processors and memories, computer programs.

[0171] In a further exemplary embodiment of the present invention, a computer program or a computer program element is provided, which is characterized by being adapted to execute the method steps of the method according to one of the preceding embodiments on a suitable system.

[0172] Thus, the computer program element may be stored on a computer unit, which may also be part of an embodiment of the present invention. The computing unit may be adapted to perform or induce the performance of the steps of the above method. Furthermore, it may be adapted to operate components of the above device. The computing unit may be adapted to automatically operate and / or perform the user's order. The computer program may be loaded into a working memory of a data processor. Thus, the data processor may be equipped to perform the method of the present invention.

[0173] This exemplary embodiment of the invention covers both a computer program which uses the invention from scratch and a computer program which by an update turns an existing program into a program which uses the invention.

[0174] Furthermore, the computer program element may be able to provide all necessary steps to complete the procedures of an exemplary embodiment of the method as described above.

[0175] According to a further exemplary embodiment of the present invention, a computer readable medium, such as a CD-ROM, is presented, wherein the computer readable medium has a computer program element stored thereon, the computer program element being described by the preceding section.

[0176] The computer program may be stored and / or distributed on a suitable medium (particularly, but not necessarily, a non-transitory medium) such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0177] However, the computer program may also be presented over a network like the World Wide Web and may be downloaded from such a network into a working memory of a data processor. According to a further exemplary embodiment of the invention, a medium for making a computer program element available for downloading is provided, the computer program element being arranged to perform a method according to one of the previously described embodiments of the invention.

[0178] It must be noted that embodiments of the present invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method type claims, while other embodiments are described with reference to device type claims. However, a person skilled in the art will gather from the above and following descriptions that, unless otherwise stated, any combination of features relating to different subject matters, in addition to any combination of features belonging to one type of subject matter, is also considered to be disclosed by the present application. However, all features may be combined to provide a synergistic effect that is more than the simple sum of the features.

[0179] Although the present invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description are to be considered illustrative or exemplary rather than restrictive. The present invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments may be understood and effected by those skilled in the art in practicing the claimed invention by studying the drawings, the disclosure, and the dependent claims.

[0180] In the claims, the word "comprising" does not exclude other elements or steps and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope. Such reference signs may consist of numbers, letters or any alphanumeric combination.

Claims

1. A preprocessor (PP) component for a machine learning system (MLS) for processing medical data, comprising: at least one input interface (IN) for receiving human-generated initial findings for a patient and medical images associated with said findings; an encoder (ENC) for encoding the findings and the medical image into encoded data, the encoded data comprising encoded image data and encoded findings data; a combiner (COM) component for combining the encoded findings and the encoded image data into combined encoded data; and An output interface (OUT) for providing the combined encoded data to the machine learning system.

2. The system according to claim 1, wherein: The input interface (IN) is used to receive context data, which provides context information related to a report and / or an image, the encoder is configured to encode at least a portion of the context data into encoded data, and the combiner is used to combine the encoded context data with the image and the encoded report to obtain combined data.

3. The system according to claim 1 or 2, wherein: The contextual data includes any one or more of: i) patient medical history, ii) imaging request for the image, iii) statistics related to misdiagnosis.

4. A system according to any one of the preceding claims, wherein: The combiner (COM) and / or the encoder (ENC) are implemented as corresponding machine learning models.

5. The system according to claim 4, wherein: The machine model for the encoder (ENC) comprises a processing pipeline (SDP) configured for loop processing.

6. The system according to claim 5, wherein: The processing channel (SDP) is configured to process at least the encoded patient history.

7. A machine learning arrangement (MA) comprising a pre-processor component (PP) according to any of the preceding claims and a machine learning system (MLS).

8. The arrangement of claim 7, wherein: The machine learning system includes a machine learning model (TRF) configured to convert the combined encoded output into output data indicative of at least one second finding that is a replacement for or equal to the initial finding.

9. The arrangement of claim 8, wherein: The output includes: a natural text string or a medical finding code.

10. An arrangement according to claim 9, comprising a localizer (LC) configured to map the output data to image locations in the image.

11. A training system (TS) configured to train a machine learning model (ENC, COM, TRF) according to any one or both of claims 4 and 8 based on training data.

12. A method for preprocessing medical data for machine learning, comprising: receiving ( S510 ) a human-generated initial finding for a patient and a medical image related to the finding; encoding (S520_11) the findings and the medical image into encoded data, the encoded data comprising encoded image data and encoded findings data; combining (S520_12) the encoded discovery and the encoded image data into combined encoded data; and The combined encoded data is provided (S520_13) to a machine learning system.

13. A method for training a machine learning model (ENC, COM, TRF) according to any one or both of claims 4 and 8 based on training data.

14. A computer program element which, when executed by at least one processing unit, is adapted to cause the at least one processing unit (PU) to perform the method according to any one of claims 12 or 13.

15. At least one computer-readable medium having a program element according to claim 14 stored thereon, or having a machine learning module according to claim 4 or 8 stored thereon.

Images